Cross-modal source-channel joint coding and decoding method for coping with variable scenes
By combining the Transformer model, hidden diffusion model and Cross Attention module in the cross-modal channel, the differences in audio and video and haptic transmission requirements and image loss problems in cross-modal services are solved, and efficient image signal reconstruction and model generalization are achieved. Reduces training costs and power consumption.
Patent Information
- Application Number
- CN202411075764.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-07
AI Technical Summary
There are significant differences in the transmission requirements of audio and video and haptics in cross-modal services, resulting in the transmission system facing image loss and additional model training costs, especially in changing scenarios.
A cross-modal source channel joint encoding and decoding method is designed, including encoder and decoder based on Transformer model, combining hidden diffusion model and transfer learning, optimizing the transmission and reconstruction of image signals, and using Wasserstein to generate adversarial networks and PatchGAN authentication. The device performs image reconstruction and fuses haptic features through the Cross Attention module to improve image generation quality.
It reduces network power consumption, improves image reconstruction quality, reduces model training costs, and optimizes model generalization capabilities through transfer learning, which is suitable for load balancing and resource optimization in variable scenarios.
Smart Images

Figure CN119011844A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cross-modal image signal reconstruction, and in particular to a cross-modal signal source channel joint encoding and decoding method for coping with variable scenarios. Background Art
[0002] With the development of wireless communication technology and people's satisfaction with traditional multimedia services such as audio and video, they will further pursue multi-sensory immersive experience. Then, multimodal services integrating audio, video and tactile signals will gradually become the mainstream in various scenarios (remote operation, online education, e-health, digital twins). For example, in remote education scenarios, audio and video services that integrate tactile perception and feedback can solve the problem that remote practical teaching cannot be effectively carried out, and improve students' learning effects and immersive learning experience. In virtual world interaction, tactile perception can bring users a more realistic interactive experience.
[0003] However, in multimodal services, there are significant differences in the transmission requirements of audio, video and touch. For the former, high throughput needs to be met, and touch is more sensitive to low latency and high reliability. Traditional audio and video communications and tactile communications cannot meet the requirements. Based on this, cross-modal communication came into being. Cross-modal communication makes full use of the potential correlation and semantic public information between modalities, establishes a general cross-modal flow scheduling scheme, and uses AI methods to achieve efficient multimodal flow, processing and recovery. It has also achieved initial results in application scenarios such as remote acupuncture teaching, remote operation, and remote throat swabs.
[0004] However, since it is necessary to transmit the fused features extracted from the cross-modal signals, higher requirements are placed on the transmission system. Recent studies have shown that the Transformer model has demonstrated powerful semantic processing capabilities, especially in the semantic transmission of long texts. Secondly, the latent diffusion model is a technology that has attracted much attention. It has achieved outstanding success in text-image generation tasks. However, the existing cross-modal transmission faces the problem of image loss, so a tactile-image generation method is needed. Finally, for different cross-modal communication tasks and environments, additional model training costs are often incurred. In summary, it is necessary to design a source-channel joint encoding and decoding method to improve the image reconstruction quality in variable scenarios in cross-modal communication. Summary of the invention
[0005] In view of the above-mentioned problems, the present invention is proposed.
[0006] Therefore, the technical problem solved by the present invention is that in cross-modal services, there are significant differences in the transmission requirements of audio, video and touch. Due to the particularity of cross-modal transmission, there may be a situation where a certain modal signal is lost, which puts higher requirements on the transmission system. 3. Different cross-modal communication tasks and scenarios often lead to additional model training costs.
[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a cross-modal source channel joint encoding and decoding method for coping with variable scenarios, comprising:
[0008] For the scenario where image signals and tactile signals are transmitted simultaneously, the system capacity is maximized and semantic errors are minimized based on the cross-modal source encoder;
[0009] Considering the scenario of image signal loss in cross-modal communication, a cross-modal source encoding and decoding optimization scheme for tactile-image generation based on implicit diffusion model is designed;
[0010] For different communication tasks and channel environments, transfer learning is used to optimize the generalization ability of the model and accelerate the model training process.
[0011] As a preferred solution of the source-channel joint encoding and decoding method of the present invention, wherein: the cross-modal source encoder includes a tactile feature extraction module, an image feature extraction module and a modality fusion module;
[0012] The tactile feature extraction module and the image feature extraction module extract features within each modality;
[0013] After the features of the image and tactile signal are obtained respectively, they are spliced together along the channel dimension and input into the modal fusion module for fusion. They are trained through the classification task, and the loss function is expressed as:
[0014]
[0015] in, Represents the tactile feature extraction network Image feature extraction and modal fusion Output image features, tactile features and fusion features, y j The label of the object. Represents the tactile feature extraction network Image feature extraction Modal Fusion and cross-modal source encoder integration network Parameters;
[0016] At the receiving end, the corresponding image signal needs to be reconstructed from the fused features through the cross-modal source decoder;
[0017] The Wasserstein generative adversarial network is used to achieve image reconstruction, and the discriminator D adopts the design of PatchGAN;
[0018] Generator G generates fused features as images The discriminator D is responsible for distinguishing between real images v and generated images The training process is the process of confrontation between D and G, and the loss function is expressed as:
[0019]
[0020]
[0021] in, ~U[0,1].
[0022] As a preferred solution of the source-channel joint encoding and decoding method of the present invention, a Transformer encoder is designed, the encoder comprises several identical layers, each layer is composed of a multi-head self-attention sublayer and a position-based feedforward network sublayer, and each sublayer uses a residual connection layer and layer normalization;
[0023] The multi-head self-attention layer uses a scaled dot product model to calculate the self-attention score. When calculating the self-attention, Q, K, and V are the same. Q, K, and V are projected through h different linear transformations, and the self-attention scores of different representation subspaces are concatenated to obtain the final attention score.
[0024] The position-based feedforward network performs a nonlinear transformation on the representation of all positions in the sequence, through a two-layer fully connected layer, the activation function of the first layer is ReLu, and the second layer is a linear activation function, expressed as,
[0025] FFN out =max(0,xW1+b1)W2+b2,
[0026] Among them, x is the fused feature vector obtained in the previous step, W1, W2, b1 and b2 are all learnable parameters in the feedforward network, FFN out is the output of the feedforward network;
[0027] The residual connection layer enables the network to be used only for the current difference during the update iteration process;
[0028] The normalization layer normalizes the input of each layer of neurons to the same mean variance, accelerating the convergence of the model;
[0029] The calculation process of the sub-layer is expressed as:
[0030] sub_layer_output=LayerNorm(x+sublayer(x)),
[0031] Design the Transformer decoder, which adds an encoder-decoder attention sublayer between the multi-head self-attention sublayer and the feedforward network sublayer;
[0032] The channel is modeled as a neural network layer representing the process of encoded fusion features passing through the channel, considering the additive Gaussian white noise channel and Rayleigh slow fading channel;
[0033] In the additive white Gaussian noise channel, the channel transfer function can be expressed as:
[0034] y=z+n
[0035] Among them, z is the channel transmission symbol, n is the channel noise, which are independent and identically distributed, and σ 2 is the average noise power;
[0036] The signal-to-noise ratio of the channel is By changing σ 2 To change the signal-to-noise ratio of the channel;
[0037] The transfer function of the Rayleigh slow fading channel is,
[0038] y=hz+n
[0039] Where h is the channel gain, n is the channel noise, h and n are subject to the parameters and Different normal distributions.
[0040] As a preferred solution of the source-channel joint coding and decoding method of the present invention, wherein: the cross-modal source coding and decoding optimization solution includes mapping the image signal from the pixel domain to the feature domain;
[0041] For a given image x in the pixel domain, Using Encoders Encode x as latent variable features Using the decoder Reconstruct the image from the feature space, expressed as,
[0042]
[0043] Where z∈R h×w×c .
[0044] As a preferred solution of the source-channel joint coding and decoding method of the present invention, wherein: a learning process of adding noise and removing noise is performed on the image features in the feature domain;
[0045] For the forward noise addition process, a forward diffusion process is defined, expressed as,
[0046]
[0047] After the forward noise addition process, the reverse q(x t |x t-1 ) process to obtain the recreated real sample x t ;
[0048] Design the U-Net model to learn approximate conditional probability.
[0049] As a preferred solution of the source-channel joint encoding and decoding method of the present invention, wherein: the design of the U-Net model to learn the approximate conditional probability includes introducing a Cross Attention module;
[0050] The Cross Attention is a multi-head attention mechanism that establishes an association between two different input sequences and passes the information of one sequence to the other sequence. Through the Cross Attention module, the U-Net model associates a part of the noisy image feature matrix with specific information of the tactile feature label;
[0051] Convert the tactile condition data y into the tactile feature expression inside the model τ θ (y is mapped to the middle layer of the Unet model through the Cross Attention module to interact with the image information, expressed as
[0052]
[0053] in, represents the process operation before the latent variable feature z enters the Unet network after the forward noise addition process, τ θ (y) represents the corresponding tactile feature information, are three trainable parameter projection matrices;
[0054] Calculate the attention weights and perform weighted fusion, expressed as,
[0055]
[0056] Among them, d represents the dimension of query vector Q and key vector K. Represents the attention weight.
[0057] As a preferred solution of the source channel joint coding and decoding method of the present invention, the interactive information between the tactile features and the image features is obtained and fused into the image features. The loss function is expressed as:
[0058]
[0059] Among them, ∈ represents the original image features without noise addition, ∈θ(z t ,t,τθ(y)) represents the Unet network output;
[0060] The FID parameter index is used to guide the image generation, which is expressed as:
[0061]
[0062] Where Tr represents the trace of the matrix, x and g represent the real picture and the generated image, μ represents the mean, and Σ is the covariance matrix;
[0063] The total loss function is expressed as,
[0064] L HVLDM =L C_LDM +L FID (x,g).
[0065] As a preferred solution of the source-channel joint encoding and decoding method of the present invention, wherein: the use of transfer learning to optimize the generalization ability of the model and accelerate the training process of the model includes specifying a first transmission task and a second transmission task for a scenario in which an image signal and a tactile signal are transmitted simultaneously; the first transmission task is to transmit the fusion features of the image signal and the tactile signal; the second transmission task is to transmit only the tactile signal features;
[0066] Load the pre-trained Transformer encoder C for the first transfer task α (·), decoder Weights; Input feature dataset D required for the second transfer task; Freeze C α (·), The third layer network parameters; retrain the first and second layer network parameters C α1、2 (·), Obtain the first and second layer encoder parameters of the new Transformer; at the same time, cross-modal communication will also face different channel signal-to-noise ratio changes;
[0067] Input the pre-trained Transformer encoder model C α (·), decoder Different channel signal-to-noise ratio parameters N1; freeze C α(·), The first and second layer network parameters; retrain the third layer network C α3 (·), Get the newly trained network C α3 (·), parameter.
[0068] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the source-channel joint coding and decoding method as described above when executing the computer program.
[0069] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the source-channel joint encoding and decoding method as described above.
[0070] Beneficial effects of the present invention: The source-channel joint encoding and decoding method provided by the present invention reduces the power consumption in the network, uses the fitness function to measure the load balance between the cloud network and the server (host), and converts the load balance problem into an optimization problem, so that it can be applied to processing resource optimization. Using the results of the particle swarm as the initial population of the genetic algorithm, the present invention achieves better results in terms of execution cost, load balance and completion time. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0072] Figure 1 An overall flow chart of a cross-modal source channel joint encoding and decoding method for coping with variable scenarios provided by an embodiment of the present invention.
[0073] Figure 2 A diagram of the structure of a three-layer Transformer encoder in a cross-modal source-channel joint encoding and decoding method for coping with variable scenarios provided by an embodiment of the present invention.
[0074] Figure 3 An embodiment of the present invention provides a cross-modal source channel joint encoding and decoding method based on a diffusion model.
[0075] Figure 4 A structural diagram of a U-Net denoising model in a cross-modal source channel joint encoding and decoding method for coping with variable scenarios provided by an embodiment of the present invention.
[0076] Figure 5A source-channel joint encoding and decoding method based on transfer learning is provided as an embodiment of the present invention.
[0077] Figure 6 An embodiment of the present invention provides a cross-modal source channel joint encoding and decoding method for coping with variable scenarios, and shows the relationship between the image reconstruction quality PSNR and the signal-to-noise ratio.
[0078] Figure 7 A comparison diagram of a source-channel joint encoding and decoding method based on a diffusion model provided by an embodiment of the present invention and other experiments.
[0079] Figure 8 An ablation experiment related to a source-channel joint encoding and decoding method based on a diffusion model is provided for one embodiment of the present invention.
[0080] Fig. 9 A comparison chart of model training convergence speed in a source-channel joint encoding and decoding method based on transfer learning provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0081] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0082] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0083] Example 1
[0084] Reference Figure 1-Figure 5 , is an embodiment of the present invention, and provides a cross-modal source channel joint encoding and decoding method for coping with variable scenarios, comprising:
[0085] Step 1: Figure 1 As shown in the figure, a Transformer-based cross-modal channel codec communication system is designed for the scenario where image signals and tactile signals are transmitted simultaneously. The main goal of this system is to maximize system capacity and minimize semantic errors, rather than just focusing on bit or symbol errors in traditional communications.
[0086] Step 1-1: Design a cross-modal source encoder Including tactile feature extraction Image feature extraction and modal fusion Three modules. Among them, v represents the image signal, h represents the tactile signal, and θ represents the module parameters. The main purpose of the tactile feature extraction module and the image feature extraction module is to extract features within each modality. After obtaining the features of the image and tactile signal respectively, they are spliced together along the channel dimension and input into the modal fusion module for fusion. All three modules are trained with classification tasks. The output of the last fully connected layer is (None, 6), and the output of the second to last layer is (None, x), where the value of x is selected as 640 after parameter experiments, and its loss function is designed as:
[0087]
[0088] in, Represents the tactile feature extraction network Image feature extraction and modal fusion Output image features, tactile features and fusion features. j Represents the label of the object. Represents the tactile feature extraction network Image feature extraction Modal Fusion f and cross-modal source encoder integration network Parameters.
[0089] Step 1-2: Design a cross-modal source decoder. At the receiving end, the corresponding image signal needs to be reconstructed from the fused features. The Wasserstein Generative Adversarial Network (WGAN) method is used to achieve image reconstruction. The discriminator D adopts the design of PatchGAN. The generator G generates the fused features into an image The discriminator D is responsible for distinguishing between real images v and generated images The generator G needs to produce pictures that are closer to reality, while the discriminator D needs to have a stronger ability to distinguish true from false. Therefore, the entire training process is a process of confrontation between D and G. The loss function here is defined as:
[0090]
[0091] in, ~U[0,1].
[0092] Steps 1-3, such as Figure 2As shown. Design a Transformer-based cross-modal channel encoder and decoder. Here, the encoder-decoder structure is mainly used. Each layer of the decoder takes the output of the final encoder as input. This method designs an overall architecture of the Tansformer model consisting of 3 layers of encoders and 3 layers of decoders. First, the Transformer encoder is designed, which contains several identical layers. Each layer consists of a multi-head self-attention sublayer and a position-based feedforward network sublayer, and each sublayer uses residual connections and normalization layers. The number of self-attention units in each layer is selected to be 128, and the activation function is a linear function. The multi-head self-attention layer uses a scaled dot product model to calculate the self-attention score. When calculating the self-attention, Q, K, and V are the same. Then, Q, K, and V are projected through h different linear transformations, and the self-attention scores of different representation subspaces are spliced together to obtain the final attention score. The position-based feedforward network performs nonlinear transformations on the representations of all positions in the sequence, which increases the expressiveness of the model. The feedforward network (which performs nonlinear changes) is a two-layer fully connected layer, with the activation function of the first layer being ReLu and the second layer being a linear activation function:
[0093] FFN out =max(0,xW1+b1)W2+b2,
[0094] Among them, x is the fused feature vector obtained in the previous step, W1, W2, b1 and b2 are all learnable parameters in the feedforward network, and FFNout represents the output of the feedforward network. The residual connection layer enables the network to focus only on the current difference during the update and iteration process, and is used to deal with problems such as grid degradation and gradient disappearance caused by the multi-layer network training process; the normalization layer normalizes the input of each layer of neurons to the same mean variance to accelerate model convergence. The entire calculation process of these two sub-layers can be expressed as:
[0095] sub_layer_output=LayerNorm(x+sublayer(x)),
[0096] Next, the decoder of Transformer is designed. Its design has a similar structure to that of the encoder, but the decoder also adds an encoder-decoder attention sublayer between the multi-head self-attention sublayer and the feedforward network sublayer. Its activation function is a linear function. The two attention sublayers have the same structure, but the sources of query, key, and value are different. Specifically, the query of the encoder-decoder attention sublayer is the output result of the previous decoder layer, and the key and value are generated by decomposing the output of the entire encoder, while the multi-head self-attention sublayer obtains the query, key, and value from the previous decoder layer. In addition, the mask function is introduced to this layer to retain its self-return property, ensuring that the prediction depends only on the generated output word, ensuring the consistency of the training phase and the inference phase. The output of the decoder is a float vector, which is input into a linear layer and then a softmax layer to generate the final output word. Among them, the linear layer uses a fully connected neural network to map the output vector of the decoder to a larger linear space. Then, the softmax layer converts these vectors into probabilities, and finally sorts them according to the probability values, locates the dimension with the highest probability and generates the corresponding features, which is the output of this time step. This method uses MSE loss to train the model.
[0097] Step 1-4: In order to represent the process of the encoded fusion features being transmitted through the channel, the channel is modeled as a series of non-trainable but differentiable neural network layers. Consider two commonly used channel models: additive white Gaussian noise (AWGN) channel and Rayleigh slow fading (Rayleigh) channel. Under the AWGN channel, the channel transfer function can be expressed as: y = z + n. z is the channel transmission symbol, n is the channel noise, which is independent and identically distributed, and σ 2 is the average noise power. The signal-to-noise ratio of the channel is By changing σ 2 To change the signal-to-noise ratio of the channel. The transfer function of the Rayleigh slow fading channel is: y = hz + n. h is the channel gain, n is the channel noise, h and n are subject to the parameters and Different normal distributions.
[0098] Step 2: Figure 3 As shown in the figure, considering the scenario of image signal loss in cross-modal communication, a cross-modal source encoding and decoding optimization scheme for tactile-image generation based on implicit diffusion model is designed. The method is as follows: 1. Design a feature extraction module to realize the mapping of images from pixel domain to feature domain; 2. Perform a learning process of adding noise and denoising on the obtained image features in the feature domain; 3. Train the tactile sense to obtain tactile features in the conditional domain, and use the obtained tactile features and guide the corresponding image generation through design. Its model structure is as follows Figure 2 shown.
[0099] Step 2 considers the scenario of image signal loss in cross-modal communication, and designs a cross-modal source encoding and decoding optimization scheme for tactile-image generation based on implicit diffusion model. The method is as follows: 1. Design a feature extraction module to realize the mapping of images from pixel domain to feature domain; 2. Perform a learning process of adding noise and denoising on the obtained image features in the feature domain; 3. Train the tactile sense to obtain tactile features in the conditional domain, and use the obtained tactile features to guide the corresponding image generation through design.
[0100] Step 2-1: Map the image signal from the pixel domain to the feature domain. For a given image x in the pixel domain, Using Encoders Encode x into latent variable features: Using the decoder Reconstructing the image from the feature space can be obtained:
[0101]
[0102] Where z∈R h×w×c The specific encoder and decoder structures can refer to the source encoder structure and source decoder structure in step 1.
[0103] Step 2-2: Perform the learning process of adding noise and denoising on the image features obtained in the previous step in the feature domain. For the forward denoising process, its purpose is to make all samples tend to be isotropically distributed by continuously adding noise of the same distribution to the samples. Specifically, it first defines a forward diffusion process. In this process, as shown in the following formula, a small amount of Gaussian noise is added to the sample. The step length of adding noise is determined by Control, thus generating a series of noise samples x1,…,x T .
[0104]
[0105] After the forward noise addition process, it is necessary to reverse q(x t |x t-1 ) process to obtain the recreated real sample x t However, since this process traverses the entire data set, it is impossible to obtain the true sample through simple estimation. Therefore, the U-Net model is designed to learn to approximate these conditional probabilities.
[0106] Step 2-3, such as Figure 4As shown, the overall structure of the U-Net network proposed in the present invention is similar to the standard U-Net, but the innovation lies in the introduction of the SpatialTransformer (Cross Attention) module. Therefore, this module is introduced in detail here. Cross Attention is a multi-head attention mechanism that can establish an association between two different input sequences and pass the information of one sequence to the other. The Cross Attention module is used to control the fusion interaction of tactile and image information, that is, through this module, U-Net can associate a part of the noisy image feature matrix with the specific information of the tactile feature label.
[0107] In order to preprocess the conditional tactile information y, the tactile feature extraction module in step 1 is used, denoted as τ θ , which is used to convert the tactile condition data y into the tactile feature expression inside the model Then τ θ (y) Through the Cross Attention module, it is mapped to the middle layer of the Unet model to fuse and interact with the image information, so as to achieve the effect of guiding image generation with tactile information. Then the above process can be expressed as follows: First, the query vector Q, key vector K, and value vector V are calculated respectively through three independent fully connected layers, as shown in the formula:
[0108]
[0109] in, Represents the process operation of the latent variable feature z after the forward noise addition process and before entering the Unet network. τ θ (y) represents the corresponding tactile feature information, are three trainable parameter projection matrices. Next, the attention weights are calculated and weighted fusion is performed as shown in formula (4.7):
[0110]
[0111] Where d represents the dimension of the query vector Q and the key vector K. Represents the attention weight. In this way, the interactive information between tactile features and image features is obtained and integrated into the image features.
[0112] Step 2-4 Based on the above tactile-image condition generation process, the current loss function can be described by the following formula:
[0113]
[0114] Where ∈ represents the original image features without noise addition, ∈θ(z t,t,τθ(y)) represents the Unet network output.
[0115] In addition, the above loss function is derived from the mathematical principle of the model. In the tactile-visual generation task proposed in this paper, the FID (Fréchet Inception Distance) parameter indicator can be used to guide the image generation. FID is an indicator used to evaluate the difference between the generated model and the real data distribution. In this paper, it can be expressed as:
[0116]
[0117] Where Tr is the trace of the matrix. x and g are the real and generated images, μ is the mean, and Σ is the covariance matrix. A lower FID means that the distributions of the two images are closer, that is, the model generates better images.
[0118] The total loss function of the above model can be expressed as
[0119] L HVLDM =L C_LDM +L FID (x,g).
[0120] Step 3: For different communication tasks and environments, the transfer learning method is used to ensure the generalization ability of the model, accelerate the model training process, and reduce the model training cost. Figure 5 shown.
[0121] Step 3-1, load the pre-trained channel encoder and channel decoder according to the assumed known task K0 and channel condition N0. For applications with different background knowledge, the present invention only needs to retrain part of the Transformer codec layer. Specifically: According to the two transmission scenarios faced by the present invention, it is specified that Task 1: requires the transmission of the fusion features of the image signal and the tactile signal; Task 2: only the tactile signal features are transmitted.
[0122] Step 3-11: Load the pre-trained Transformer encoder C for task 1 α (·), decoder Weight.
[0123] Step 3-12: Input the feature dataset D required for task 2.
[0124] Step 3-13, Freeze C α (·), The third layer network parameters.
[0125] Step 3-14: Retrain the first and second layer network parameters C α1、2(·),
[0126] Step 3-15: Get the first and second layer encoder parameters of the new Transformer.
[0127] Step 3-2: At the same time, cross-modal communication will also face different channel signal-to-noise ratio changes.
[0128] Step 3-21. Input: pre-trained Transformer encoder model C α (·), decoder Different channel signal-to-noise ratio parameters N1.
[0129] Step 3-22, Freeze C α (·), The first and second layer network parameters.
[0130] Step 3-23, retrain the third layer network C α3 (·),
[0131] Step 3-24, obtain the newly trained network C α3 (·), parameter.
[0132] In the above process, the training loss function is set to L HVLDM , and the Adam optimization algorithm is used for optimization. The Adam optimization algorithm is based on the gradient descent method and adds an adaptive adjustment learning rate. Its expression is as follows:
[0133] m t =β1m t-1 +(1-β1)h t
[0134]
[0135] Among them, g t θ is the gradient of the parameter, β1, β2 are the decay coefficients of the two exponentially weighted averages, and m and v are the bias-corrected moving averages of the gradient. t and θ t+1 They are the network parameters before and after the update. The network parameters here refer to the first, second, and third layer network parameters in the Transformer encoder and decoder.
[0136] Embodiment 2, an embodiment of the present invention, which is different from the previous embodiment in that:
[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0138] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0139] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0140] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0141] Example 3, reference Figure 6-Figure 9 , which is an embodiment of the present invention, provides a cross-modal source channel joint encoding and decoding method for coping with variable scenarios. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.
[0142] This example uses the AU dataset, which contains multimodal information of 63 objects, including images, kinesthetic and tactile data (audio vibration). These objects are mainly toys and household items. Some objects have similar shapes and colors in images, but similar materials and textures but different colors in touch. These objects reflect the different effects of images and touch on perception.
[0143] The present invention conducts experiments on three modules of step 1, step 2 and step 3 respectively.
[0144] Step 1 The related comparative experiment adopts the source channel joint coding based on neural network and the typical source channel separation coding method. Source channel joint coding (JSCC) based on deep neural network: The network consists of a bidirectional long short-term memory (BLSTM) layer. It is marked as JSCC-LSTM and JSCC-CNN in the simulation diagram. The traditional method uses the following technologies respectively: Channel coding: Turbo coding and Reed-Solomon (RS) coding. The turbo decoding method used in the method of the present invention is a log-MAP algorithm with 5 iterations.
[0145] Step 2 uses comparative experiments and ablation experiments. The comparative experiments use the following two methods:
[0146] Style GAN: It generates an intermediate latent space through a mapping network, which reduces the entanglement of the input latent space. At the same time, it proposes two new indicators to quantify the coupling of latent spaces - perceptual path length and linear separability. Finally, it has achieved outstanding success in the task of face image generation.
[0147] Projected GAN: Generative adversarial networks generate high-quality images but are challenging to train. They require careful regularization, extensive computation, and expensive hyperparameter sweeps. This model makes significant progress on these problems by projecting generated and real samples into a fixed, pre-trained feature space. Since the discriminator is found to be unable to fully utilize the deeper features of the pre-trained model, a more effective strategy is proposed to mix features across channels and resolutions. ProjectedGAN improves image quality, sampling efficiency, and convergence speed.
[0148] The present invention: the method of this embodiment.
[0149] The following is an introduction to the measurement indicators used in this invention: The IS indicator is an indicator used to evaluate the quality of generated images, which combines the diversity and authenticity of the image. The Inception Score is an indicator of the KL divergence, which is the exponential mean of the KL divergence of the conditional distribution of each image and the overall distribution. The mathematical expression is as follows:
[0150] IS = exp(E x [KL(p(y|x)|p(y))]),
[0151] Among them, p(y|x) is the conditional distribution, which represents the probability distribution of category y given image x; p(y) is the overall distribution, which represents the probability distribution of category y for all images; E x It means taking the expectation for all images.
[0152] FID is an indicator used to evaluate the difference between the generated model and the real data distribution. Then in the present invention, it can be expressed as:
[0153]
[0154] Where Tr represents the trace of the matrix. x and g represent the real image and the generated image, μ represents the mean, and Σ is the covariance matrix.
[0155] In addition, PSNR is also a commonly used objective image evaluation indicator, and its mathematical expression is defined as:
[0156]
[0157] Among them, MSE is the mean square error between the real image and the generated image. The larger the PSNR, the better.
[0158] Table 1 is the experimental results of the present invention.
[0159]
[0160] From Table 1 and Figure 6 It can be seen that compared with the above state-of-the-art methods, our proposed method has obvious advantages. Figure 6 In the traditional baseline shown, when turbo coding is used for channel coding, on the AWGN channel, when the signal-to-noise ratio is greater than 12dB, the performance of the traditional method is better than that of the method based on deep neural network. All methods that support deep learning are more competitive under low signal-to-noise ratio conditions. On the sharp fading channel, the performance of the traditional scheme under various conditions is not as good as that of the deep learning-based method. The method of the present invention achieves the best performance indicators under most conditions. This is also because in the environment of complex multiplicative noise in the sharp fading channel, the traditional method has a higher degree of distortion and cannot accurately decode the corresponding eigenvector, resulting in a decrease in performance when the final source is decoded.
[0161] like Figure 7 The results of the present invention are shown to be superior to the compared methods, including Style GAN and Projected GAN. The present method trains the diffusion model in a fixed latent space, avoiding the trade-off between reconstruction quality and learning the latent space prior. The experimental results are significantly better than the two methods in all indicators. The tactile modality information is successfully mapped to the latent feature representation of the UNet network through the multi-head attention mechanism, which can successfully guide the generation of images.
[0162] like Figure 8 As shown in the figure, HVLDMwithoutLc_LDM represents the case of unconditional generation, and the model's ability to generate pictures corresponding to the input tactile sense is greatly reduced. In the five experiments, the model was unable to accurately restore the original picture in most cases. This shows that in this structure, the tactile information in the conditional field plays an important guiding role in picture generation.
[0163] HVLDMwithoutLfid represents the model whose loss function is only composed of L C-LDM The experimental results show that the images generated by the model without Lfid are not as good as those of the complete model in terms of IS and FID indicators, that is, the similarity of the generated images lags behind the HVLDM model. This also shows that the Lfid loss function plays a role in constraining image generation in this task and making it closer to the real image category.
[0164] Fig. 9The training efficiency and performance under different background knowledge are shown, where the model will be trained and retrained on the new background knowledge using the same channel (AWGN) under different background knowledge. The model has the same structure and is retrained with the same parameters in each scenario. In Figures (a) and (c), it is reduced from 15 epochs to 5 epochs and 25 epochs to 10 epochs to reach convergence. This shows that transfer learning-assisted TransJSCC can help transceivers adapt to new communication environment requirements. As can be seen from Figures (b) and (d), the use of pre-trained models can speed up the training process of sharp channels. At the same time, due to the smaller number of epochs required during the retraining process, the required complexity is significantly reduced.
[0165] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A cross-modal source channel joint encoding and decoding method for coping with variable scenarios, characterized in that: include: For the scenario where image signals and tactile signals are transmitted simultaneously, the system capacity is maximized and semantic errors are minimized based on the cross-modal source encoder; Considering the scenario of image signal loss in cross-modal communication, a cross-modal source encoding and decoding optimization scheme for tactile-image generation based on implicit diffusion model is designed; For different communication tasks and channel environments, transfer learning is used to optimize the generalization ability of the model and accelerate the model training process.
2. The source-channel joint encoding and decoding method according to claim 1, characterized in that: The cross-modal source encoder includes a tactile feature extraction module, an image feature extraction module and a modality fusion module; The tactile feature extraction module and the image feature extraction module extract features within each modality; After the features of the image and tactile signal are obtained respectively, they are spliced together along the channel dimension and input into the modal fusion module for fusion. They are trained through the classification task, and the loss function is expressed as: in, Represents the tactile feature extraction network Image feature extraction and modal fusion Output image features, tactile features and fusion features, y j The label of the object. Represents the tactile feature extraction network Image feature extraction Modal Fusion and cross-modal source encoder integration network Parameters; At the receiving end, the corresponding image signal needs to be reconstructed from the fused features through the cross-modal source decoder; The Wasserstein generative adversarial network is used to achieve image reconstruction, and the discriminator D adopts the design of PatchGAN; Generator G generates fused features as images The discriminator D is responsible for distinguishing between real images v and generated images The training process is the process of confrontation between D and G, and the loss function is expressed as: in, 3. The source-channel joint encoding and decoding method according to claim 2, characterized in that: Design the Transformer encoder, which contains several identical layers. Each layer consists of a multi-head self-attention sublayer and a position-based feedforward network sublayer, and each sublayer uses a residual connection layer and layer normalization. The multi-head self-attention layer uses a scaled dot product model to calculate the self-attention score. When calculating the self-attention, Q, K, and V are the same. Q, K, and V are projected through h different linear transformations, and the self-attention scores of different representation subspaces are concatenated to obtain the final attention score. The position-based feedforward network performs a nonlinear transformation on the representation of all positions in the sequence, through a two-layer fully connected layer, the activation function of the first layer is ReLu, and the second layer is a linear activation function, expressed as, <h2 style=";text-align:left;direction:ltr">FFN<h2 style=";text-align:left;direction:ltr"> out <h2 style=";text-align:left;direction:ltr"> =max(0,xW1+b1)W2+b2, Among them, x is the fused feature vector obtained in the previous step, W1, W2, b1 and b2 are all learnable parameters in the feedforward network, FFN out is the output of the feedforward network; The residual connection layer enables the network to be used only for the current difference during the update iteration process; The normalization layer normalizes the input of each layer of neurons to the same mean variance, accelerating the convergence of the model; The calculation process of the sub-layer is expressed as: sub_layer_output=LayerNorm(x+sublayer(x)), Design the Transformer decoder, which adds an encoder-decoder attention sublayer between the multi-head self-attention sublayer and the feedforward network sublayer; The channel is modeled as a neural network layer representing the process of encoded fusion features passing through the channel, considering the additive Gaussian white noise channel and Rayleigh slow fading channel; In the additive white Gaussian noise channel, the channel transfer function can be expressed as: y=z+n Among them, z is the channel transmission symbol, n is the channel noise, which are independent and identically distributed, and σ 2 is the average noise power; The signal-to-noise ratio of the channel is By changing σ 2 To change the signal-to-noise ratio of the channel; The transfer function of the Rayleigh slow fading channel is, y=hz+n Where h is the channel gain, n is the channel noise, h and n are subject to the parameters and Different normal distributions.
4. The source-channel joint coding and decoding method according to claim 3, characterized in that: The cross-modal source coding optimization scheme includes mapping the image signal from the pixel domain to the feature domain; For a given image in the pixel domain Using the encoder Encode x as latent variable features Using the decoder Reconstruct the image from the feature space, expressed as, Where z∈R h×w×c .
5. The source-channel joint encoding and decoding method according to claim 4, characterized in that: The learning process of adding noise and denoising the image features in the feature domain; For the forward noise addition process, a forward diffusion process is defined, expressed as, After the forward noise process, the reverse q(x t |x t-1 ) process to obtain the recreated real sample x t ; Design the U-Net model to learn approximate conditional probability.
6. The source-channel joint coding and decoding method according to claim 5, characterized in that: The design of the U-Net model to learn approximate conditional probability includes introducing a Cross Attention module; The Cross Attention is a multi-head attention mechanism that establishes an association between two different input sequences and passes the information of one sequence to the other sequence. Through the Cross Attention module, the U-Net model associates a part of the noisy image feature matrix with specific information of the tactile feature label; Convert the tactile condition data y into the tactile feature expression inside the model τ θ (y) Through the CrossAttention module, it is mapped to the middle layer of the Unet model to interact with the image information, which is expressed as in, represents the process operation before the latent variable feature z enters the Unet network after the forward noise addition process, τ θ (y) represents the corresponding tactile feature information, are three trainable parameter projection matrices; Calculate the attention weights and perform weighted fusion, expressed as, Among them, d represents the dimension of query vector Q and key vector K. Represents the attention weight.
7. The source-channel joint coding and decoding method according to claim 6, characterized in that: The interactive information between tactile features and image features is obtained and fused into the image features. The loss function is expressed as: Among them, ∈ represents the original image features without noise addition, ∈ θ (z t ,t,τ θ (y)) represents the Unet network output; The FID parameter index is used to guide the image generation, which is expressed as: Where Tr represents the trace of the matrix, x and g represent the real picture and the generated image, μ represents the mean, and Σ is the covariance matrix; The total loss function is expressed as, L HVLDM =L C_LDM +L FID (x,g)。 8. The source-channel joint coding and decoding method according to claim 7, characterized in that: The method of using transfer learning to optimize the generalization ability of the model and accelerate the training process of the model includes specifying a first transmission task and a second transmission task for a scenario in which an image signal and a tactile signal are transmitted simultaneously; the first transmission task is to transmit the fusion features of the image signal and the tactile signal; the second transmission task is to transmit only the tactile signal features; Load the pre-trained Transformer encoder C for the first transfer task α (·), decoder Weights; Input feature dataset D required for the second transfer task; Freeze C α (·), The third layer network parameters; retrain the first and second layer network parameters C α1、2 (·), Obtain the first and second layer encoder parameters of the new Transformer; at the same time, cross-modal communication will also face different channel signal-to-noise ratio changes; Input the pre-trained Transformer encoder model C α (·), decoder Different channel signal-to-noise ratio parameters N1; freeze C α (·), The first and second layer network parameters; retrain the third layer network C α3 (·), Get the newly trained network C α3 (·), parameter.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the source-channel joint encoding and decoding method according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the source-channel joint coding and decoding method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Visual-auditory cross-modal object material retrieval method and system
CN108520758A
Image reconstruction method and device for cross-modal communication system
CN113628294A
Image-tactile signal mutual reconstruction method and device
CN114595739A
Audio-visual assisted fine-grained tactile signal reconstruction method
CN115905838A
Cited By
Internet of vehicles channel prediction method in complex traffic scene and related equipment
CN120238221A
Three-dimensional scene automatic generation method and system based on scene semantics and geometric constraints
CN120278911A
Cross-modal semantic transmission method for mobile scene
CN120769298A
Cross-scene hidden danger migration identification method based on large model hidden space mapping
CN121093168A
Cross-modal joint source-channel coding and decoding method adaptable to changeable scenarios
WO2026031415A1
