A cross-modal source-channel joint coding method for coping with variable scenes

Abstract

The application discloses a cross-modal source channel joint coding method for coping with variable scenes, relates to the technical field of cross-modal image signal reconstruction, and comprises the following steps: firstly, a cross-modal channel coding optimization scheme of a Transformer encoder is designed to improve the performance and robustness of a channel encoder and a channel decoder; secondly, a cross-modal source coding optimization scheme of haptic-image generation based on a hidden diffusion model is designed to guide image generation by using haptic information in the case of image signal loss; finally, a transfer learning technology is introduced to reduce the additional training cost of the system in the face of variable cross-modal communication scenes, such as variable channel signal-to-noise ratios and different transmission tasks. The source channel joint coding method provided by the application can solve the problems that the receiver cannot well complete image reconstruction and additional model training costs caused by variable channel environments and scenes in the variable scene of cross-modal communication.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cross-modal image signal reconstruction, in particular to a cross-modal source channel joint coding method for dealing with variable scenes. BACKGROUND

[0002] With the development of wireless communication technology and people's satisfaction in audio, video and traditional multimedia services, further pursuit of multi-sensory immersive experience will gradually become the mainstream in various scenes (remote operation, online education, e-health, digital twin). For example, in the remote education scene, audio and video services integrated with haptic perception and feedback can solve the problem that remote practical teaching cannot be effectively carried out, and improve the learning effect and immersive learning experience of students. In virtual world interaction, haptic perception can bring users a more realistic interactive experience.

[0003] However, due to the significant differences in transmission requirements of audio, video and haptic in multi-modal services, for the former, high throughput needs to be met, and haptic is more sensitive to low delay and high reliability. Traditional audio and video communication and haptic communication cannot meet the demand. Based on this, cross-modal communication emerged. Cross-modal communication makes full use of the potential correlation and semantic public information between modalities, establishes a general cross-modal stream scheduling scheme, and realizes efficient multi-modal stream, processing and recovery with the help of AI methods. It has achieved preliminary results in remote acupuncture teaching, remote operation, remote throat swab and other application scenarios.

[0004] And because the fusion features extracted from the cross-modal signals need to be transmitted, higher requirements are put forward for the transmission system. Recent research shows that the Transformer model has shown strong semantic processing capability, especially in long text semantic transmission. Secondly, the implicit diffusion model is a technology that has attracted much attention, which has achieved outstanding success in text-image generation tasks. However, the existing cross-modal transmission faces the situation of image loss, so a method of haptic-image generation is needed. Finally, different cross-modal communication tasks and environments often result in additional model training costs. In summary, it is necessary to design a source channel joint coding method for improving the image reconstruction quality in variable scenes in cross-modal communication. SUMMARY

[0005] In view of the above problems, the present application is proposed.

[0006] Therefore, the technical problem solved by the present application is that in cross-modal services, there are significant differences in the transmission requirements of audio and video and haptics. Due to the particularity of cross-modal transmission, the situation of loss of a certain modal signal may occur, which puts higher requirements on the transmission system. 3. Different cross-modal communication tasks and scenes often result in additional model training costs.

[0007] To solve the above technical problems, the present application provides the following technical solutions: a cross-modal source channel joint coding method for coping with variable scenes, comprising:

[0008] For the scenario of simultaneous transmission of image signals and haptic signals, the cross-modal source encoder is based on maximizing the system capacity and minimizing the semantic error;

[0009] Considering the scenario of loss of image signals in cross-modal communication, a cross-modal source coding and decoding optimization scheme based on a hidden diffusion model is designed for haptic-image generation;

[0010] For different communication tasks and channel environments, the generalization ability of the model is optimized by using transfer learning, and the training process of the model is accelerated.

[0011] As a preferred scheme of the source channel joint coding method described in the present application, wherein: the cross-modal source encoder comprises a haptic feature extraction module, an image feature extraction module and a modal fusion module;

[0012] The haptic feature extraction module and the image feature extraction module extract features within each modal;

[0013] After obtaining the features of the image and haptic signals respectively, they are spliced together along the channel dimension and input to the modal fusion module for fusion, and the model is trained through a classification task, and the loss function is represented as,

[0014]

[0015] wherein, represents the image feature, the haptic feature and the fusion feature output by the haptic feature extraction network image feature extraction and modal fusion respectively, j y represents the label of the object, respectively, image feature extraction modal fusion and the parameters of the cross-modal source encoder integrated network ;

[0016] Through the cross-modal source decoder at the receiving end, the corresponding image signal needs to be reconstructed from the fusion feature.

[0017] The Wasserstein generative adversarial network is used for image reconstruction, and the discriminator D adopts the design of PatchGAN;

[0018] The generator G generates the fusion feature into an image The discriminator D is responsible for distinguishing the real image v and the generated image The training process is a process of D and G confrontation, and the loss function is represented as,

[0019]

[0020]

[0021] Wherein, ∈~U[0,1]。

[0022] As a preferred scheme of the source channel joint coding method, wherein: the encoder of the Transformer is designed, the encoder contains several identical layers, each layer is composed of a multi-head self-attention sublayer and a position-based feedforward network sublayer, and each sublayer uses a residual connection layer and layer normalization;

[0023] The multi-head self-attention layer adopts a scaled dot product model to calculate the self-attention score, and the same Q, K and V are taken during self-attention calculation; the self-attention scores of different representation subspaces are spliced by projecting Q, K and V through h different linear transformations to obtain the final attention score;

[0024] The position-based feedforward network performs nonlinear transformation on the representation of all positions in the sequence, and passes through a two-layer fully connected layer, the activation function of the first layer is ReLu, and the second layer is a linear activation function, represented as,

[0025] FFN out =max(0,xW1+b1)W2+b2,

[0026] Wherein, x is the fusion feature vector obtained in the last step, W1, W2, b1 and b2 are all learnable parameters in the feedforward network, and FFN out Is the output of the feedforward network;

[0027] The residual connection layer is used only for part of the current difference in the process of updating iteration;

[0028] The normalization layer normalizes the input of each layer of neurons to the same mean and variance, and accelerates the convergence of the model;

[0029] The calculation process of the sublayer is represented as,

[0030] sub_layer_output=LayerNorm(x+sublayer(x)),

[0031] Design a decoder for the Transformer, which adds an encoder-decoder attention sublayer between the multi-head self-attention sublayer and the feedforward network sublayer;

[0032] The channel is modeled as a neural network layer representing the process of encoded fused features transmitted through the channel, considering additive white Gaussian noise channels and Rayleigh slow fading channels;

[0033] In an additive white Gaussian noise channel, the channel's transfer function can be expressed as:

[0034] y = z + n

[0035] Where z represents the channel transmission symbol, and n represents the channel noise, which are independent and identically distributed. σ 2 It is the average noise power;

[0036] The signal-to-noise ratio of the channel is By changing σ 2 To change the signal-to-noise ratio of the channel;

[0037] The transfer function for a Rayleigh slow fading channel is:

[0038] y = hz + n

[0039] Where h is the channel gain. n is the channel noise. h and n respectively obey the parameters as follows: and Different normal distributions.

[0040] As a preferred embodiment of the source-channel joint encoding and decoding method described in this invention, the cross-modal source encoding and decoding optimization scheme includes mapping the image signal from the pixel domain to the feature domain;

[0041] For a given image in the pixel domain Using encoder Encode x as latent variable features Using decoder Reconstructing an image from the feature space, denoted as,

[0042]

[0043] Where, z∈R h×w×c .

[0044] As a preferred embodiment of the source-channel joint encoding and decoding method described in this invention, the method includes: a learning process of adding and removing noise to image features in the feature domain;

[0045] For the forward noise addition process, a forward diffusion process is defined as follows:

[0046]

[0047] After the forward noise addition process, the q(x) is reversed. t |x t-1 The process yields a recreated real sample x t ;

[0048] Design the U-Net model to learn approximate conditional probabilities.

[0049] As a preferred embodiment of the source-channel joint encoding and decoding method described in this invention, the design of the U-Net model to learn approximate conditional probabilities includes the introduction of a Cross Attention module;

[0050] The Cross Attention is a multi-head attention mechanism that establishes a connection between two different input sequences and passes information from one sequence to the other. Through the Cross Attention module, the U-Net model associates a part of the noisy image feature matrix with specific information from the tactile feature label.

[0051] Transform the tactile conditional data y into an internal tactile feature representation within the model. τ θ (y) is mapped to the intermediate layer of the UNet model through the Cross Attention module and fused with image information for interaction, as shown below.

[0052]

[0053] in, τ represents the process operation of the latent variable feature z before it enters the Unet network after the forward noise addition process. θ (y) represents the corresponding tactile feature information. These are three trainable parameter projection matrices;

[0054] Calculate the attention weights and perform weighted fusion, represented as follows:

[0055]

[0056] Where d represents the dimensions of the query vector Q and the key vector K. This represents attention weight.

[0057] As a preferred scheme of the source channel joint coding method described in the application, wherein: the interaction information between the tactile feature and the image feature is obtained and fused into the image feature, and the loss function is represented as,

[0058]

[0059] Wherein, ∈ represents the original image feature without adding noise, and ε θ (z t ,t,τ θ (y)) represents the Unet network output;

[0060] The FID parameter index is used to guide and constrain the image generation, and is represented as,

[0061]

[0062] Wherein, Tr represents the trace of the matrix, x and g represent the real picture and the generated image, μ represents the mean, and Σ is the covariance matrix;

[0063] The total loss function is represented as,

[0064] L HVLDM =L C_LDM +L FID (x,g)。

[0065] As a preferred scheme of the source channel joint coding method described in the application, wherein: the generalization ability of the model is optimized by using transfer learning, and the training process of the model is accelerated, which includes defining a first transmission task and a second transmission task for the scene where the image signal and the tactile signal are transmitted simultaneously; the first transmission task is to transmit the fusion features of the image signal and the tactile signal; the second transmission task is to transmit only the tactile signal features;

[0066] Load the pre-trained Transformer encoder C α (·) and decoder weight; input the feature data set D required for the second transmission task; freeze the third layer network parameters of C α (·), ; retrain the first layer and the second layer network parameters C α1 ,2(·), Obtain new Transformer first and second layer encoder parameters; meanwhile, different channel signal-to-noise ratios will be encountered in cross-modal communication;

[0067] Input the pre-trained Transformer encoder model C α (·) and decoder Different channel signal-to-noise ratio parameters N1; frozen C α (·), First and second layer network parameters; retrain third layer network C α3 (·), Obtain a newly trained network C α3 (·), Parameters.

[0068] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the source channel joint coding method as described above when executing the computer program.

[0069] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the source channel joint coding method as described above.

[0070] The source channel joint coding method provided by the present application reduces power consumption in the network, measures load balancing between the cloud network and the server (host) using the fitness function, converts the load balancing problem into an optimization problem, and can be applied to processing resource optimization. The results of the particle swarm are used as the initial population of the genetic algorithm, and the present application achieves better results in terms of execution cost, load balancing and completion time. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0072] Figure 1 A cross-modal source channel joint coding method for coping with variable scenarios provided by an embodiment of the present application.

[0073] Figure 2 A three-layer Transformer encoder structure diagram in a cross-modal source channel joint coding method for coping with variable scenarios provided by an embodiment of the present application.

[0074] Figure 3 A cross-modal source channel joint coding method based on a diffusion model provided by an embodiment of the present application.

[0075] Figure 4 A U-Net denoising model structure diagram in a cross-modal source channel joint coding method for coping with variable scenarios provided by an embodiment of the present application.

[0076] Figure 5 A source channel joint coding method based on transfer learning is provided for an embodiment of the present application.

[0077] Figure 6 A cross-modal source channel joint coding method for dealing with variable scenes is provided for an embodiment of the present application.

[0078] Figure 7 A source channel joint coding method based on diffusion model is provided for an embodiment of the present application.

[0079] Figure 8 A source channel joint coding method based on diffusion model is provided for an embodiment of the present application.

[0080] Figure 9 A source channel joint coding method based on transfer learning is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0082] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0083] Embodiment 1

[0084] Reference Figures 1-5 For an embodiment of the present application, a cross-modal source channel joint coding method for dealing with variable scenes is provided, comprising:

[0085] Step 1, as shown in Figure 1 For the scene of simultaneous transmission of image signals and tactile signals, a cross-modal channel coding and decoding communication system based on Transformer is designed. The main goal of this system is to maximize the system capacity and minimize semantic errors, rather than just focusing on bit or symbol errors in traditional communication.

[0086] Step 1-1, design cross-modal source encoder including tactile feature extraction image feature extraction and modal fusion Three modules. Where v represents the image signal, h represents the tactile signal, and represents the module parameters. The main purpose of the tactile feature extraction module and the image feature extraction module is to extract features within each modality. After obtaining the features of the image and tactile signals respectively, they are spliced together along the channel dimension and input to the modal fusion module for fusion. All three modules are trained with a classification task. The output of the fully connected layer in the last layer is (None, 6), and the output of the second last layer is (None, x), where the value of x is selected as 640 after parameter experiment, and the loss function is designed as:

[0087]

[0088] where, represents the image feature, tactile feature and fusion feature output respectively by the tactile feature extraction network image feature extraction and modal fusion . y j represents the label of the object. represent the parameters of the tactile feature extraction network image feature extraction modal fusion and cross-modal source encoder integrated network respectively.

[0089] Step 1-2, design cross-modal source decoder, at the receiving end, it is necessary to reconstruct the corresponding image signal from the fusion feature. The method of Wasserstein generative adversarial network (WGAN) is used to realize image reconstruction. The discriminator D adopts the design of PatchGAN. The generator G generates the fusion feature into an image The discriminator D is responsible for distinguishing between the real image v and the generated image The generator G needs to produce more realistic pictures, while the discriminator D needs to have stronger ability to distinguish true and false, therefore, the whole training process is a process of D and G against. Here the loss function is defined as:

[0090]

[0091] where, ∈~U[0,1].

[0092] Step 1-3, as Figure 2The Transformer-based cross-modal channel encoder and decoder are designed. Here, the main structure is the encoder-decoder structure, and each layer decoder takes the output of the final encoder as input. The overall architecture of the Transformer model is designed to consist of 3 layers of encoder and 3 layers of decoder. First, the encoder of the Transformer is designed, which contains several identical layers, each of which consists of a multi-head self-attention sublayer and a position-based feed-forward network sublayer, and each sublayer uses a residual connection and a normalization layer. The self-attention unit of each layer is selected as 128, and the activation function is selected as the linear function. The scaled dot-product model is used to calculate the self-attention score in the multi-head self-attention layer, and the same Q, K, and V are used for self-attention calculation. Then, the Q, K, and V are projected through h different linear transformations, and the self-attention scores of different representation subspaces are spliced to obtain the final attention score. The position-based feed-forward network performs nonlinear transformation on the representation of all positions in the sequence, increasing the representation ability of the model. The feed-forward network (which performs nonlinear transformation) is a two-layer fully connected layer, and the activation function of the first layer is ReLu, and the second layer is a linear activation function:

[0093] FFN out = max(0, xW1 + b1)W2 + b2,

[0094] where x is the fusion feature vector obtained in the previous step, W1, W2, b1, and b2 are learnable parameters in the feed-forward network, and FFNout represents the feed-forward network output. The residual connection layer makes the network focus only on the current difference in the process of updating iteration, and is used to handle problems such as grid degradation and gradient disappearance in the training process of multi-layer network; the normalization layer normalizes the input of each layer of neurons to the same mean and variance to speed up model convergence. The entire calculation process of the two sub-layers can be represented as:

[0095] sub_layer_output = LayerNorm(x + sublayer(x)),

[0096] Then the decoder of the Transformer is designed, which has a similar structure with the encoder, but the decoder adds an encoder-decoder attention sublayer between the multi-head self-attention sublayer and the feed-forward network sublayer. The activation function of the encoder-decoder attention sublayer is Linear function. The two kinds of attention sublayers have the same structure, but the sources of the query, key and value are different. Specifically, the query of the encoder-decoder attention sublayer is the output of the previous layer of the decoder, and the key and value are generated by decomposing the output of the entire encoder, while the multi-head self-attention sublayer gets the query, key and value from the previous layer of the decoder. In addition, the layer introduces a mask function to preserve its self-recurrent property, ensuring that the prediction only depends on the generated output tokens, ensuring consistency between the training and inference stages. The output of the decoder is a float vector, which is input into a linear layer and then a softmax layer to generate the final output word. Among them, the linear layer uses a fully connected neural network to map the output vector of the decoder to a larger linear space. Then, the softmax layer converts these vectors into probabilities, and finally sorts them according to the probability value, locates the dimension with the highest probability, and generates the corresponding feature, which is the output of this time step. The method uses MSE loss to train the model.

[0097] Step 1-4, in order to represent the process of transmitting the fused features after encoding, the channel is modeled as a series of untrainable but differentiable neural network layers. Two commonly used channel models are considered: additive white Gaussian noise (AWGN) channel and Rayleigh slow fading (Rayleigh) channel. In the AWGN channel, the transmission function of the channel can be represented as: y = z + n. z is the channel transmission symbol, n is the channel noise, and σ 2 is the average noise power. The signal-to-noise ratio of the channel is By changing σ 2 , the signal-to-noise ratio of the channel is changed. The transmission function of the Rayleigh slow fading channel is: y = hz + n. h is the channel gain, n is the channel noise, h and n respectively obey different normal distributions with parameters and .

[0098] Step 2, as shown in Figure 3 , considering the scenario of image signal loss in cross-modal communication, a cross-modal source coding and decoding optimization scheme for tactile-image generation based on the implicit diffusion model is designed. The method is as follows: 1. Design a feature extraction module to map the picture from the pixel domain to the feature domain; 2. In the feature domain, the obtained picture features are subjected to a noise adding and denoising learning process; 3. In the conditional domain, the tactile features are trained to obtain the tactile features, and the obtained tactile features are used to guide the generation of the corresponding picture. The model structure is asFigure 2 as shown.

[0099] Step 2 considers the scenario of image signal loss under cross-modal communication, and designs a cross-modal source codec optimization scheme for haptic-image generation based on the implicit diffusion model. The method is as follows: 1. Design a feature extraction module to map the picture from the pixel domain to the feature domain; 2. In the feature domain, the obtained picture features are subjected to a learning process of adding noise and denoising; 3. In the conditional domain, the haptic features are trained, and the obtained haptic features are used to guide the generation of the corresponding pictures.

[0100] Step 2-1, map the image signal from the pixel domain to the feature domain. For a given image in the pixel domain Use the encoder Encode x into latent variable features: Use the decoder The reconstructed image from the feature space can be obtained as follows:

[0101]

[0102] where z∈R h×w×c The specific encoder and decoder structure can refer to the source encoder structure and source decoder structure in step 1.

[0103] Step 2-2, the learning process of adding noise and denoising in the feature domain for the image features obtained in the previous step. For the forward noise adding process, the purpose is to make all samples eventually tend to isotropic distribution by continuously adding noise of the same distribution to the samples. Specifically, it first defines a forward diffusion process, in which a small amount of Gaussian noise is added to the sample as shown in the following formula, and the step size of adding noise is controlled by , thereby generating a series of noise samples x1,…,x T .

[0104]

[0105] After the forward noise adding process, it is necessary to reverse the q(x t |x t-1 ) process to obtain the recreated real sample x t . However, since this process traverses the entire data set, it is impossible to obtain the real sample by simple estimation. Therefore, a U-Net model is designed to learn the approximate conditional probability.

[0106] Step 2-3, as Figure 4As shown, the U-Net network structure proposed by the application is similar to the standard U-Net, but the innovation lies in the introduction of the Spatial Transformer (Cross Attention) module. Therefore, this module is highlighted here. Cross Attention is a multi-head attention mechanism that can establish a connection between two different input sequences and transfer information from one sequence to another. The Cross Attention module is used to control the fusion interaction of tactile and image information, that is, through this module, the U-Net can associate a part of the noisy image feature matrix with specific information of the tactile feature label.

[0107] In order to preprocess the conditional tactile information y, the tactile feature extraction module τ θ is used in step 1, which functions to convert the tactile condition data y into tactile feature expression inside the model Then τ θ (y) is mapped to the middle layer of the Unet model through the Cross Attention module to interact with the image information, thereby realizing the effect of guiding image generation with tactile information. The above process can be represented as follows: first, calculate the query vector Q, the key vector K, and the value vector V through three independent fully connected layers respectively, as shown in the formula:

[0108]

[0109] wherein, represents the process operation of the hidden variable feature z after the forward noise adding process before entering the Unet network. τ θ (y) represents the corresponding tactile feature information, are three trainable parameter projection matrices. Secondly, calculate the attention weight and perform weighted fusion as shown in formula (4.7):

[0110]

[0111] wherein d represents the dimension of the query vector Q and the key vector K, represents the attention weight. In this way, the interaction information between the tactile feature and the image feature is obtained and fused into the image feature.

[0112] Step 2-4 based on the above tactile-image condition generation process, the loss function can be described as follows:

[0113]

[0114] wherein ∈ represents the original image feature without noise adding, and ε θ (z tt, τ θ (y)) represents the output of the Unet network.

[0115] In addition, the above loss function is based on the mathematical derivation of the model principle, while in the haptic-visual generation task proposed in this paper, the FID (Frechet Inception Distance) parameter index can be used to guide and constrain image generation. FID is an index for evaluating the difference between the generated model and the real data distribution. Then in this paper it can be represented as:

[0116]

[0117] In the formula, Tr represents the trace of the matrix. x and g represent the real picture and the generated image, μ represents the mean, and Σ is the covariance matrix. A lower FID indicates that the distributions of the two images are closer, i.e. the performance of the model generated image is better.

[0118] The total loss function of the model can be represented as

[0119] L HVLDM =L C_LDM +L FID (x,g)。

[0120] Step 3: For different communication tasks and environments, the method of transfer learning is used to ensure the generalization ability of the model and to speed up the training process of the model and reduce the training cost of the model. As shown in Figure 5 .

[0121] Step 3-1, according to the assumed known task K0 and channel condition N0, load the pre-trained channel encoder and channel decoder. For applications with different background knowledge, the present application only needs to retrain part of the Transformer encoder and decoder layers. Specifically: according to the two transmission scenarios faced by the present application, task 1: demand transmission of image signal and haptic signal fusion features; task 2: only haptic signal features are transmitted.

[0122] Step 3-11, load the pre-trained Transformer encoder C α (·) and decoder weights for task 1.

[0123] Step 3-12, input the feature data set D required for task 2.

[0124] Step 3-13, freeze the third layer network parameters of C α (·), .

[0125] Step 3-14, retrain the first layer and second layer network parameters Cα1、2 (·),

[0126] Step 3-15, obtaining new Transformer first and second layer encoder parameters.

[0127] Step 3-2, the cross-modal communication will also face different channel SNR changes.

[0128] Step 3-21, input: pre-trained Transformer encoder model C α (·) decoder Different channel SNR parameters N1.

[0129] Step 3-22, freeze C α (·), The first and second layer network parameters of C.

[0130] Step 3-23, retrain the third layer network C α3 (·),

[0131] Step 3-24, obtain the newly trained network C α3 (·), Parameters.

[0132] In the above process, the loss function of the training is set as L HVLDM , and the Adam optimization algorithm is used for optimization. The Adam optimization algorithm is based on the gradient descent method and adds adaptive adjustment of the learning rate. Its expression is as follows:

[0133] m t = β1m t-1 + (1-β1)g t

[0134]

[0135] Where g t is the gradient of the parameter, β1and β2are the decay coefficients of the two exponential weighted averages, and m and v are the moving averages of the gradient bias correction. θ t and θ t+1 are the network parameters before and after updating, respectively. The network parameters here refer to the first, second and third layer network parameters in the Transformer encoder and decoder.

[0136] Embodiment 2, one embodiment of the present application, which is different from the previous embodiment:

[0137] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0138] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with these instructions execution systems, apparatuses, or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport programs for use by an instruction execution system, apparatus, or device, or in conjunction with these instruction execution systems, apparatuses, or devices.

[0139] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting, or otherwise processing, if necessary, in other suitable ways to be electronically obtained, and then stored in the computer memory.

[0140] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, any of the following technologies known in the art or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0141] Embodiment 3, reference Figures 6-9 For an embodiment of the application, a cross-modal source channel joint coding method is provided to cope with variable scenarios. In order to verify the beneficial effects of the application, scientific demonstration is carried out through simulation experiments.

[0142] The present embodiment uses the AU dataset data set, which contains multi-modal information of 63 objects, including images, kinesthetic and tactile data (audio vibration). These objects are mainly toys and household items. Some objects have similar shapes and colors on the image, while the materials and textures are similar but the colors are different. These objects reflect the different effects of images and tactile on perception.

[0143] The present application carries out experiments on step 1, step 2 and step 3 modules respectively.

[0144] Step 1 related comparative experiment adopts neural network based source channel joint coding and typical source channel separation coding method. Deep neural network based source channel joint coding (JSCC): the network is composed of bidirectional long short term memory (BLSTM) layer. It is marked as JSCC-LSTM in the simulation diagram, and JSCC-CNN. Traditional method, respectively using the following technology: channel coding: Turbo coding and Reed-Solomon (RS) coding. The turbo decoding method used in the present application is 5 times iteration log-MAP algorithm.

[0145] Step 2 uses comparative experiment and ablation experiment, and the comparative experiment selects the following two methods:

[0146] Style GAN: generates intermediate latent space through mapping network, which reduces the entanglement of input latent space, and at the same time puts forward two new indicators to quantify the coupling of latent space: perceptual path length and linear separability. Finally, it achieves excellent success in face image generation task.

[0147] Projected GAN: Generative adversarial networks can generate high-quality images, but are challenging to train. They require careful regularization, large amounts of computation, and expensive hyperparameter sweeps. This model makes significant progress on these issues by projecting generated and real samples into a fixed, pre-trained feature space. Since it was found that discriminators could not fully exploit the deeper features of the pre-trained model, a more effective strategy was proposed that could mix features across channels and resolutions. Projected GAN can improve image quality, sampling efficiency, and convergence speed.

[0148] The present invention: The method of the present embodiment.

[0149] The following is an introduction to the measurement indicators used in the present invention: IS indicator is an indicator for evaluating the quality of generated images, which combines the diversity and authenticity of images. Inception Score is an indicator of KL divergence, which is the exponential mean of the KL divergence of the conditional distribution of each image and the overall distribution. The mathematical expression is as follows:

[0150] IS = exp(E x [KL(p(y|x) p(y))],

[0151] where p(y|x) is the conditional distribution, representing the probability distribution of class y given image x; p(y) is the overall distribution, representing the probability distribution of class y of all images; E x represents the expectation over all images.

[0152] FID is an indicator for evaluating the difference between the generated model and the real data distribution. So in the present invention it can be represented as:

[0153]

[0154] In the formula, Tr represents the trace of the matrix. x and g represent real pictures and generated pictures, μ represents the mean, and Σ is the covariance matrix.

[0155] In addition, PSNR is also a commonly used objective evaluation indicator for images, and its mathematical expression is defined as:

[0156]

[0157] where MSE is the mean square error of the real image and the generated image. The larger the PSNR is, the better.

[0158] Table 1 is the experimental results of the present invention

[0159]

[0160] From Table 1 andFigure 6 It can be seen that compared with the above-mentioned most advanced method, the method proposed by us has obvious advantages. In Figure 6 In the traditional baseline shown in the figure, when turbo coding is used for channel coding, on the AWGN channel, when the signal-to-noise ratio is greater than 12dB, the performance of the traditional method is better than that of the deep neural network-based method. All deep learning-supported methods are more competitive under low signal-to-noise ratio conditions. And on the sharp fading channel, the performance of the traditional scheme under various conditions is not as good as that of the deep learning-based method. The method of the application achieves the best performance under most conditions. This is also in the environment of complex multiplicative noise of sharp fading channel, the distortion of the traditional method is higher, which cannot accurately decode the corresponding feature vector, thereby causing the performance decline of the final signal source decoding.

[0161] As Figure 7 shown, the results of the application are better than the methods compared, including Style GAN and Projected GAN. The method of the application trains the diffusion model in a fixed latent space, avoiding the trade-off problem between reconstruction quality and learning latent space prior. The experimental results are significantly better than the two methods in all indicators. Through the multi-head attention mechanism, the tactile modal information is successfully mapped into the latent feature representation of the UNet network, which can successfully guide the generation of images.

[0162] As Figure 8 shown, HVLDM without Lc_LDM represents the case of unconditional generation, and the model's ability to generate pictures corresponding to the input tactile information has been greatly reduced. In five experiments, the model cannot accurately recover the original picture in most cases. This shows that in this structure, the tactile information in the conditional field plays an important guiding role in picture generation.

[0163] HVLDM without Lfid represents that the loss function of the model is only composed of L C-LDM The experimental results show that the images generated by the model after lacking Lfid are not as good as the complete model in IS and FID indicators, that is, the similarity of the generated pictures lags behind the HVLDM model. This also shows that the loss function Lfid plays a function of constraining image generation to approximate the real image category on this task.

[0164] Figure 9The training efficiency and performance under different background knowledge are shown, in which the model will be trained and retrained under new background knowledge using the same channel (AWGN). The model has the same structure and is retrained using the same parameters in each scenario. In figures (a) (c), the number of epochs is reduced from 15 to 5, and from 25 to 10, respectively, to reach convergence. This shows that the transfer learning assisted TransJSCC can help the transceiver to adapt to the new communication environment requirements. As can be seen from figures (b) (d), the use of a pre-trained model can speed up the training process in sharp channels. At the same time, due to the smaller number of epochs required in the retraining process, the required complexity is significantly reduced.

[0165] It should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A cross-modal source-channel joint coding method for coping with variable scenes, characterized in that, The application relates to a cross-modal source encoder, a cross-modal source decoder and a cross-modal source coding and decoding optimization scheme. The application maximizes system capacity and minimizes semantic errors based on the cross-modal source encoder for the simultaneous transmission of image signals and tactile signals. The application designs a cross-modal source coding and decoding optimization scheme based on a hidden diffusion model for the image signal loss in cross-modal communication. The application uses transfer learning to optimize the generalization ability of the model and accelerate the training process of the model for different communication tasks and channel environments. The cross-modal source encoder comprises a tactile feature extraction module, an image feature extraction module and a modal fusion module. The tactile feature extraction module and the image feature extraction module extract features in each modal. After the features of the image and tactile signals are obtained, the features are spliced along the channel dimension and input to the modal fusion module for fusion. wherein, denote the parameters of the tactile feature extraction network image feature extraction and modal fusion the output image feature, tactile feature and fused feature, y j denote the label of the object, denote the parameters of the tactile feature extraction network image feature extraction modal fusion and the cross-modal source encoder integration network respectively. The cross-modal source decoder is used at the receiving end to reconstruct the corresponding image signal from the fused features. The Wasserstein generative adversarial network is used to realize image reconstruction, and the discriminator D adopts the PatchGAN design. The generator G generates fused features into images The discriminator D is responsible for distinguishing real images v and generated images The training process is a process of D and G against each other, and the loss function is represented as wherein The cross-modal source coding and decoding optimization scheme comprises mapping the image signal from the pixel domain to the feature domain. For a given image in the pixel domain With the encoder Encode x as a latent variable feature With the decoder Reconstruct the image from the feature space, denoted as where z e R h×w×c .

2. The source-channel joint coding method of claim 1, wherein: The encoder of the Transformer is designed, and the encoder comprises a plurality of identical layers, each layer is composed of a multi-head self-attention sublayer and a position-based feedforward network sublayer, and each sublayer uses a residual connection layer and layer normalization. The multi-head self-attention layer adopts a scaled dot product model to calculate the self-attention score, and the self-attention calculation takes Q, K and V as the same; the self-attention scores of different representation subspaces are spliced to obtain the final attention score through h different linear transformations on Q, K and V. The position-based feedforward network performs nonlinear transformation on the representation of all positions in the sequence, and passes through a two-layer fully connected layer, the activation function of the first layer is ReLu, and the second layer is a linear activation function, and is expressed as, FFN out = max(0, xW1 + b1)W2 + b2, wherein x is the fusion feature vector obtained in the previous step, W1, W2, b1 and b2 are all learnable parameters in the feedforward network, FFN out is the output of the feedforward network. The residual connection layer is used for only the part of the current difference in the process of updating iteration. The normalization layer normalizes the input of each layer of neurons to the same mean and variance, and accelerates the convergence of the model. The calculation process of the sublayer is expressed as, sub_layer_output=LayerNorm(x+sublayer(x)), The decoder of the Transformer is designed, and the encoder-decoder attention sublayer is added between the multi-head self-attention sublayer and the feedforward network sublayer. The channel is modeled as a neural network layer to represent the transmission process of the coded fused features through the channel, and additive white Gaussian noise channels and Rayleigh slow fading channels are considered. Under the additive white Gaussian noise channel, the transmission function of the channel can be expressed as, y=z+n where z is the channel transmission symbol, n is the channel noise, which is independent and identically distributed, and σ 2 is the average noise power; The signal-to-noise ratio of the channel is By changing σ 2 to change the signal-to-noise ratio of the channel; The transmission function of the Rayleigh slow fading channel is, y=hz+n where h is a channel gain, n is a channel noise, h and n are subject to different normal distributions with parameters and respectively.

3. The source-channel joint coding method of claim 2, wherein: The learning process of image feature noise adding and noise removing in the feature domain is performed. For the forward noise adding process, a forward diffusion process is defined, and is expressed as, After the forward noise addition process, the q(x) is reversed. t |x t-1 The process yields a recreated real sample x t ; The U-Net model is designed to learn the approximate conditional probability.

4. The source-channel joint coding method of claim 3, wherein: The design of the U-Net model learning the approximate conditional probability comprises introducing a Cross Attention module; The Cross Attention is a multi-head attention mechanism that establishes the association between two different input sequences and transmits the information of one sequence to the other sequence, and through the Cross Attention module, the U-Net model associates a part of the feature matrix of the noisy image with specific information of the tactile feature label; transforming haptic condition data y into a haptic feature representation within the model transforming τ θ (y) is mapped to the middle layer of the Unet model by the CrossAttention module to interact with the image information, denoted as wherein, represents the process operation before the hidden variable feature z enters the Unet network after the forward noise adding process, τ θ (y) represents the corresponding tactile feature information, are three trainable parameter projection matrices; The attention weight is calculated and weighted fusion is performed, and is expressed as, where d represents the dimension of the query vector Q, the key vector K, represents the attention weight.

5. The source-channel joint coding method of claim 4, wherein: The interaction information between the tactile feature and the image feature is obtained and fused into the image feature, and a loss function is expressed as, Wherein, ∈ represents the original image feature without adding noise, ε θ (z t ,t,τ θ (y)) represents the Unet network output; An FID parameter index is used to guide and constrain the image generation, and is expressed as, Wherein, Tr represents the trace of a matrix, x and g represent the real picture and the generated image, μ represents the mean, and Σ is the covariance matrix; A total loss function is expressed as, L HVLDM = L C_LDM + L FID (x, g).

6. The source-channel joint coding method of claim 5, wherein: The method for optimizing the generalization ability of the model by using the transfer learning and accelerating the training process of the model comprises defining a first transmission task and a second transmission task for a scene in which the image signal and the tactile signal are transmitted simultaneously; the first transmission task is to transmit the fusion feature of the image signal and the tactile signal; and the second transmission task is to transmit only the tactile signal feature; loading pre-trained transformer encoder C for first transmission task α (·), decoder weights; input feature dataset D required for second transmission task; freeze C α (·), third layer network parameters; retrain first and second layer network parameters C α1、2 (·), obtain new transformer first and second layer encoder parameters; meanwhile, cross-modal communication will also face different channel SNR changes; input a pre-trained transformer encoder model C α (·) decoder different channel signal-to-noise ratio parameters N1; freeze C α (·), first and second layer network parameters; retrain third layer network C α3 (·), obtain a newly trained network C α3 (·), parameters. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6. The processor executes the computer program to implement the steps of the source channel joint coding method in any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the source channel joint coding method in any one of claims 1 to 6.

Images