Cross-modal joint source-channel coding and decoding method adaptable to changeable scenarios

Abstract

The present invention relates to the technical field of cross-modal image signal reconstruction. Disclosed is a cross-modal joint source-channel coding and decoding method adaptable to changeable scenarios. The method comprises: first designing a Transformer encoder-based cross-modal channel coding and decoding optimization solution, so as to achieve the performance improvement and robustness of a channel encoder and a channel decoder; then designing a cross-modal source coding and decoding optimization solution for haptic-to-image generation based on a latent diffusion model, so that under an image signal loss scenario, haptic information is used to guide image generation; and finally, incorporating transfer learning technology, so as to reduce additional training costs caused by a system facing changeable cross-modal communication scenarios such as a changeable channel signal-to-noise ratio and different transmission tasks. Under cross-modal changeable communication scenarios, the joint source-channel coding and decoding method provided in the present invention can solve the problems of the inability of a receiving end to well complete image reconstruction, and additional model training costs caused by changeable channel environments and scenarios.

Description

A cross-modal source channel joint coding method for coping with variable scenes TECHNICAL FIELD

[0001] The present application relates to the technical field of cross-modal image signal reconstruction, in particular to a cross-modal source channel joint coding method for coping with variable scenes. BACKGROUND

[0002] With the development of wireless communication technology and people's satisfaction in audio, video and traditional multimedia services, further pursuit of multi-sensory immersive experience will gradually become the mainstream in various scenes (remote operation, online education, e-health, digital twin). For example, in the remote education scene, audio and video services integrated with haptic perception and feedback can solve the problem that remote practical teaching cannot be effectively carried out, and improve the learning effect and immersive learning experience of students. In virtual world interaction, haptic perception can bring users a more realistic interactive experience.

[0003] However, due to the significant differences in transmission requirements of audio, video and haptic in multi-modal services, for the former, high throughput needs to be met, and haptic is more sensitive to low delay and high reliability. Traditional audio and video communication and haptic communication cannot meet the demand. Based on this, cross-modal communication emerged. Cross-modal communication makes full use of the potential correlation and semantic public information between modalities, establishes a general cross-modal stream scheduling scheme, and realizes efficient multi-modal stream, processing and recovery through AI methods. It has achieved preliminary results in remote acupuncture teaching, remote operation, remote throat swab and other application scenarios.

[0004] And because the fusion features extracted from the cross-modal signals need to be transmitted, higher requirements are put forward for the transmission system. Recent research shows that the Transformer model has shown strong semantic processing capability, especially in long text semantic transmission. Secondly, the implicit diffusion model is a technology that has attracted much attention, which has achieved outstanding success in text-image generation tasks. However, the existing cross-modal transmission faces the situation of image loss, so a method of haptic-image generation is needed. Finally, different cross-modal communication tasks and environments often result in additional model training costs. In summary, it is necessary to design a cross-modal communication source channel joint coding method for improving the image reconstruction quality in variable scenes. SUMMARY

[0005] In view of the above problems, the present application is proposed.

[0006] Therefore, the technical problem solved by the present application is that in cross-modal services, there are significant differences in the transmission requirements of audio and video and haptics. Due to the particularity of cross-modal transmission, the situation of loss of a certain modal signal may occur, which puts higher requirements on the transmission system. 3. Different cross-modal communication tasks and scenes often result in additional model training costs.

[0007] To solve the above technical problems, the present application provides the following technical solutions: a cross-modal source channel joint coding method for coping with variable scenes, comprising:

[0008] For the scenario of simultaneous transmission of image signals and haptic signals, the cross-modal source encoder is based on maximizing the system capacity and minimizing the semantic error.

[0009] Considering the scenario of image signal loss in cross-modal communication, a cross-modal source coding and decoding optimization scheme based on implicit diffusion model haptic-image generation is designed.

[0010] For different communication tasks and channel environments, transfer learning is used to optimize the generalization ability of the model and accelerate the training process of the model.

[0011] As a preferred scheme of the cross-modal source channel joint coding method for coping with variable scenes according to the present application, wherein: the cross-modal source encoder comprises a haptic feature extraction module, an image feature extraction module and a modal fusion module.

[0012] The haptic feature extraction module and the image feature extraction module extract features within each modal.

[0013] When the features of the image and haptic signals are obtained respectively, they are spliced together along the channel dimension and input to the modal fusion module for fusion, and the model is trained through a classification task, and the loss function is represented as:

[0014] wherein, y represents the image features, haptic features and fusion features output by the haptic feature extraction network S v , the image feature extraction S h and the modal fusion S f , y j represents the label of the object, S respectively represent the parameters of the haptic feature extraction network S v , the image feature extraction S h , the modal fusion S f and the cross-modal source encoder integrated network S en .

[0015] Through the cross-modal source decoder at the receiving end, the corresponding image signal needs to be reconstructed from the fusion features.

[0016] The Wasserstein generative adversarial network is used for image reconstruction, and the discriminator D adopts the design of PatchGAN.

[0017] The generator G generates the fusion features into images The discriminator D is responsible for distinguishing the real image v and the generated image The training process is a process of D and G confrontation, and the loss function is represented as:

[0018] Wherein,

[0019] As a preferred scheme of the cross-modal source channel joint coding method for coping with variable scenes, the encoder of the Transformer is designed, the encoder comprises a plurality of same layers, each layer is composed of a multi-head self-attention sublayer and a position-based feedforward network sublayer, and each sublayer uses a residual connection layer and layer normalization.

[0020] The multi-head self-attention layer adopts a scaled dot-product model to calculate the self-attention score, and the same Q, K and V are taken during self-attention calculation; the self-attention scores of different representation subspaces are spliced by projecting Q, K and V through h different linear transformations to obtain the final attention score.

[0021] The position-based feedforward network performs nonlinear transformation on the representation of all positions in the sequence, and passes through a two-layer fully connected layer, the activation function of the first layer is ReLu, and the second layer is a linear activation function, which is represented as: FFN out = max (0, xW1+ b1) W2+ b2,

[0022] Wherein, x is the fusion feature vector obtained in the previous step, W1, W2, b1 and b2 are all learnable parameters in the feedforward network, and FFN out Is the output of the feedforward network.

[0023] The residual connection layer is used for only part of the current difference in the process of updating iteration.

[0024] The normalization layer normalizes the input of each layer of neurons to the same mean and variance, and accelerates the convergence of the model.

[0025] The calculation process of the sublayer is represented as: sub_layer_output = LayerNorm (x + sublayer (x)),

[0026] The decoder of the Transformer is designed, and the encoder-decoder attention sublayer is added between the multi-head self-attention sublayer and the feedforward network sublayer.

[0027] The channel is modeled as a neural network layer to represent the process of transmitting the fused features after encoding through the channel, considering an additive white Gaussian noise channel and a Rayleigh slow fading channel.

[0028] Under the additive white Gaussian noise channel, the transmission function of the channel can be expressed as: y = z + n

[0029] Where z is the channel transmission symbol, n is the channel noise, is independently and identically distributed, and n ~ N(0, σ 2 ), σ 2 is the average noise power.

[0030] The signal-to-noise ratio of the channel is By changing σ 2 , the signal-to-noise ratio of the channel is changed.

[0031] The transmission function of the Rayleigh slow fading channel is: y = hz + n

[0032] Where h is the channel gain, n is the channel noise, h and n respectively obey different normal distributions with parameters and .

[0033] As a preferred scheme of the cross-modal source channel joint coding method for dealing with variable scenes, the cross-modal source coding optimization scheme includes mapping the image signal from the pixel domain to the feature domain.

[0034] For a given image x in the pixel domain, Encode x into latent variable feature z = ε(x) using encoder ε, and decode z into image Reconstruct the image from the feature space, denoted as:

[0035] Where,

[0036] As a preferred scheme of the cross-modal source channel joint coding method for dealing with variable scenes, the learning process of adding noise and denoising to the image features in the feature domain;

[0037] For the forward noise adding process, define a forward diffusion process, denoted as:

[0038] After the forward noise adding process, reverse q(xt |x t-1 The process yields a recreated real sample x t ;

[0039] Design the U-Net model to learn approximate conditional probabilities.

[0040] As a preferred embodiment of the cross-modal source-channel joint encoding and decoding method for dealing with variable scenarios described in this invention, the design of the U-Net model to learn approximate conditional probabilities includes the introduction of a Cross Attention module.

[0041] The Cross Attention is a multi-head attention mechanism that establishes a connection between two different input sequences and passes information from one sequence to the other. Through the Cross Attention module, the U-Net model associates a portion of the noisy image feature matrix with specific information from the tactile feature labels.

[0042] Transform the tactile conditional data y into an internal tactile feature representation within the model. τ θ (y) is mapped to the intermediate layer of the Unet model through the Cross Attention module and fused with image information for interaction, as shown below:

[0043] in, τ represents the process operation of the latent variable feature z before it enters the Unet network after the forward noise addition process. θ (y) represents the corresponding tactile feature information. These are three trainable parameter projection matrices;

[0044] Calculate the attention weights and perform weighted fusion, as follows:

[0045] Where d represents the dimensions of the query vector Q and the key vector K. This represents attention weight.

[0046] As a preferred embodiment of the cross-modal source-channel joint encoding and decoding method for dealing with changing scenarios described in this invention, wherein: the interaction information between tactile features and image features is obtained and fused into the image features, and the loss function is expressed as:

[0047] Where ∈ represents the original image features without noise addition, ò θ (z t ,t,τ θ (y) represents the output of the Unet network;

[0048] The FID parameter index is used to guide and constrain the image generation, which is represented as:

[0049] Wherein, Tr represents the trace of the matrix, x and g represent the real picture and the generated image, μ represents the mean, and Σ is the covariance matrix;

[0050] The total loss function is represented as: L HVLDM =L C_LDM +L FID (x,g)。

[0051] As a preferred scheme of the cross-modal source channel joint coding method for coping with variable scenes, wherein: the generalization ability of the model optimized by transfer learning and the acceleration of the training process of the model include defining a first transmission task and a second transmission task for the scene where image signals and tactile signals are transmitted simultaneously; the first transmission task is to transmit the fusion features of image signals and tactile signals; the second transmission task is to transmit only tactile signal features;

[0052] Load the pre-trained Transformer encoder C α (·) and the decoder weights for the first transmission task; input the feature data set D required for the second transmission task; freeze the third layer network parameters of C α (·), ; retrain the first layer and the second layer network parameters C α1、2 (·), to obtain new Transformer first and second layer encoder parameters; meanwhile, different channel signal-to-noise ratios will be encountered in cross-modal communication;

[0053] Input the pre-trained Transformer encoder model C α (·) and the decoder with different channel signal-to-noise ratio parameters N1; freeze the first and second layer network parameters of C α (·), ; retrain the third layer network C α3 (·), to obtain new trained network C α3 (·), parameters.

[0054] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the cross-modal source channel joint coding method for coping with variable scenes when executing the computer program.

[0055] A computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the steps of the cross-modal source-channel joint coding method for coping with variable scenes as described above.

[0056] The source-channel joint coding method provided by the application reduces power consumption in the network, measures load balancing between the cloud network and the server (host) using the fitness function, converts the load balancing problem into an optimization problem, so that it can be applied to processing resource optimization. The results of the particle swarm are used as the initial population of the genetic algorithm, and the application achieves better results in terms of execution cost, load balancing and completion time. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0058] Fig. 1 is a whole flowchart of a cross-modal source-channel joint coding method for coping with variable scenes provided by an embodiment of the application.

[0059] Fig. 2 is a three-layer Transformer encoder structure diagram in a cross-modal source-channel joint coding method for coping with variable scenes provided by an embodiment of the application.

[0060] Fig. 3 is a cross-modal source-channel joint coding method based on a diffusion model provided by an embodiment of the application.

[0061] Fig. 4 is a U-Net denoising model structure diagram in a cross-modal source-channel joint coding method for coping with variable scenes provided by an embodiment of the application.

[0062] Fig. 5 is a source-channel joint coding method based on transfer learning provided by an embodiment of the application.

[0063] Fig. 6 is a relationship between picture reconstruction quality PSNR and signal-to-noise ratio in a cross-modal source-channel joint coding method for coping with variable scenes provided by an embodiment of the application.

[0064] Fig. 7 is a comparison diagram of a source-channel joint coding method based on a diffusion model and other experiments provided by an embodiment of the application.

[0065] Fig. 8 is a related ablation experiment of a source-channel joint coding method based on a diffusion model provided by an embodiment of the application.

[0066] Fig. 9 is a comparison chart of model training convergence speed in a source channel joint coding method based on transfer learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0068] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0069] Embodiment 1

[0070] Referring to Figs. 1-5, according to an embodiment of the present application, a cross-modal source channel joint coding method for coping with variable scenarios is provided, which includes:

[0071] Step 1, as shown in Fig. 1, for the scenario of simultaneous transmission of image signals and tactile signals, a cross-modal channel encoder communication system based on Transformer is designed. The main goal of this system is to maximize the system capacity and minimize the semantic error, rather than just focusing on bit or symbol error in traditional communication.

[0072] Step 1-1, design a cross-modal source encoder including tactile feature extraction image feature extraction and modal fusion Three modules. Where v represents the image signal, h represents the tactile signal, and θ represents the module parameters. The main purpose of the tactile feature extraction module and the image feature extraction module is to extract features within each modality. After obtaining the features of the image and tactile signals respectively, they are spliced together along the channel dimension and input to the modal fusion module for fusion. All three modules are trained with a classification task. The output of the last fully connected layer is (None, 6), and the output of the second last layer is (None, x), where x is selected as 640 after parameter experiment, and its loss function is designed as:

[0073] where, S v, image feature extraction S h and modal fusion S f Output image features, haptic features and fusion features. j Label representing the object. Respectively represent the haptic feature extraction network S v , image feature extraction S h , modal fusion S f and cross-modal source encoder integration network S en Parameters.

[0074] Step 1-2, design cross-modal source decoder, at the receiving end, need to be reconstructed from the fusion features corresponding image signal. Using the method of Wasserstein generative adversarial network (WGAN) to realize image reconstruction. Discriminator D adopts the design of PatchGAN. Generator G generates fusion features into images The discriminator D is responsible for distinguishing between real images v and generated images The generator G needs to produce more realistic pictures, and the discriminator D needs to have stronger ability to distinguish true and false, so the whole training process is a process of D and G against. Here the loss function is defined as:

[0075] Where,

[0076] Step 1-3, as shown in Figure 2. Design cross-modal channel encoder and decoder based on Transformer. Here mainly adopts the structure of encoder-decoder, each layer decoder takes the output of the final encoder as input, this method designs a whole architecture of Transformer model composed of 3 layers of encoder and 3 layers of decoder. First, design the encoder of Transformer, which contains several identical layers, each layer is composed of multi-head self-attention sublayer and position-based feedforward network sublayer, and each sublayer uses residual connection and normalization layer. Each layer of self-attention unit selects 128, and the activation function selects linear function. The multi-head self-attention layer adopts the scaling dot product model to calculate the self-attention score, and the self-attention calculation takes Q, K and V. Then the Q, K and V are projected through h different linear transformations, and the self-attention scores of different representation subspaces are spliced to obtain the final attention score. The position-based feedforward network performs nonlinear transformation on the representation of all positions in the sequence, which increases the representation ability of the model. The feedforward network (which performs nonlinear transformation) is a two-layer fully connected layer, the activation function of the first layer is ReLu, and the second layer is linear activation function: FFN out = max(0, xW1+b1)W2+b2,

[0077] where x is the fusion feature vector obtained in the previous step, W1, W2, b1 and b2 are all learnable parameters in the feed-forward network, and FFNout represents the output of the feed-forward network. The residual connection layer makes the network focus only on the part of the current difference in the process of update iteration, which is used to deal with the problems of grid degradation and gradient disappearance in the process of multi-layer network training; the normalization layer normalizes the input of each layer of neurons to the same mean and variance to speed up the model convergence. The entire calculation process of the two sub-layers can be represented as:

[0078] sub_layer_output = LayerNorm(x + sublayer(x)),

[0079] Then the decoder of the Transformer is designed, which has a similar structure as the encoder, but the decoder also adds an encoder-decoder attention sub-layer between the multi-head self-attention sub-layer and the feed-forward network sub-layer. The activation function of this layer is a linear function. The two attention sub-layers have the same structure, but the sources of the query, key and value are different. Specifically, the query of the encoder-decoder attention sub-layer is the output result of the previous layer of the decoder, and the key and value are generated by decomposing the output of the entire encoder, while the multi-head self-attention sub-layer obtains the query, key and value from the previous layer of the decoder. In addition, this layer introduces a mask function to retain its self-recurrent property, ensuring that prediction only depends on the generated output tokens, ensuring consistency between the training and inference stages. The output of the decoder is a float vector, which is input into a linear layer and then a softmax layer to generate the final output word. Among them, the linear layer uses a fully connected neural network to map the output vector of the decoder to a larger linear space. Then, the softmax layer converts these vectors into probabilities, and finally sorts the probability values to locate the dimension with the highest probability and generate the corresponding feature, which is the output of this time step. This method uses the MSE loss to train the model.

[0080] Steps 1-4, in order to represent the process of transmitting the encoded fusion features through the channel, the channel is modeled as a series of untrainable but differentiable neural network layers. Two commonly used channel models are considered: additive white Gaussian noise (AWGN) channel and Rayleigh slow fading (Rayleigh) channel. In the AWGN channel, the transmission function of the channel can be represented as: y = z + n. z is the channel transmission symbol, n is the channel noise, which is independent and identically distributed, and n ~ N(0, σ 2 ), σ 2 is the average noise power. The signal-to-noise ratio of the channel is By changing σ 2to change the signal-to-noise ratio of the channel. The transmission function of the Rayleigh slow fading channel is: y = hz + n. h is the channel gain, n is the channel noise, h and n respectively obey different normal distributions with parameters and .

[0081] Step 2, as shown in Figure 3, considering the scenario of image signal loss under cross-modal communication, a cross-modal source codec optimization scheme of haptic-image generation based on implicit diffusion model is designed. The method is as follows: 1. Design a feature extraction module to map the picture from the pixel domain to the feature domain; 2. In the feature domain, the learning process of adding noise and denoising the obtained picture features is carried out; 3. In the conditional domain, the haptic feature is trained to obtain the haptic feature, and the corresponding picture generation is guided by the design. The model structure is shown in Figure 2.

[0082] Step 2, considering the scenario of image signal loss under cross-modal communication, a cross-modal source codec optimization scheme of haptic-image generation based on implicit diffusion model is designed. The method is as follows: 1. Design a feature extraction module to map the picture from the pixel domain to the feature domain; 2. In the feature domain, the learning process of adding noise and denoising the obtained picture features is carried out; 3. In the conditional domain, the haptic feature is trained to obtain the haptic feature, and the corresponding picture generation is guided by the design.

[0083] Step 2-1, map the image signal from the pixel domain to the feature domain. For a given image x in the pixel domain, use the encoder to encode x into hidden variable features: use the decoder to reconstruct the image from the feature space, which can be obtained as follows:

[0084] where The specific encoder and decoder structure can refer to the source encoder structure and source decoder structure in step 1.

[0085] Step 2-2, the learning process of adding noise and denoising the image features obtained in the previous step in the feature domain. For the forward noise adding process, the purpose is to add noise of the same distribution to the sample constantly, so that all samples eventually tend to be isotropic distribution. Specifically, it first defines a forward diffusion process, in which process, as shown in the following formula, a small amount of Gaussian noise is added to the sample, and the step size of adding noise is controlled by , thereby generating a series of noise samples x1,...,x T .

[0086] After the forward noise adding process, the q(x t |x t-1 ) process is needed to obtain the recreated real samples x t . However, since this process traverses the entire dataset, it is impossible to obtain the real samples by simple estimation. Therefore, the U-Net model is designed to learn the approximation of these conditional probabilities.

[0087] Step 2-3, as shown in FIG. 4, the overall structure of the U-Net network proposed by the application is similar to the standard U-Net, but the innovation lies in the introduction of the Spatial Transformer (Cross Attention) module. Therefore, this module is highlighted here. Cross Attention is a multi-head attention mechanism that can establish a connection between two different input sequences and transfer information from one sequence to another. The Cross Attention module is used to control the fusion interaction of tactile and image information, that is, through this module, the U-Net can associate a certain part of the noisy image feature matrix with specific information of the tactile feature label.

[0088] In order to preprocess the conditional tactile information y, the tactile feature extraction module τ θ in step one is used, which functions to convert the tactile condition data y into tactile feature expression inside the model . Then τ θ (y) is mapped to the middle layer of the Unet model through the Cross Attention module to interact with the image information, thereby realizing the effect of guiding image generation with tactile information. Then the above process can be represented as follows: first, calculate the query vector Q, the key vector K, and the value vector V through three independent fully connected layers, as shown in the formula:

[0089] wherein, represents the process operation of the hidden variable feature z before entering the Unet network after the forward noise adding process. τ θ (y) represents the corresponding tactile feature information, are three trainable parameter projection matrices. Secondly, calculate the attention weight and perform weighted fusion as shown in formula (4.7):

[0090] wherein d represents the dimension of the query vector Q and the key vector K, represents the attention weight. In this way, the interaction information between the tactile feature and the image feature is obtained and fused into the image feature.

[0091] Step 2-4 Based on the above tactile-image condition generation process, the current loss function can be described by the following formula:

[0092] Wherein ∈ represents the original image feature without adding noise, ò θ (z t ,t,τ θ (y)) represents the Unet network output.

[0093] In addition, the above loss function is based on the mathematical derivation of the model principle, and in the tactile-vision generation task proposed in this paper, the FID (Frechet Inception Distance) parameter index can be used to guide and constrain image generation. FID is an index for evaluating the difference between the generated model and the real data distribution. Then in this paper it can be expressed as:

[0094] In the formula, Tr represents the trace of the matrix. x and g represent the real picture and the generated image, μ represents the mean, and Σ is the covariance matrix. Lower FID indicates that the distribution of the two images is closer, that is, the performance of the model generated image is better.

[0095] The total loss function of the model can be represented as L HVLDM = L C_LDM + L FID (x,g).

[0096] Step 3: For different communication tasks and environments, the method of transfer learning is used to ensure the generalization ability of the model and speed up the training process of the model, and reduce the training cost of the model. As shown in Figure 5.

[0097] Step 3-1, according to the assumed known task K0 and channel condition N0, load the pre-trained channel encoder and channel decoder. For applications with different background knowledge, the present application only needs to retrain part of the Transformer encoder and decoder layers. Specifically: according to the two transmission scenarios faced by the present application, task 1: demand transmission of image signal and tactile signal fusion features; task 2: only tactile signal features are transmitted.

[0098] Step 3-11, load the pre-trained Transformer encoder C α (·) and decoder weights for task 1.

[0099] Step 3-12, input the feature data set D required for task 2.

[0100] Step 3-13, freeze C α (·), The parameters of the third layer network.

[0101] Step 3-14: Retrain the parameters C of the first and second layers of the network. α1、2 (·),

[0102] Step 3-15: Obtain the encoder parameters for the first and second layers of the new Transformer.

[0103] Step 3-2: At the same time, cross-modal communication will also face different channel signal-to-noise ratio variations.

[0104] Step 3-21, Input: Pre-trained Transformer encoder model C α (·), decoder Different channel signal-to-noise ratio parameters N1.

[0105] Step 3-22, Freeze C α (·), The parameters of the first and second layer networks.

[0106] Step 3-23: Retrain the third layer network C α3 (·),

[0107] Step 3-24: Obtain the newly trained network C α3 (·), parameter.

[0108] In the above process, the loss function for training is set to L. HVLDM The Adam optimization algorithm is used for optimization. The Adam optimization algorithm is based on gradient descent and incorporates adaptive adjustment of the learning rate. Its expression is as follows: m t =β1m t-1 +(1-β1)g t

[0109] Among them, g t θ is the gradient of the parameters, β1 and β2 are the decay coefficients of the two exponentially weighted averages, and m and v are the bias-corrected moving averages of the gradient. t and θ t+1 These are the network parameters before and after the update, respectively. The network parameters here refer to the parameters of the first, second, and third layers of the Transformer encoder and decoder.

[0110] Example 2, an embodiment of the present invention, differs from the previous embodiment in that:

[0111] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0112] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution systems, apparatus or devices. For the purpose of this specification, the "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0113] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be electronically obtained and then stored in the computer memory.

[0114] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies known in the art or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0115] Embodiment 3, referring to FIGS. 6-9, provides a cross-modal source channel joint coding method for coping with variable scenarios, which is an embodiment of the present application. In order to verify the beneficial effects of the present application, scientific demonstration is carried out through simulation experiments.

[0116] The present embodiment uses the AU dataset data set, which contains multi-modal information of 63 objects, including images, kinesthetic and tactile data (audio vibration). These objects are mainly toys and household items. Some objects have similar shapes and colors on the image, while having similar materials and textures but different colors in the tactile. These objects reflect the different effects of images and tactile on perception.

[0117] The present application respectively carries out experiments on step 1, step 2 and step 3 modules.

[0118] Step 1 related comparative experiment adopts neural network based source channel joint coding and typical source channel separation coding method. Deep neural network based source channel joint coding (JSCC): the network is composed of bidirectional long short term memory (BLSTM) layer. It is marked as JSCC-LSTM in the simulation diagram, and JSCC-CNN. Traditional method, respectively using the following technology: channel coding: Turbo coding and Reed-Solomon (RS) coding. The turbo decoding method used in the present application is 5 times iteration log-MAP algorithm.

[0119] Step 2 uses comparative experiment and ablation experiment, and the comparative experiment selects the following two methods:

[0120] Style GAN: generates intermediate latent space through mapping network, which reduces the entanglement of input latent space, and at the same time puts forward two new indexes to quantify the coupling of latent space-perception path length and linear separability. Finally, it achieves excellent success in the task of face image generation.

[0121] Projected GAN: Generative adversarial networks can generate high-quality images, but are challenging to train. They require careful regularization, large amounts of computation, and expensive hyperparameter sweeps. This model makes significant progress on these issues by projecting generated and real samples into a fixed, pre-trained feature space. Since it was found that discriminators could not fully exploit the deeper features of the pre-trained model, a more effective strategy was proposed that could mix features across channels and resolutions. Projected GAN can improve image quality, sampling efficiency, and convergence speed.

[0122] The present invention: The method of the present embodiment.

[0123] The following is an introduction to the measurement indicators used in the present invention: IS indicator is an indicator for evaluating the quality of generated images, which combines the diversity and authenticity of images. Inception Score is an indicator of KL divergence, which is the exponential mean of the KL divergence of the conditional distribution of each image and the overall distribution. The mathematical expression is as follows:

[0124] IS = exp (E x [KL (p (y | x) | p (y))],

[0125] Where p (y | x) is the conditional distribution, representing the probability distribution of class y given image x; p (y) is the overall distribution, representing the probability distribution of class y for all images; E x represents the expectation over all images.

[0126] FID is an indicator for evaluating the difference between the generated model and the real data distribution. So in the present invention it can be represented as:

[0127] In the formula, Tr represents the trace of the matrix. x and g represent real pictures and generated pictures, μ represents the mean, and Σ is the covariance matrix.

[0128] In addition, PSNR is also a commonly used objective evaluation indicator for images, and its mathematical expression is defined as:

[0129] Where MSE is the mean square error of the real image and the generated image. The larger the PSNR is, the better.

[0130] Table 1 is the experimental results of the present invention

[0131] From Table 1 and Fig. 6, it can be seen that compared with the above-mentioned most advanced method, the method proposed by us has obvious advantages. In the traditional baseline shown in Fig. 6, when turbo coding is used for channel coding, in the AWGN channel, when the signal-to-noise ratio is greater than 12dB, the performance of the traditional method is better than that of the deep neural network-based method. All deep learning-supported methods are more competitive under low signal-to-noise ratio conditions. And in the sharp fading channel, the performance of the traditional scheme under various conditions is not as good as that of the deep learning-based method. The method of the present application achieves the best performance under most conditions. This is also in the environment of complex multiplicative noise of sharp fading channel, the distortion of the traditional method is higher, which cannot accurately decode the corresponding feature vector, resulting in the performance decline of the final signal source decoding.

[0132] As shown in Fig. 7, the results of the present application are better than the methods compared therewith, including Style GAN and Projected GAN. The method of the present application trains the diffusion model in a fixed latent space, avoiding the trade-off problem between reconstruction quality and learning latent space prior. The experimental results are obviously better than the two methods in all indicators. Through the multi-head attention mechanism, the tactile modal information is successfully mapped into the latent feature representation of the UNet network, which can successfully guide the generation of images.

[0133] As shown in Fig. 8, HVLDM without Lc_LDM represents the case of unconditional generation, and the model's ability to generate pictures corresponding to the input tactile is greatly reduced. In five experiments, the model cannot accurately recover the original picture in most cases. This shows that in this structure, the tactile information of the conditional field plays an important guiding role in picture generation.

[0134] HVLDM without Lfid represents that the loss function of the model is only composed of L C-LDM The experimental results show that the images generated by the model after lacking Lfid are not as good as the complete model in IS and FID indicators, that is, the similarity of the generated pictures lags behind the HVLDM model. This also shows that the loss function Lfid plays a function of constraining image generation to approximate the real image class in this task.

[0135] FIG. 9 shows the training efficiency and performance under different backgrounds, where the model will be trained and retrained on a new background using the same channel (AWGN) under different backgrounds. The model has the same structure and is retrained using the same parameters in each scenario. In figures (a) (c), the number of epochs is reduced from 15 to 5, 25 to 10, respectively, to reach convergence. This shows that the transfer learning assisted TransJSCC can help the transceiver adapt to the new communication environment requirements. As can be seen from figures (b) (d), using a pre-trained model can speed up the training process in sharp channels. At the same time, due to the smaller number of epochs required in the retraining process, the required complexity is significantly reduced.

[0136] It should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A cross-modal source-channel joint coding method for coping with variable scenes, characterized in that, The application relates to a cross-modal source encoder, a cross-modal source decoder and a cross-modal channel model. The application maximizes the system capacity and minimizes semantic errors based on the cross-modal source encoder for the simultaneous transmission of image signals and tactile signals. The application designs a cross-modal source coding and decoding optimization scheme for tactile-image generation based on a hidden diffusion model for the scenario of image signal loss in cross-modal communication. The application uses transfer learning to optimize the generalization ability of the model and accelerate the training process of the model for different communication tasks and channel environments.

2. The cross-modal source-channel joint coding method for coping with multiple variable scenarios of claim 1, wherein: The cross-modal source encoder comprises a tactile feature extraction module, an image feature extraction module and a modal fusion module. The tactile feature extraction module and the image feature extraction module extract features in each modal. When the features of the image and the tactile signal are obtained respectively, they are spliced together along the channel dimension and input to a modal fusion module for fusion, and the loss function is represented as: wherein denote the image feature output by the image feature extraction network S v , the tactile feature output by the tactile feature extraction network S h , and the fused feature output by the modal fusion network S f , respectively. j denote the label of the object, respectively represent parameters of the tactile feature extraction network S v , the image feature extraction S h , the modal fusion S f , and the cross-modal source encoder integration network S en . The cross-modal source decoder is used at the receiving end to reconstruct the corresponding image signal from the fused features. The Wasserstein generative adversarial network is used to realize image reconstruction, and the discriminator D adopts the PatchGAN design. The generator G generates the fusion feature as an image The discriminator D is responsible for distinguishing between real images v and generated images The training process is a process of D and G against each other, and the loss function is represented as: wherein, 3. The cross-modal source-channel joint coding method for coping with multiple variable scenarios of claim 2, wherein: The encoder of the Transformer is designed, and the encoder comprises a plurality of identical layers, each layer is composed of a multi-head self-attention sublayer and a position-based feedforward network sublayer, and each sublayer uses a residual connection layer and layer normalization. The multi-head self-attention layer adopts a scaled dot-product model to calculate the self-attention score, and the self-attention calculation takes Q, K and V as the same; the self-attention scores of different representation subspaces are spliced to obtain the final attention score through h different linear transformations on Q, K and V. The position-based feedforward network performs nonlinear transformation on the representation of all positions in the sequence, and passes through a two-layer fully connected layer, the activation function of the first layer is ReLu, and the second layer is a linear activation function, and is represented as: FFN out = max(0, xW1 + b1)W2 + b2, wherein x is the fusion feature vector obtained in the previous step, W1, W2, b1 and b2 are all learnable parameters in the feedforward network, FFN out is the output of the feedforward network. The residual connection layer is used for only the part of the current difference in the process of updating iteration. The normalization layer normalizes the input of each layer of neurons to the same mean and variance, and accelerates the convergence of the model. The calculation process of the sublayer is represented as: sub_layer_output=LayerNorm(x+sublayer(x)), The decoder of the Transformer is designed, and the encoder-decoder attention sublayer is added between the multi-head self-attention sublayer and the feedforward network sublayer of the decoder. The channel is modeled as a neural network layer to represent the transmission process of the coded fused features through the channel, and additive white Gaussian noise channels and Rayleigh slow fading channels are considered. Under the additive white Gaussian noise channel, the transmission function of the channel can be represented as: y=z+n where z is a channel transmission symbol, n is a channel noise, is independently and identically distributed, and n ~ N(0, σ 2 ), σ 2 is the average noise power; The signal-to-noise ratio of the channel is by changing σ 2 to change the signal-to-noise ratio of the channel; The transmission function of the Rayleigh slow fading channel is: y=hz+n where h is the channel gain, n is the channel noise, h and n are subject to parameters and The cross-modal source coding and decoding optimization scheme comprises the following steps.

4. The cross-modal source-channel joint coding method for coping with multiple variable scenarios of claim 3, wherein: The image signal is mapped from the pixel domain to the feature domain. For a given image x in the pixel domain, Using an encoder ε to encode x into latent variable features z = ε(x), using a decoder Reconstructing the image from the feature space is represented as: where z e R h×w×c .

5. The cross-modal source-channel joint coding method for coping with multiple scenarios of claim 4, wherein: The learning process of adding noise and denoising the image features in the feature domain is performed. For the forward noise addition process, a forward diffusion process is defined, denoted as: After the forward noise addition process, the q(x) is reversed. t |x t-1 The process yields a recreated real sample x t ; The U-Net model is designed to learn the approximate conditional probability.

6. The cross-modal source-channel joint coding method for coping with multiple scenarios of claim 5, wherein: The design of the U-Net model to learn the approximate conditional probability comprises the following steps. The Cross Attention module is introduced. The Cross Attention is a multi-head attention mechanism that establishes the association between two different input sequences and transmits the information of one sequence to the other sequence, through the Cross Attention module, the U-Net model associates a part of the noisy image feature matrix with the specific information of the tactile feature label; Converting haptic condition data y into a haptic feature representation within the model Will τ θ (y) is fused and interacted with image information by the Cross Attention module in the middle layer of the Unet model, denoted as: wherein denotes the process operation of the latent variable feature z before entering the Unet network after the forward noise adding process, τ θ (y) denotes the corresponding tactile feature information, are three trainable parameter projection matrices; The attention weights are calculated and fused with weighting, denoted as: wherein d represents the dimension of the query vector Q, the key vector K, represent the attention weights.

7. The cross-modal source-channel joint coding method for coping with multiple scenarios of claim 6, wherein: The interaction information between the tactile features and the image features is obtained and fused into the image features, and a loss function is represented as: Wherein, ∈ represents the original image features without adding noise, ò θ (z t ,t,τ θ (y)) represents the Unet network output; The FID parameter index is used to guide and constrain the image generation, which is expressed as: Wherein, Tr represents the trace of the matrix, x and g represent the real picture and the generated image, μ represents the mean, and Σ is the covariance matrix; The total loss function is represented as: L HVLDM = L C_LDM + L FID (x, g).

8. The cross-modal source-channel joint coding method for coping with multiple scenarios of claim 7, wherein: The generalization ability of the model is optimized by using the transfer learning, and the training process of the model is accelerated, which comprises defining a first transmission task and a second transmission task for a scene in which image signals and tactile signals are transmitted simultaneously; the first transmission task is to transmit fused features of the image signals and the tactile signals; and the second transmission task is to transmit only tactile signal features; loading a pre-trained Transformer encoder C oriented to the first transmission task α (·), decoder weights; inputting a feature data set D required for a second transmission task; freezing C α (·), third layer network parameters; retrain the first and second layer network parameters C α1、2 (·), New Transformer first and second layer encoder parameters are obtained; meanwhile, different channel signal-to-noise ratio changes will be faced in cross-modal communication; inputting a pre-trained Transformer encoder model C α (·), decoder Different channel SNR parameters N1; freeze C α (·), the first and second layer network parameters of the first network C α3 (·), obtaining a newly trained network C α3 (·), parameters. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the cross-modal source-channel joint coding method for dealing with variable scenes according to any one of claims 1 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the cross-modal source-channel joint coding method for dealing with variable scenes according to any one of claims 1 to 8.

Images