Information source channel joint coding and decoding method based on cross-modal communication system
By designing the joint codec method of source channel in a cross-modal communication system, extracting and fusing image and tactile features, the problem of variability in the multimodal data transmission environment is solved, and better signal transmission effect and robustness are achieved .
Patent Information
- Application Number
- CN202411075763.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-07
AI Technical Summary
Existing cross-modal communication systems cannot effectively handle the transmission of multimodal data, especially in the case of changing transmission environments, lack of a transmission framework that comprehensively considers the source and channel, resulting in insufficient signal transmission quality and robustness.
A joint codec method for source channel based on a cross-modal communication system is designed. Image and tactile features are extracted through a cross-modal source encoder to fuse, and the fusion features are converted into A bitstream suitable for wireless transmission, and the image signal is reconstructed through the source decoder at the receiving end, the loss function is used to update the parameters, and the training is carried out in combination with neural networks and knowledge distillation technology.
It achieves better signal feature extraction effect, overcomes the cliff effect in traditional codecs, improves the cross-modal source generation effect, and maintains the accuracy and reliability of signals under harsh channel conditions.
Smart Images

Figure CN119011843A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cross-modal image signal reconstruction, and in particular to a source channel joint encoding and decoding method based on a cross-modal communication system. Background Art
[0002] With the development of wireless communication technology and people's satisfaction with traditional multimedia services such as audio and video, they will further pursue multi-sensory immersive experience. Then, multimodal services integrating audio, video and tactile signals will gradually become the mainstream in various scenarios (remote operation, online education, e-health, digital twins). For example, in remote education scenarios, audio and video services that integrate tactile perception and feedback can solve the problem that remote practical teaching cannot be effectively carried out, and improve students' learning effects and immersive learning experience. In virtual world interaction, tactile perception can bring users a more realistic interactive experience.
[0003] However, in multimodal services, there are significant differences in the transmission requirements of audio, video and touch. For the former, high throughput needs to be met, and touch is more sensitive to low latency and high reliability. Traditional audio and video communications and tactile communications cannot meet the requirements. Based on this, cross-modal communication came into being. Cross-modal communication makes full use of the potential correlation and semantic public information between modalities, establishes a general cross-modal flow scheduling scheme, and uses AI methods to achieve efficient multimodal flow, processing and recovery. It has also achieved initial results in application scenarios such as remote acupuncture teaching, remote operation, and remote throat swabs.
[0004] On the other hand, in order to further improve the transmission efficiency, the source-channel joint coding and decoding method (JSCC) does not perform traditional separate coding on the image, but maps the image pixel values to the channel input symbols to achieve joint optimization of the source and channel. Experiments show that the results obtained by the joint method are better than the communication method based on the separation scheme, and can overcome the cliff effect when the channel environment deteriorates. This method is becoming a focus of attention in academia and industry. Compared with traditional bit-based communication, by highly abstracting the semantic information in the source, it can further compress data redundancy and improve the network information transmission capacity; and by reducing the amount of transmitted data, it can reduce the network information processing delay and is more robust to poor channels. However, JSCC only targets single mode and cannot handle multi-modal situations. To summarize, taking tactile-image cross-modal communication as an example, the challenges faced by existing cross-modal communication are as follows: there is a lack of a specific and feasible framework. On the one hand, the existing communication transmission framework only focuses on single-modal transmission such as tactile stream or audio-visual stream; on the other hand, the transmission environment faced by cross-modal communication is changeable. Therefore, it is necessary to design a cross-modal transmission framework that comprehensively considers the source and channel to ensure the quality and robustness of signal transmission. Summary of the invention
[0005] In view of the above-mentioned problems, the present invention is proposed.
[0006] Therefore, the technical problem solved by the present invention is: integrating a cross-modal source encoder, a channel encoder, a channel decoder and a source decoder to achieve efficient processing and transmission of multimodal data.
[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a source channel joint coding and decoding method based on a cross-modal communication system, comprising:
[0008] Extracting image features and tactile features corresponding to the image signal and tactile signal to be transmitted through a cross-modal source encoder, and fusing the image features and the tactile features to obtain fused features;
[0009] Design a cross-modal channel encoder and a cross-modal channel decoder, input the fused features into the cross-modal channel encoder, and convert them into a bit stream suitable for wireless transmission. After the channel transmission process is completed, the channel decoder maps the received bit stream back to the fused features.
[0010] The fused features are decoded by a cross-modal source decoder and the image signal is reconstructed from the fused features.
[0011] As a preferred solution of the source channel joint encoding and decoding method based on the cross-modal communication system described in the present invention, wherein: the cross-modal source encoder includes a tactile feature extraction module, an image feature extraction module and a modality fusion module;
[0012] The loss function is designed to update the relevant parameters of the cross-modal source encoder; the loss function is designed to use the cross entropy loss function to classify image features, tactile features and fusion features, expressed as:
[0013]
[0014] in, Represents the tactile feature extraction network Image feature extraction and modal fusion Output image features, tactile features and fusion features, y j The label of the object. Represents the tactile feature extraction network Image feature extraction Modal Fusion and cross-modal source encoder integration network Parameters;
[0015] By minimizing the loss function L s, perform preliminary training on the cross-modal source encoder parameters.
[0016] As a preferred solution of the source channel joint encoding and decoding method based on the cross-modal communication system described in the present invention, wherein: the cross-modal channel encoder is expressed as,
[0017]
[0018] in, represents the parameters of the channel encoder model;
[0019] The cross-modal channel encoder inputs the cross-modal fusion feature f and outputs the representation z of the feature mapped to the channel symbol.
[0020] As a preferred solution of the source channel joint encoding and decoding method based on the cross-modal communication system described in the present invention, the channel is modeled as a process in which the encoded fusion feature z is transmitted through the channel by a neural network layer;
[0021] The channel modeling includes an additive white Gaussian noise channel and a Rayleigh slow fading channel;
[0022] In the AWGN channel, the channel transfer function is expressed as:
[0023] y=z+n
[0024] Where n is the channel noise, which is independent and identically distributed, and σ 2 is the average noise power;
[0025] The signal-to-noise ratio of the channel is By changing σ 2 To change the signal-to-noise ratio of the channel;
[0026] The transfer function of the Rayleigh slow fading channel is,
[0027] y=hz+n
[0028] Where h is the channel gain, n is the channel noise, h and n are subject to the parameters and Different normal distributions.
[0029] As a preferred solution of the source channel joint coding and decoding method based on the cross-modal communication system described in the present invention, wherein: the cross-modal channel decoder, after being transmitted through the channel, receives the complex valued signal interfered by noise The estimated value mapped back to the original feature input by the channel decoder
[0030] Corresponding to the channel encoder, the channel decoder consists of a series of one-dimensional transposed convolutions, parameterized ReLU (PReLU) activation functions, and anti-normalization layers;
[0031] The channel decoder is represented as,
[0032]
[0033] in, represents the parameters of the channel decoder model;
[0034] The final channel codec is jointly designed and optimized by minimizing the cross-modal fusion feature f and the fusion feature estimated after transmission. The mean square error between them is used to jointly optimize the channel codec parameters, expressed as:
[0035]
[0036] As a preferred solution of the source channel joint encoding and decoding method based on the cross-modal communication system of the present invention, wherein: the cross-modal source decoder, at the receiving end, obtains the fusion feature from the fusion feature. Reconstructing the corresponding image signal;
[0037] The Wasserstein generative adversarial network method is used to achieve image reconstruction. The discriminator D adopts the design of PatchGAN, and the generator G generates the fused features into an image. The discriminator D is responsible for distinguishing between real images v and generated images
[0038] The loss function is defined as:
[0039]
[0040] in, ~U[0,1].
[0041] As a preferred solution of the source-channel joint encoding and decoding method based on the cross-modal communication system described in the present invention, wherein: using the knowledge distillation technology method, the key distribution information of the generated image is extracted from the teacher model and applied to the student model;
[0042] Select the VGG16 model pre-trained on the ImageNet dataset as the teacher model and the generator G as the student model;
[0043] The generator G is derived from the fusion features To Image In the generation process of , each layer of the generator represents the information of different levels of the image, so the feature information of the lth layer is expressed as Represents the parameters of the lth layer;
[0044] When the image is input into the VGG16 network, layers of different depths extract information at different levels. Deeper layers often represent deep information. The feature information of the mth layer is expressed as represents the parameters of the mth layer, and the layer-wise constraint loss is expressed as,
[0045]
[0046] The layers in the generator correspond to some layer structures in VGG16, all have the same feature output size, and calculate the layer-by-layer constraint loss. The loss function for the entire image generation is expressed as,
[0047] L genv =L Gv +δL layerloss ,
[0048] Among them, δ is a hyperparameter used to balance the two loss functions.
[0049] As a preferred solution of the source channel joint encoding and decoding method based on the cross-modal communication system described in the present invention, wherein: training optimization is performed based on a joint optimization algorithm, and the joint optimization algorithm includes a first stage and a second stage;
[0050] The first stage pre-trains the network parameters of the cross-modal source encoder by minimizing the loss function, combines the cross-modal source encoder with the cross-modal source decoder for adversarial training, and conducts adversarial training with the discriminator, by calculating the loss function and performing gradient updates on the network parameters until the model converges;
[0051] The second stage performs a gradient descent algorithm on the parameters of the channel encoder and the channel decoder by optimizing the loss function until the entire system reaches a stable state;
[0052] After the above training process, the optimal cross-modal source encoder, cross-modal source decoder and channel encoder, and cross-modal channel decoder are obtained.
[0053] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the source channel joint coding and decoding method based on a cross-modal communication system are implemented.
[0054] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the source-channel joint encoding and decoding method based on a cross-modal communication system as described above.
[0055] Beneficial effects of the present invention:
[0056] The source-channel joint coding and decoding method based on the cross-modal communication system provided by the present invention utilizes cross-modal information to achieve a better signal feature extraction effect. The design of the joint coding and decoding method overcomes the cliff effect faced by traditional coding and decoding. It can achieve a better cross-modal source generation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0058] Figure 1 A framework diagram of a source-channel joint encoding and decoding method based on a cross-modal communication system provided by an embodiment of the present invention.
[0059] Figure 2 A schematic diagram of hierarchical constraints of a source-channel joint encoding and decoding method based on a cross-modal communication system is provided as an embodiment of the present invention.
[0060] Figure 3 A diagram showing the cross-modal image signal reconstruction result of a source-channel joint encoding and decoding method based on a cross-modal communication system and other comparative methods provided by an embodiment of the present invention.
[0061] Figure 4 A diagram of cross-modal image signal reconstruction results under different channel conditions based on a source-channel joint encoding and decoding method of a cross-modal communication system provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0062] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0063] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0064] Example 1, reference Figure 1-Figure 2, is an embodiment of the present invention, and provides a source-channel joint encoding and decoding method based on a cross-modal communication system, comprising:
[0065] Step 1: Design a cross-modal source encoder. Extract image features and tactile features corresponding to the image signal and tactile signal to be transmitted respectively, and fuse the image features and tactile features to obtain fused features.
[0066] Step 1-1: Design a cross-modal source encoder Including tactile feature extraction Image feature extraction and modal fusion There are three modules. v represents the image signal, h represents the tactile signal, and θ represents the module parameters. The main purpose of the tactile feature extraction module and the image feature extraction module is to extract features within each modality. In the image feature extraction module, the VGG16 network pre-trained on the ImageNet dataset is used, but the fully connected layer in the VGG16 network is not included. In order to better utilize the category information, two fully connected layers are added after the VGG16 model, one of which is used as the feature output t of the image signal. v , and the last layer is used for image label classification. The output of the last fully connected layer is (None, 6), and the output of the second-to-last layer is (None, x), where the value of x is selected as 640 after parameter experiments. It is worth noting that during the training process, the parameters of the VGG16 network are frozen, and only the parameters of the last two fully connected layers are updated.
[0067] In contrast, the tactile feature extraction module and the modality fusion module use fully connected layers of different sizes. The tactile feature extraction module accepts tactile sequences as input, so different fully connected layers are designed to extract tactile features from tactile data. Similar to the image module, the last layer is responsible for tactile label classification, while the second to last layer is used as tactile feature output t h The output of the last fully connected layer is (None, 6), and the output of the second-to-last layer is (None, x), where the value of x is selected to be 640 after parameter experiments.
[0068] After obtaining the features of the image and tactile signal respectively, they are concatenated along the channel dimension and input into the modality fusion module for fusion, which is composed of multiple fully connected layers of different sizes. Similar to the first two modules, the output of the second-to-last layer is selected as the fused feature f, and the last layer is used for label classification. Formula section (next section) Formula section (next section) The output of the last fully connected layer is (None, 6), and the output of the second-to-last layer is (None, x), where the value of x is selected as 1280 after parameter experiments.
[0069] Step 1-2, accurately designing the loss function is crucial for updating the relevant parameters of the cross-modal source encoder. Therefore, three different loss functions are designed, corresponding to the three aforementioned modules, and the cross entropy loss function is used for image classification, tactile feature classification, and fusion feature classification. This method helps to jointly update the parameters of the cross-modal signal encoder. The details are as follows:
[0070]
[0071] in, Represents the tactile feature extraction network Image feature extraction and modal fusion Output image features, tactile features and fusion features. j Represents the label of the object. Represents the tactile feature extraction network Image feature extraction Modal Fusion and cross-modal source encoder integration network Parameters of . Among them, By minimizing the loss function L s , perform preliminary training on the parameters of the cross-modal source encoder. In order to achieve better communication effect, it will be jointly trained with the cross-modal decoder.
[0072] Step 2: Design a cross-modal channel codec, which includes a cross-modal channel encoder and a cross-modal channel decoder. After obtaining the fused features in step 1, input them into the channel encoder and convert them into a bit stream suitable for wireless transmission. After the channel transmission process is completed, the channel decoder maps the received bit stream back to the fused features.
[0073] Step 2-1. Design a cross-modal channel encoder, whose input is the cross-modal fusion feature f obtained in step 1, and whose output is the representation z of the feature mapped to the channel symbol. The specific structure of the channel encoder consists of a one-dimensional convolution layer, a PReLU activation function, and a normalization layer. Such a design helps to capture the statistical characteristics of the signal and effectively map the signal features to the channel transmission symbols. The size of the one-dimensional convolution layer is set to (128-256-512-z), where z represents the channel output size of the experimental setting. The combination of one-dimensional convolution and nonlinear activation function allows the model to learn the complex relationship between the signal and the channel symbols. The introduction of the normalization layer ensures that the average power of the signal meets the specific power constraint: Where * represents the complex conjugate operation. The channel encoder model is expressed as: Represents the parameters of the channel encoder model.
[0074] Step 2-2: In order to represent the process of the encoded fusion feature z being transmitted through the channel, the channel is modeled as a series of non-trainable but differentiable neural network layers. Consider two commonly used channel models: additive white Gaussian noise (AWGN) channel and Rayleigh slow fading (Rayleigh) channel. Under the AWGN channel, the channel transfer function can be expressed as: y = z + n. n is the channel noise, which is independent and identically distributed, and σ 2 is the average noise power. The signal-to-noise ratio of the channel is By changing σ 2 To change the signal-to-noise ratio of the channel. The transfer function of the Rayleigh slow fading channel is: y = hz + n. h is the channel gain, n is the channel noise, h and n are subject to the parameters and Different normal distributions.
[0075] Step 2-3: Design a cross-modal channel decoder. After the channel is transmitted, the complex-valued signal is interfered by noise. The estimated value mapped back to the original feature input by the channel decoder Corresponding to the channel encoder, the channel decoder consists of a series of one-dimensional transposed convolutions, parameterized ReLU (PReLU) activation functions, and anti-normalization layers. The size of the one-dimensional convolution layer is set to (512-256-128-1). The mathematical model of the channel decoder can be expressed as: Represents the parameters of the channel decoder model.
[0076] Step 2-4: The final channel codec is jointly designed and optimized by minimizing the cross-modal fusion feature f and the fusion feature estimated after transmission The mean square error between the two is used to jointly optimize the channel codec parameters. This process ensures that the codec system can maintain signal accuracy and reliability in the face of channel distortion and noise interference.
[0077]
[0078] Where N represents the total number of signals and J represents the signal currently being processed.
[0079] Step 3: Design a cross-modal source decoder. Reconstruct the image signal from the fused features by decoding the fused features.
[0080] Step 3-1: Design a cross-modal source decoder. At the receiving end, it is necessary to extract the fusion features from the Reconstruct the corresponding image signal. The Wasserstein Generative Adversarial Network (WGAN) method is used to achieve image reconstruction. The discriminator D adopts the design of PatchGAN. The generator G generates the fused features into an image The discriminator D is responsible for distinguishing between real images v and generated images The generator G needs to produce pictures that are closer to reality, while the discriminator D needs to have a stronger ability to distinguish true from false. Therefore, the entire training process is a process of confrontation between D and G. The loss function here is defined as:
[0081]
[0082] in,
[0083] Step 3-2: In addition, considering the real image signal v and the generated image signal There is inconsistency in data distribution in high-dimensional pixel space, and relying solely on reconstruction loss may result in blurred image signals. To address this challenge, the knowledge distillation technique is used to extract the key distribution information of the generated image from the teacher model and then apply it to the student model to improve the generation accuracy of the image signal. The VGG16 model pre-trained on the ImageNet dataset is selected as the teacher model, and the generator G is used as the student model. Figure 2 As shown, specifically, the generator G is derived from the fusion feature To Image In the generation process of , each layer of the generator represents the information of different levels of the image, so the feature information of the lth layer is expressed as Indicates the parameters of the lth layer. When the image is input into the VGG16 network, layers of different depths extract information at different levels. Deeper layers often represent deep information. The feature information of the mth layer is expressed as represents the parameters of the mth layer. Then, the layer-wise constraint loss is defined as:
[0084]
[0085] The layers in the generator correspond to some layer structures in VGG16 and have the same feature output size, which makes it easier to calculate the layer-by-layer constraint loss. Finally, the loss function for the entire image generation is:
[0086] L genv =L Gv +δL layerloss ,
[0088] Among them, δ is a hyperparameter used to balance the two loss functions.
[0089] Specifically, the generator consists of seven layers of two-dimensional convolutions, and BatchNormalization is performed after each layer of convolution, and the activation function is set to leaky_relu to better handle the nonlinear relationship of features. The discriminator consists of five layers of two-dimensional convolutions, and BatchNormalization is performed after each layer of convolution, and leaky_relu is used as the activation function. In terms of knowledge distillation of the model, we chose the VGG16 model pre-trained on ImageNet as the teacher model, and the generator as the student model. In this way, we distilled the knowledge of layers 1, 4, 11, 17, and 18 in the VGG16 model to layers 2, 3, 4, 6, and 7 of the generator to improve the accuracy of the generator in image generation.
[0090] Step 4 includes the following steps:
[0091] Step 4-1: Perform the first phase of image generation training. In this phase, the network parameters of the cross-modal source encoder are first pre-trained by minimizing the loss function. Then, using this as a starting point, the cross-modal source encoder is combined with the cross-modal source decoder and adversarially trained. At the same time, adversarially trained with the discriminator, by calculating the loss function and performing gradient updates on the network parameters until the model converges.
[0092] Step 4-11: Initialize the cross-modal source codec network parameters Cross-modal source decoder parameters θ gv ,θ dv .
[0093] Step 4-12: According to L s Update the cross-modal source encoder network parameters
[0094]
[0095] Step 4-13, according to L Dv Update θ dv :
[0096]
[0097] Step 4-14: According to L genv renew θ gv :
[0098]
[0099] Step 4-15: In summary, cross-modal source codec network training is implemented:
[0100]
[0101] Step 4-2, the second stage focuses on the training process of the channel encoder and the channel decoder. This process optimizes the loss function and performs a gradient descent algorithm on the parameters of the channel encoder and the channel decoder until the entire system reaches a stable state. Specifically, the goal of this training stage is to minimize the impact of channel distortion on signal quality and ensure the integrity of the signal during transmission.
[0102] Step 4-21, according to L mse renew
[0103]
[0104] Step 4-22: Implement channel codec optimization.
[0105]
[0106] After the above training process, the optimal cross-modal source encoder, cross-modal source decoder and channel codec are obtained.
[0107] Embodiment 2, an embodiment of the present invention, which is different from the previous embodiment in that:
[0108] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0109] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0110] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0111] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0112] Example 3, reference Figure 3 and Figure 4 , which is an embodiment of the present invention, provides a source channel joint encoding and decoding method based on a cross-modal communication system. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.
[0113] This example uses the AU dataset, which contains multimodal information of 63 objects, including images, kinesthetic and tactile data (audio vibration). These objects are mainly toys and household items. Some objects have similar shapes and colors in images, but similar materials and textures but different colors in touch. These objects reflect the different effects of images and touch on perception.
[0114] There are three existing methods:
[0115] Existing method 1: Disco-AC GAN model: DiscoGAN is used to learn the conversion between ground-truth images and spectrograms. By using the trained DX and DY as discriminators to double constrain the generated results and adding an auxiliary classification layer to the original discriminator, this method allows the model to consider category labels during the generation process, thereby generating images that are more consistent with the characteristics of specific categories.
[0116] Existing method 2: Cross-modal visual-tactile mutual generation: A generative model improved based on CGAN, which generates images based on the semantic features of tactile encoding. This model involves inter-modal correlation learning to measure the correlation between the tactile modality and the image modality.
[0117] Existing method 3: A method based on CGAN and residual-fusion (RF) module, trained according to additional feature matching and perceptual loss. The residual fusion module helps the model capture and utilize the deep features of the input data, while the perceptual loss may be used to ensure that the generated data has a high similarity with the real data in the image, thereby achieving mutual generation between the image and tactile spectrum.
[0118] The present invention: the method of this embodiment.
[0119] The experiment uses the structure similarity index measure (SSIM) and peak signal to noise ratio (PSNR) as evaluation indicators to evaluate the effect of cross-modal generation. The experimental results are shown in Table 1.
[0120] Table 1 Comparison of experimental results
[0121] Method 1 Method 2 Method 3 The present invention AveragePSNR 30.961 26.360 23.613 27.96 AverageSSIM 0.890 0.8193 0.7979 0.856 PSNR Standard Deviation 7.187 5.6333 5.0238 2.87
[0122] From Table 1 and Figure 3 It can be seen that compared with the above state-of-the-art methods, our proposed method has obvious advantages.
[0123] At the same time, the present invention also conducts relevant channel experiments to verify the robustness of the model to channel changes. The experimental results are as follows Figure 4 As shown. During the experiment, the experimental results under different training signal-noise ratios were analyzed in detail. It is worth noting that when the test signal-noise ratio is lower than the training signal-noise ratio, that is, when the channel conditions are poor, the system can still successfully restore the image. Under the same test signal-noise ratio, the closer the training signal-noise ratio is to the test signal-noise ratio, the better the image quality restored by the system, which indicates that the more consistent the training channel conditions are with the test channel conditions, the better the system performance. As the channel conditions improve, the gap in image restoration effects under different training signal-noise ratios gradually decreases, which reflects the model's ability to adapt to changes in the channel environment. This shows that as the channel conditions improve, the quality of images restored by high training signal-noise ratios will also be significantly improved.
[0124] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A source-channel joint encoding and decoding method based on a cross-modal communication system, characterized in that: include: Extracting image features and tactile features corresponding to the image signal and tactile signal to be transmitted through a cross-modal source encoder, and fusing the image features and the tactile features to obtain fused features; Design a cross-modal channel encoder and a cross-modal channel decoder, input the fused features into the cross-modal channel encoder, and convert them into a bit stream suitable for wireless transmission. After the channel transmission process is completed, the channel decoder maps the received bit stream back to the fused features. The fused features are decoded by a cross-modal source decoder and the image signal is reconstructed from the fused features.
2. The source-channel joint encoding and decoding method based on a cross-modal communication system according to claim 1, characterized in that: The cross-modal source encoder includes a tactile feature extraction module, an image feature extraction module and a modality fusion module; The loss function is designed to update the relevant parameters of the cross-modal source encoder; the loss function is designed to use the cross entropy loss function to classify image features, tactile features and fusion features, expressed as: in, Represents the tactile feature extraction network Image feature extraction and modal fusion Output image features, tactile features and fusion features, y j The label of the object. Represents the tactile feature extraction network Image feature extraction Modal Fusion and cross-modal source encoder integration network Parameters; By minimizing the loss function L s , perform preliminary training on the cross-modal source encoder parameters.
3. The source-channel joint encoding and decoding method based on a cross-modal communication system according to claim 2, characterized in that: The cross-modal channel encoder is expressed as, in, represents the parameters of the channel encoder model; The cross-modal channel encoder inputs the cross-modal fusion feature f and outputs the representation z of the feature mapped to the channel symbol.
4. The source-channel joint encoding and decoding method based on a cross-modal communication system according to claim 3, characterized in that: The channel is modeled as a neural network layer representing the process of the encoded fusion feature z being transmitted through the channel; The channel modeling includes an additive white Gaussian noise channel and a Rayleigh slow fading channel; In the AWGN channel, the channel transfer function is expressed as: y=z+n Where n is the channel noise, which is independent and identically distributed, and σ 2 is the average noise power; The signal-to-noise ratio of the channel is By changing σ 2 To change the signal-to-noise ratio of the channel; The transfer function of the Rayleigh slow fading channel is, y=hz+n Where h is the channel gain, n is the channel noise, h and n are subject to the parameters and Different normal distributions.
5. The source-channel joint encoding and decoding method based on a cross-modal communication system according to claim 4, characterized in that: The cross-modal channel decoder, after transmission through the channel, receives the complex valued signal interfered by noise The estimated value mapped back to the original feature input by the channel decoder Corresponding to the channel encoder, the channel decoder consists of a series of one-dimensional transposed convolutions, parameterized ReLU (PReLU) activation functions, and anti-normalization layers; The channel decoder is represented as, in, represents the parameters of the channel decoder model; The final channel codec is jointly designed and optimized by minimizing the cross-modal fusion feature f and the fusion feature estimated after transmission. The mean square error between them is used to jointly optimize the channel codec parameters, expressed as:
6. The source-channel joint coding and decoding method based on a cross-modal communication system according to claim 5, characterized in that: The cross-modal source decoder, at the receiving end, extracts the fused features from the Reconstructing the corresponding image signal; The Wasserstein generative adversarial network method is used to achieve image reconstruction. The discriminator D adopts the design of PatchGAN, and the generator G generates the fused features into an image. The discriminator D is responsible for distinguishing between real images v and generated images The loss function is defined as: in, 7. The source-channel joint encoding and decoding method based on a cross-modal communication system according to claim 6, characterized in that: Using knowledge distillation technology, the key distribution information of the generated image is extracted from the teacher model and applied to the student model; Select the VGG16 model pre-trained on the ImageNet dataset as the teacher model and the generator G as the student model; The generator G is derived from the fusion features To Image In the generation process of , each layer of the generator represents the information of different levels of the image, so the feature information of the lth layer is expressed as Represents the parameters of the lth layer; When the image is input into the VGG16 network, layers of different depths extract information at different levels. Deeper layers often represent deep information. The feature information of the mth layer is expressed as represents the parameters of the mth layer, and the layer-wise constraint loss is expressed as, The layers in the generator correspond to some layer structures in VGG16, all have the same feature output size, and calculate the layer-by-layer constraint loss. The loss function for the entire image generation is expressed as, L genv =L Gv +δL layerloss , Among them, δ is a hyperparameter used to balance the two loss functions.
8. The source-channel joint encoding and decoding method based on a cross-modal communication system according to claim 7, characterized in that: Performing training optimization based on a joint optimization algorithm, wherein the joint optimization algorithm includes a first stage and a second stage; The first stage pre-trains the network parameters of the cross-modal source encoder by minimizing the loss function, combines the cross-modal source encoder with the cross-modal source decoder for adversarial training, and conducts adversarial training with the discriminator, by calculating the loss function and performing gradient updates on the network parameters until the model converges; The second stage performs a gradient descent algorithm on the parameters of the channel encoder and the channel decoder by optimizing the loss function until the entire system reaches a stable state; After the above training process, the optimal cross-modal source encoder, cross-modal source decoder and channel encoder, and cross-modal channel decoder are obtained.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the source channel joint encoding and decoding method based on a cross-modal communication system according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the source channel joint encoding and decoding method based on a cross-modal communication system according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Visual-auditory cross-modal object material retrieval method and system
CN108520758A
Cross-modal image generation method and device based on audio-tactile signal fusion
CN113627482A
Image reconstruction method and device for cross-modal communication system
CN113628294A
Audio-visual assisted fine-grained tactile signal reconstruction method
CN115905838A
Cited By
Cross-modal semantic transmission method for mobile scene
CN120769298A
Semantic communication and classification method and device for multi-modal feature adaptive fusion and compression
CN121662059A
