A multi-modal lightweight semantic communication method

Abstract

The application discloses a kind of multimodal lightweight semantic communication methods, specific process is: in transmitting end, by encoder to the RGB image and infrared image of transmission are respectively encoded, then the first feature map and second feature map after encoder encoding are input to channel fusion module, in channel fusion module, after splicing, channel mix and separate operation, obtain two different mixed features, again by dimension reduction convolution layer processing, two modal dimension reduction features each after channel attention layer, dot product, spatial attention layer and addition, then carry out feature fusion, finally by dimension reduction convolution layer processing and superposition additive gaussian white noise, after reshaping shape, obtain the output of channel fusion module.In receiving end, first by three layers of dimension recovery processing of successive cascade, then after four convolution blocks are handled, finally after a layer of convolution layer, output semantic segmentation result.

Description

Technical Field

[0001] The present invention belongs to the field of semantic communication, and in particular relates to a multimodal lightweight semantic communication method. Background Art

[0002] With the growing demand for intelligent services, such as virtual reality (VR) and holographic communication (HC), increasing communication system capacity by simply increasing the dimensions of information transmission and changing the way resources are mined and utilized is approaching the Shannon limit. To meet this growing demand, semantic communication, an emerging intelligent communication technology, has emerged. The fundamental concept of semantic communication is to shift the focus of information transmission from the data itself to its meaning, or semantics. Semantic communication relies on advanced artificial intelligence technologies to perform semantic analysis and processing on data. Semantic communication filters the information sent by the transmitter and eliminates redundant information, transmitting only the most relevant information. It focuses on providing a better user experience (QoE) and ensuring that users can easily access the information they need.

[0003] However, the deep learning methods used in current semantic communication methods cannot effectively reduce and process the ubiquitous communication noise, and the model training used is very energy-consuming and the model size and number of training parameters are very large. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and provide a multimodal lightweight semantic communication method.

[0005] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0006] A multimodal lightweight semantic communication method, the specific steps are as follows:

[0007] S1. At the transmitting end, obtain an RGB image and an infrared image to be transmitted, input the RGB image to a first encoder to obtain a first feature map, and input the infrared image to a second encoder to obtain a second feature map. The RGB image and the infrared image have the same height and width.

[0008] S2. At the transmitting end, the first feature map and the second feature map are Figure 1The first feature map and the second feature map are input into the channel fusion module. In the channel fusion module, the first feature map and the second feature map are first spliced ​​together to obtain the spliced ​​features. The spliced ​​features are shuffled and separated to obtain two different mixed features. The first mixed feature is processed by the first dimensionality reduction convolution layer to obtain the third feature map. The third feature map is then passed through the first channel attention layer to obtain the fourth feature map. The second mixed feature is processed by the second dimensionality reduction convolution layer to obtain the fifth feature map. The fifth feature map is then passed through the second channel attention layer to obtain the sixth feature map. The third feature map and the sixth feature map are dot-producted to obtain the seventh feature map. The fifth feature map and the fourth feature map are dot-producted to obtain the eighth feature map. The eighth feature map and the third feature map are dot-producted to obtain the eighth feature map. Figure 1 Input to the first spatial attention layer, output the ninth feature map, and combine the seventh feature map and the fifth feature map Figure 1 The first feature map is input into the second spatial attention layer, and the tenth feature map is output. The eighth feature map and the ninth feature map are added to obtain the eleventh feature map. The seventh feature map and the tenth feature map are added to obtain the twelfth feature map. The eleventh feature map and the twelfth feature map are spliced ​​together to obtain a spliced ​​fusion feature map. The spliced ​​fusion feature map is processed by the third dimensionality reduction convolution layer to obtain the thirteenth feature map. After superimposing additive Gaussian white noise on the thirteenth feature map, a noisy feature map is obtained. After reshaping the shape of the noisy feature map, it is used as the feature map output by the channel fusion module.

[0009] S3. After the feature map output by the channel fusion module is sent to the receiving end, at the receiving end, the feature map output by the channel fusion module is input into the decoder, and first the dimensionality is restored by three layers of cascaded dimensionality-increasing convolutional layers to obtain the fourteenth feature map. The fourteenth feature map is then processed by four convolution blocks to obtain the fifteenth feature map. After the fifteenth feature map passes through the first convolutional layer, the semantic segmentation result is output.

[0010] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0011] Preferably, the first encoder uses MobileNetv3 as the backbone network for feature extraction.

[0012] Preferably, the second encoder uses MobileNetv3 as the backbone network for feature extraction.

[0013] Preferably, the convolution kernel size of the first dimensionality reduction convolution layer is 1×1.

[0014] Preferably, the convolution kernel size of the second dimensionality reduction convolution layer is 1×1.

[0015] Preferably, the convolution kernel size of the third dimensionality reduction convolution layer is 1×1.

[0016] Preferably, in the three-dimensionality-increasing convolutional layers of the decoder, the convolution kernel size of the first dimensionality-increasing convolutional layer is 3×3, the convolution kernel size of the second dimensionality-increasing convolutional layer is 3×3, and the convolution kernel size of the third dimensionality-increasing convolutional layer is 3×3.

[0017] Preferably, in the decoder, the convolution kernel size of the first convolutional layer is 1×1.

[0018] Preferably, in each convolution block of the decoder, the input feature map is first upsampled, and then the upsampled feature map is passed through a second convolution layer to obtain the output feature map of each convolution block.

[0019] Preferably, in each convolution block, the convolution kernel size of the second convolution layer is 3×3.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The method of the present invention can effectively reduce the energy consumed by model training, has strong anti-noise ability, and has a small number of parameters, which is conducive to deployment at the communication transmitting end. In addition, for the task of image segmentation in communication, the segmentation effect is good and the accuracy is high.

[0022] Compared with other image segmentation methods, the method of the present invention has fewer parameters and can meet semantic requirements, reduce the amount of data transmitted in communication, and has better image segmentation effect than other multimodal data transmission communication methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of the steps of the method of the present invention;

[0024] Figure 2 Schematic diagram of the overall implementation process of the method of the present invention;

[0025] Figure 3 Schematic diagram of experimental results of image segmentation accuracy of three data sets under different signal-to-noise ratio conditions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0027] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0028] The purpose of the present invention is to provide a lightweight, energy-saving, and noise-resistant semantic communication method. The method of the present invention is technically decomposed into three parts: encoding, channel fusion, and decoding. The encoding part and the channel fusion part are deployed at the transmitting end, while the decoding part is deployed at the receiving end, i.e., the base station. The RGB image and the infrared image are input to the corresponding branch of each modality, and MobileNetv3 is used as the backbone network for feature extraction. The encoder parameters of each branch are not shared. In the encoder of the infrared image branch, the number of input channels in the first convolutional layer is set to 1, and then four encoder layers are used continuously to extract features. The channel fusion module is used as the encoder for channel information fusion of the RGB-T image (RGB image and infrared image), further reducing the redundant signal information in the feature map, and fusing the feature map information of the two modalities. Taking into account the noise interference during transmission, the signal normalization noise reduction processing is performed, and the signal information is converted into a vector format that is easy to receive at the receiving end. Finally, multiple convolution and upsampling operations are used as a decoder, and the fused multimodal information feature map is used to restore the image signal, which is further transmitted to the classifier for classification. The convolution layer is used as a classifier to finally obtain the semantic segmentation result.

[0029] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned multimodal lightweight semantic communication method includes the following steps S1 to S3. The specific implementation process is described in detail below.

[0030] S1. At the transmitting end, obtain the RGB image and infrared image to be transmitted, input the RGB image to the first encoder to obtain a first feature map, and input the infrared image to the second encoder to obtain a second feature map. The RGB image and the infrared image have the same height and width.

[0031] In this embodiment, the first encoder uses MobileNetv3 as the backbone network for feature extraction, and the second encoder uses MobileNetv3 as the backbone network for feature extraction. The parameters of the first encoder and the second encoder are not shared.

[0032] S2. At the transmitting end, the first feature map and the second feature map are Figure 1The first feature map and the second feature map are input to the channel fusion module. In the channel fusion module, the first feature map and the second feature map are spliced ​​together to obtain the spliced ​​features. The spliced ​​features are shuffled and separated to obtain two different mixed features. The first mixed feature is processed by the first dimensionality reduction convolution layer to obtain the third feature map D′ rgb , then the third feature map D′ rgb Through the first channel attention layer, the fourth feature map M′ is obtained rgb , the second mixed feature is processed by the second dimensionality reduction convolution layer to obtain the fifth feature map D′ thr , then the fifth feature map D′ thr Through the second channel attention layer, the sixth feature map M′ is obtained thr , the third feature map D′ rgb and the sixth characteristic graph M′ thr Perform dot product to obtain the seventh feature map F′ rgb , the fifth feature map D′ thr and the fourth characteristic graph M′ rgb Perform dot product to obtain the eighth feature map F′ thr , the eighth feature map F′ thr and the third characteristic graph D′ rgb Input together into the first spatial attention layer and output the ninth feature map F″ thr , the seventh feature map F′ rgb and the fifth characteristic graph D′ thr Input together to the second spatial attention layer and output the tenth feature map F″ rgb , the eighth feature map F′ thr and the ninth characteristic graph F″ thr Add together to get the eleventh feature map N′ thr , the seventh feature map F′ rgb and the tenth characteristic graph F″ rgb Add together to get the twelfth feature map N′ rgb , the eleventh feature map N′ thr and the twelfth characteristic graph N′ rgb After splicing, a spliced ​​fused feature map is obtained. The spliced ​​fused feature map is processed by the third dimensionality reduction convolution layer to obtain the thirteenth feature map. After superimposing additive white Gaussian noise on the thirteenth feature map, a noisy feature map is obtained. After reshaping the shape of the noisy feature map, it is used as the feature map output by the channel fusion module.

[0033] In this embodiment, if Figure 2As shown, the RGB image (RGB input) and the infrared image (Infrared input) are the inputs of the two modal images. The blue background part is the process of feature extraction using MobileNetv3. Since the network has too many feature extraction layers, only the change process of the number of channels is shown. The green background part is the channel fusion part of the multimodal information, that is, the channel fusion module. In the process of channel fusion using the channel fusion module, the orange squares represent the feature maps of the RGB image, and the blue squares represent the feature maps of the infrared image. After the encoder part extracts the semantic information of RGB-T, the channel fusion module first splices the feature maps of the two modalities and shuffles the channels, and then separates them to obtain two different mixed features. This is the most preliminary fusion. The two mixed feature maps are each processed by a dimensionality reduction convolution layer to obtain the third feature map D′ rgb and the fifth characteristic graph D′ thr , the feature maps of the two branches each pass through a channel attention layer to obtain the fourth feature map M′ rgb and the sixth characteristic graph M′ thr The third feature map D′ rgb and the sixth characteristic graph M′ thr Perform dot product to obtain the seventh feature map F′ rgb , the same operation is performed on the fifth feature map D′ thr and the fourth characteristic graph M′ rgb Perform dot product to obtain the eighth feature map F′ thr The eighth feature map F′ thr and the third characteristic graph D′ rgb Input together into the first spatial attention layer and output the ninth feature map F″ thr , the seventh feature map F′ rgb and the fifth characteristic graph D′ thr Input together to the second spatial attention layer and output the tenth feature map F″ rgb In this embodiment, the implementation of the channel attention layer and the spatial attention layer belongs to the existing technology and will not be described in detail. thr and the ninth characteristic graph F″ thr Add together to get the eleventh feature map N′ thr , and then perform the same operation to convert the seventh feature map F′ rgb and the tenth characteristic graph F″ rgb Add together to get the twelfth feature map N′ rgb . The eleventh feature map N′ thr and the twelfth characteristic graph N′ rgbThe fused feature maps are then spliced ​​together to obtain a spliced ​​fusion feature map. This spliced ​​fusion feature map is then processed through the third dimensionality reduction convolution layer to obtain the thirteenth feature map. Additive white Gaussian noise is superimposed on the thirteenth feature map to obtain a noisy feature map. This noisy feature map is reshaped and used as the feature map output by the channel fusion module. This reshaping is performed to facilitate transmission over the wireless channel, and AWGN noise is added during transmission. Figure 2 In the above example, the two mixed feature maps are processed by a dimensionality reduction convolution layer to obtain the third feature map D′. rgb and the fifth characteristic graph D′ thr , which can be regarded as the first dimensionality reduction convolution operation, which is used to meet the needs of the communication transmitter; the spliced ​​fusion feature map is processed by the third dimensionality reduction convolution layer, which can be regarded as the second dimensionality reduction convolution operation, which is used to meet the needs of the communication receiver. Therefore, the number of channels should be reduced to reduce redundant information transmission.

[0034] It should also be noted that, in this embodiment, the convolution kernel size of the first dimensionality reduction convolution layer is 1×1, the convolution kernel size of the second dimensionality reduction convolution layer is 1×1, and the convolution kernel size of the third dimensionality reduction convolution layer is 1×1.

[0035] S3. After the feature map output by the channel fusion module is sent to the receiving end, at the receiving end, the feature map output by the channel fusion module is input into the decoder, and first the dimensionality is restored by three layers of cascaded dimensionality-increasing convolutional layers to obtain the fourteenth feature map. The fourteenth feature map is then processed by four convolution blocks to obtain the fifteenth feature map. After the fifteenth feature map passes through the first convolutional layer, the semantic segmentation result is output.

[0036] It should be noted that, in the decoder of the present invention, the convolution kernel size of the first convolutional layer is 1×1.

[0037] It should be noted that in the present invention, in each convolutional block of the decoder, the input feature map is first upsampled, and then the upsampled feature map is passed through a second convolutional layer to obtain the output feature map of each convolutional block. In addition, in each convolutional block, the convolution kernel size of the second convolutional layer is 3×3.

[0038] In this embodiment, if Figure 2 As shown in the figure, the purple background part is the decoding part, which undergoes dimensionality restoration through three layers of convolutional layers, then undergoes four layers of upsampling and convolution processing, and finally outputs through a convolutional layer to facilitate the prediction of subsequent segmentation results and finally obtain the semantic segmentation result.

[0039] It should also be noted that, in this embodiment, among the three dimensionality-increasing convolutional layers of the decoder, the convolution kernel size of the first dimensionality-increasing convolutional layer is 3×3, the convolution kernel size of the second dimensionality-increasing convolutional layer is 3×3, and the convolution kernel size of the third dimensionality-increasing convolutional layer is 3×3.

[0040] The present invention will now use a specific example to demonstrate the application effect of a multimodal lightweight semantic communication method described in S1 to S3 of the above embodiments on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0041] Example

[0042] The implementation method of this embodiment is as described in S1-S3 above, and the specific steps will not be elaborated in detail. The following only demonstrates its effect based on case data.

[0043] The present invention used computer simulations to conduct semantic communication experiments, setting the task to semantic segmentation using RGB and infrared images. The MFNet dataset was used. The MFNet dataset is a multimodal image dataset designed for semantic segmentation, consisting of 1,569 pairs of RGB and infrared urban scene images. The dataset is divided into nine categories, including unlabeled pixels and eight common obstacles in traffic environments. Each pair of RGB and infrared images is formatted as 480×640 pixels, with three channels for the RGB image and a single channel for the infrared image.

[0044] In order to verify the characteristics of this method with a small number of parameters and light weight, the embodiment of the present invention compares the number of parameters with several comparative methods, and uses four indicators: FLOPs (G), MACs (G), Params (M) and model size (in MB) for evaluation. The results are shown in Table 1. Among them, FLOPs (G) represents the number of floating point operations (FLOPs), which refers to the number of floating point operations performed during the model inference process, and the unit is billion times (G). MACs (G) represents the number of multiply-accumulate operations (MACs), which refers to the sum of the number of multiplication and addition operations required to be performed during the model inference process, and the unit is billion times (G). Params (M) represents the number of parameters (Params), which refers to the total number of trainable parameters in the model. Model size refers to the storage space required to save the model, and the unit is megabytes (MB). In Table 1, the method of the present invention (Mobilenetv2) indicates that the lightweight network model Mobilenetv2 is used as the encoder part of the present invention, and the method of the present invention (Mobilenetv3) indicates that the lightweight network model Mobilenetv3 is used as the encoder part of the present invention. The several comparative methods in Table 1 are MFNet, SpiderMesh-res50, and SpiderMesh-res152, respectively. Among them, the implementation of MFNet belongs to the prior art. For details, please refer to the prior art literature: Ha Q, Watanabe K, Karasawa T, et al. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes [C] / / 2017 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017: 5108-5115.The implementation of SpiderMesh-res50 and SpiderMesh-res152 belongs to the existing technology. For details, please refer to the existing technical literature: Fan S, Wang Z, Wang Y, et al. Spidermesh: Spatial-aware demand-guided recursive meshing for rgb-t semantic segmentation [J]. arXiv preprint arXiv: 2303.08692, 2023.

[0045] Table 1. Comparison of model parameters

[0046]

[0047] As can be seen from Table 1, the method of the present invention has the second smallest number of parameters, and the smallest is MFNet. However, the accuracy of image segmentation by the MFNet method is not as high as that of the method of the present invention.

[0048] Table 2. Segmentation results

[0049]

[0050] In addition, the segmentation performance of the present invention and the aforementioned comparative methods MFNet, SpiderMesh-res50, and SpiderMesh-res152 was compared on the MFNet dataset, with the results shown in Table 2. In Table 2, Class avg. (%) represents the class average accuracy and mIoU (%) represents the mean intersection over union. For the MFNet dataset, the present invention's method achieved significant improvements in the "Guardrail" and "Colored Cone" categories compared to MFNet and SpiderMesh. The present invention's method achieved significant improvements in certain challenging categories, such as a jump from 0 to 14.56 for "Guardrail."

[0051] Since many image segmentation algorithms do not take communication requirements into consideration, the amount of data transmitted after feature extraction is very large, which does not meet the requirements of semantic communication to reduce redundant information, but instead increases the amount of data transmitted. The communication content of traditional communication is only data transmission, and cannot complete the corresponding image segmentation task. The embodiment of the present invention compares the data compression ratios of the above-mentioned methods, and the results are shown in Table 3. Data compression ratio, this indicator is the ratio of the data size of the transmitted feature map after feature extraction to the data of the original image. Considering that other models have higher CR values, they are the result of using image pyramids for feature processing without considering communication requirements. These models give priority to improving spatial resolution at the expense of increasing the amount of data. The method of the present invention also achieved excellent performance, and the CR value was significantly reduced, which shows that the method of the present invention can maintain high-quality feature representation without generating a large amount of data, which is crucial in scenarios where communication bandwidth is limited or data transmission costs are high (remote sensing or Internet of Things devices). The method of the present invention has a lower CR value, which is particularly advantageous.

[0052] Table 3. Data compression ratio results

[0053]

[0054]

[0055] like Figure 3 Figure 2 shows experimental results of the image segmentation accuracy of our method on three datasets (MFNet, PST900, and MCubeS) under varying signal-to-noise ratios. Our method performs well on all three datasets, as the accuracy stabilizes with increasing signal-to-noise ratio, demonstrating the robustness of our method. Figure 3 In the figure, the horizontal axis is the signal-to-noise ratio, the vertical axis is the percentage, MFNet class avg represents the average accuracy of each class of image segmentation on the MFNet dataset, MFNet mIoU represents the average intersection-over-union ratio on the MFNet dataset, PST900 class avg represents the average accuracy of each class of image segmentation on the PST900 dataset, PST900 mIoU represents the average intersection-over-union ratio on the PST900 dataset, MCubeSclass avg represents the average accuracy of each class of image segmentation on the MCubeS dataset, and MCubeS mIoU represents the average intersection-over-union ratio on the MCubeS dataset.

[0056] The implementation methods of the three datasets mentioned above (MFNet, PST900, and MCubeS) are all existing technologies. The MFNet dataset is from: Ha Q, Watanabe K, Karasawa T, et al. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes [C] / / 2017 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017: 5108-5115. The PST900 dataset is from: Shivakumar SS, Rodrigues N, Zhou A, et al. Pst900: RGB-thermal calibration, dataset and segmentation network [C] / / 2020 IEEE international conference on robotics and automation (ICRA). IEEE, 2020: 9441-9447. MCubes dataset source: Liang Y, Wakaki R, Nobuhara S, et al. Multimodalmaterial segmentation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 19800-19808.

[0057] Table 4. Ablation experiment results

[0058] Ablation experiments Class avg.(%) mIoU (%) Baseline 70.24 64.66 +Scramble operation 69.42 65.54 +Spatial channel attention 70.76 66.17 Overall model 71.67 67.47

[0059] To explore the impact of channel fusion on segmentation accuracy, ablation experiments were conducted, and the results are shown in Table 4. The indicators are class average accuracy (Class avg.) and mean intersection over union (mIoU). As can be seen from Table 4, to study the effect of channel fusion, the embodiment of the present invention conducted four sets of ablation experiments. The first set is a baseline without the channel fusion module. Then, the second set adds a channel shuffling operation. The third set adds channel attention (CA) and spatial attention (SA) cross-calculation operations. Finally, the fourth set adds a channel fusion module to MMCFNet to make it more complete.

[0060] From the baseline, we found that the segmentation quality was improved at the expense of a slight decrease in overall classification accuracy. For the second group, this shows that the shuffling operation completed the segmentation task with higher accuracy, although its recognition accuracy for individual categories may have decreased. For the third group, we observed a small increase in the class average accuracy from 70.24% of the baseline to 70.76%, while the mIoU increased significantly from 64.66% to 66.17%. This result shows that the model can simultaneously use global information to enhance local features and use local features to improve global understanding. Finally, in the fourth group, our method outperformed the baseline method in class average by 1.43% and in mIoU by 2.81%, indicating that the fusion of multimodal information can be effectively improved by the channel fusion module. The CA and SA in this module can effectively improve the learning ability of the model based on the preliminary mixed information by using a feature crossover mechanism.

[0061] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A multimodal lightweight semantic communication method, characterized in that: The following steps are involved: S1. At the transmitting end, obtain an RGB image and an infrared image to be transmitted, input the RGB image to a first encoder to obtain a first feature map, and input the infrared image to a second encoder to obtain a second feature map. The RGB image and the infrared image have the same height and width. S2. At the transmitting end, the first feature map and the second feature map are input into the channel fusion module together. In the channel fusion module, the first feature map and the second feature map are first spliced ​​together to obtain the spliced ​​features. The spliced ​​features are shuffled and separated through channels to obtain two different mixed features. The first mixed feature is processed by the first dimensionality reduction convolution layer to obtain the third feature map. The third feature map is then passed through the first channel attention layer to obtain the fourth feature map. The second mixed feature is processed by the second dimensionality reduction convolution layer to obtain the fifth feature map. The fifth feature map is then passed through the second channel attention layer to obtain the sixth feature map. The third feature map and the sixth feature map are dot-producted to obtain the seventh feature map. The fifth feature map and the fourth feature map are shuffled and separated. Perform dot product to obtain the eighth feature map, input the eighth feature map and the third feature map together into the first spatial attention layer, output the ninth feature map, input the seventh feature map and the fifth feature map together into the second spatial attention layer, output the tenth feature map, add the eighth feature map and the ninth feature map to obtain the eleventh feature map, add the seventh feature map and the tenth feature map to obtain the twelfth feature map, splice the eleventh feature map and the twelfth feature map to obtain a spliced ​​fusion feature map, process the spliced ​​fusion feature map through the third dimensionality reduction convolution layer to obtain the thirteenth feature map, superimpose additive Gaussian white noise on the thirteenth feature map to obtain a noisy feature map, reshape the noisy feature map, and use it as the feature map output by the channel fusion module; S3. After the feature map output by the channel fusion module is sent to the receiving end, at the receiving end, the feature map output by the channel fusion module is input into the decoder, and first the dimensionality is restored by three layers of cascaded dimensionality-increasing convolutional layers to obtain the fourteenth feature map. The fourteenth feature map is then processed by four convolution blocks to obtain the fifteenth feature map. After the fifteenth feature map passes through the first convolutional layer, the semantic segmentation result is output.

2. A multimodal lightweight semantic communication method according to claim 1, characterized in that: The first encoder uses MobileNetv3 as the backbone network for feature extraction.

3. A multimodal lightweight semantic communication method according to claim 1, characterized in that: The second encoder uses MobileNetv3 as the backbone network for feature extraction.

4. A multimodal lightweight semantic communication method according to claim 1, characterized in that: The convolution kernel size of the first dimensionality reduction convolution layer is 1×1.

5. A multimodal lightweight semantic communication method according to claim 1, characterized in that: The convolution kernel size of the second dimensionality reduction convolution layer is 1×1.

6. A multimodal lightweight semantic communication method according to claim 1, characterized in that: The convolution kernel size of the third dimensionality reduction convolution layer is 1×1.

7. A multimodal lightweight semantic communication method according to claim 1, characterized in that: In the three layers of dimensionality-increasing convolutional layers of the decoder, the convolution kernel size of the first dimensionality-increasing convolutional layer is 3×3, the convolution kernel size of the second dimensionality-increasing convolutional layer is 3×3, and the convolution kernel size of the third dimensionality-increasing convolutional layer is 3×3.

8. A multimodal lightweight semantic communication method according to claim 1, characterized in that: In the decoder, the convolution kernel size of the first convolutional layer is 1×1.

9. A multimodal lightweight semantic communication method according to claim 1, characterized in that: In each convolution block of the decoder, the input feature map is first upsampled, and then the upsampled feature map passes through a second convolution layer to obtain the output feature map of each convolution block.

10. A multimodal lightweight semantic communication method according to claim 9, characterized in that: In each convolution block, the convolution kernel size of the second convolution layer is 3×3.

Images