Method and device for multi-modal generative semantic communication oriented to perception data backhaul

Abstract

The application provides a multi-modal generative semantic communication method and device for perception data backhaul, and the method comprises the following steps: acquiring compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a one-hot encoded semantic segmentation graph; decompressing the compressed image data to obtain the one-hot encoded semantic segmentation graph; inputting the one-hot encoded semantic segmentation graph and a modal category into a pre-trained latent space diffusion model, wherein the modal category comprises a visible light modal and an infrared modal; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model; and the application can improve the communication efficiency of perception data in an emergency environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, and in particular to a multi-modal generative semantic communication method and device for perception data backhaul. BACKGROUND

[0002] Under the emergency environment, the infrastructure is often damaged and out of service, leading to the interruption of public networks. The unmanned aerial vehicle backhaul perception data needs to rely on the temporary construction of emergency private networks, which has the problems of unstable communication link and limited transmission bandwidth, and cannot guarantee the timely backhaul of perception data.

[0003] With the continuous progress of unmanned aerial vehicle technology, the load capacity and flight time are improved, so that the unmanned aerial vehicle can carry multiple different modal perception devices at the same time, such as visible light and infrared perception devices, which can effectively improve the perception ability of the unmanned aerial vehicle in complex emergency environments. However, carrying multiple different modal perception devices leads to the need to backhaul more perception data, which brings greater challenges to the completion of perception data backhaul under the condition of communication capability limited private network.

[0004] Therefore, under the condition of communication capability limited private network in the related art, there is the technical problem of low communication efficiency caused by too much perception data. SUMMARY

[0005] The present application provides a multi-modal generative semantic communication method and device for perception data backhaul, which solves the defect of low communication efficiency caused by too much perception data under the condition of communication capability limited private network in the prior art, and improves the communication efficiency of perception data in the emergency environment.

[0006] The present application provides a multi-modal generative semantic communication method for perception data backhaul, comprising the following steps. Obtain compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a one-hot encoded semantic segmentation map; decompress the compressed image data to obtain the one-hot encoded semantic segmentation map; input the one-hot encoded semantic segmentation map and a modal category into a pre-trained latent space diffusion model, wherein the modal category includes a visible light modal and an infrared modal; output a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; output a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0007] According to the multi-modal generative semantic communication method for perception data backhaul provided by the application, before the compressed image data transmitted by the sending end is acquired, the method further comprises: calling the sending end to collect image data to obtain a visible light image and an infrared image, wherein the sending end is a drone, and the image information of the visible light image and the image information of the infrared image are consistent in time dimension and space dimension; calling a semantic segmentation model of the sending end to perform semantic extraction on the visible light image and the infrared image to obtain a semantic segmentation graph, wherein the semantic segmentation graph comprises common semantic features of the visible light image and the infrared image; calling the sending end to perform one-hot encoding on the semantic segmentation graph to obtain a one-hot encoded semantic segmentation graph; and calling the sending end to compress the one-hot encoded semantic segmentation graph to obtain compressed image data.

[0008] According to the multi-modal generative semantic communication method for perception data backhaul provided by the application, before the compressed image data transmitted by the sending end is acquired, the method further comprises: calling the sending end to collect image data to obtain a visible light image and an infrared image, wherein the sending end is a drone, and the image information of the visible light image and the image information of the infrared image are consistent in time dimension and space dimension; calling a semantic segmentation model of the sending end to perform semantic extraction on the visible light image and the infrared image to obtain a semantic segmentation graph, wherein the semantic segmentation graph comprises common semantic features of the visible light image and the infrared image; calling the sending end to perform one-hot encoding on the semantic segmentation graph to obtain a one-hot encoded semantic segmentation graph; and calling the sending end to compress the one-hot encoded semantic segmentation graph to obtain compressed image data.

[0009] According to the multi-modal generative semantic communication method for perception data backhaul provided by the application, the visible light autoencoder and the infrared autoencoder of the preset latent space diffusion model are trained based on visible light image samples and infrared image samples, a first stage training loss function is obtained, and the first stage training loss function includes the following steps: obtaining visible light image samples and infrared image samples; projecting the visible light image samples to a latent space through an encoder in the visible light autoencoder of the preset latent space diffusion model to obtain latent space features of the visible light image, and then reconstructing the latent space features of the visible light image into a visible light image through a corresponding visible light decoder to obtain a visible light perception loss generated in the encoding and decoding process; projecting the infrared image samples to the latent space through an encoder in the infrared autoencoder of the preset latent space diffusion model to obtain latent space features of the infrared image, and then reconstructing the latent space features of the infrared image into an infrared image through a corresponding infrared decoder to obtain an infrared perception loss generated in the encoding and decoding process; determining a contrast learning loss between the latent space features of the visible light image and the latent space features of the infrared image; and determining the first stage training loss function based on the visible light perception loss, the infrared perception loss, and the contrast learning loss.

[0010] According to the multi-modal generative semantic communication method for perception data backhaul provided by the application, the diffusion model of the preset latent space diffusion model is trained based on the one-hot encoded semantic segmentation image samples and the modal categories, a second stage training loss function is obtained, and the second stage training loss function includes the following steps: obtaining one-hot encoded semantic segmentation image samples, wherein the one-hot encoded semantic segmentation image samples are based on the visible light image samples and the infrared image samples; taking the one-hot encoded semantic segmentation image samples and the modal categories as condition inputs of the preset latent space diffusion model at the same time to obtain visible light modal latent space features and infrared modal latent space features output by the diffusion model of the preset latent space diffusion model; determining a mean square error loss of the visible light modal based on the visible light modal latent space features and visible light modal feature true values; determining a mean square error loss of the infrared modal based on the infrared modal latent space features and infrared modal feature true values; and determining the second stage training loss function based on the mean square error loss of the visible light modal and the mean square error loss of the infrared modal.

[0011] According to the multi-modal generative semantic communication method for perception data backhaul provided by the application, the sending end is a UAV, and the UAV includes a visible light perception device and an infrared perception device.

[0012] The present invention also provides a multimodal generative semantic communication device for perception data feedback, comprising the following modules: an acquisition module for acquiring compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding; a compression module for decompressing the compressed image data to obtain the semantic segmentation map after one-hot encoding; an input module for inputting the semantic segmentation map after one-hot encoding and the modality category into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality; a visible light output module for outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; and an infrared output module for outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multimodal generative semantic communication method for perception data feedback as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal generative semantic communication methods for perception data feedback as described above.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the multimodal generative semantic communication methods for perception data feedback as described above.

[0016] The multimodal generative semantic communication method and device for perception data feedback provided by the present invention obtains compressed image data transmitted by the sending end, decompresses the compressed image data, obtains a one-hot encoded semantic segmentation map, inputs the one-hot encoded semantic segmentation map and the modal category into a pre-trained latent space diffusion model, and then obtains a reconstructed visible light image output by a visible light decoder and a reconstructed infrared image output by an infrared decoder. Thus, the semantic segmentation map and the modal category are used as conditional parameters of the diffusion model at the same time, so that a single diffusion model can reconstruct data of different modalities, effectively reducing the deployment overhead in emergency scenarios and improving the perception data communication efficiency in emergency environments; thereby solving the technical problem of low communication efficiency under the conditions of a private network with limited communication capabilities in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced one by one as follows. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative labor based on these accompanying drawings are within the protection scope of the present application.

[0018] Figure 1 is a flow diagram of the multi-modal generative semantic communication method for perception data backhaul provided by the present application.

[0019] Figure 2 is a structural diagram of the multi-modal generative semantic communication framework provided by the present application.

[0020] Figure 3 is a flow diagram of the first-stage autoencoder training process provided by the present application.

[0021] Figure 4 is a flow diagram of the second-stage diffusion model training process provided by the present application.

[0022] Figure 5 is a diagram of the reconstruction effect of the proposed scheme and GESCO, U-deepSC provided by the present application.

[0023] Figure 6 is a structural diagram of the multi-modal generative semantic communication device for perception data backhaul provided by the present application.

[0024] Figure 7 is a structural diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the protection scope of the present application.

[0026] It is crucial to obtain the disaster situation of the emergency scene in time for the successful implementation of emergency rescue operations. In complex emergency rescue environments, the sensing device is mounted on the unmanned aerial vehicle (UAV) to quickly deploy the UAV to the disaster site, collect on-site information, and transmit the sensing data back to the rear command center. However, in emergency environments, the infrastructure is often damaged, leading to the interruption of public networks. The UAV needs to rely on the temporary emergency private network to return the sensing data, which has the problems of unstable communication link and limited transmission bandwidth, and cannot guarantee the timely return of the sensing data. In addition, the visible light sensing device carried by the UAV often faces the problem of obstruction by buildings, trees, and smoke. The single sensing device has limited capabilities.

[0027] With the continuous progress of UAV technology, the improvement of load capacity and flight time enables the UAV to carry multiple different modal sensing devices at the same time, such as visible light and infrared sensing devices, which can effectively improve the sensing capability of the UAV in complex emergency environments. However, carrying multiple different modal sensing devices leads to the need to return more sensing data, which brings greater challenges to the communication capability-limited private network conditions for completing the sensing data return.

[0028] The development of deep learning technology brings new ideas for sensing data return. A semantic communication framework is constructed, and deep learning technology is used at the sending end to extract specific semantic information from the collected different modal data. By transmitting only the key semantic information related to the task, the demand for bandwidth can be significantly reduced. With the advent of generative artificial intelligence models such as variational auto-encoder (VAE), generative adversarial networks (GAN), and denoising diffusion probabilistic model (DDPM), the receiving end can reconstruct the original sensing data to some extent based on the received semantic information, and further use the reconstructed data for image classification, depth estimation, and other downstream tasks. However, existing semantic communication research mainly focuses on single-modal semantic communication, which leads to the need to deploy multiple different semantic communication frameworks in complex multi-task scenarios, resulting in additional overhead. To address this issue, further research is needed on multi-modal generative semantic communication frameworks to achieve a single semantic communication framework that can meet multiple application requirements.

[0029] Therefore, the main problem of constructing a multi-modal generative semantic communication framework is that: on the one hand, the traditional semantic communication framework is designed for a single modality, and in a complex emergency environment with multiple modalities and multiple tasks, multiple semantic communication frameworks need to be deployed at the same time, which brings additional resource overhead. On the other hand, the features of different modal data are often not aligned, making it difficult for a single generative model to complete the reconstruction of different modal data.

[0030] To solve the above problems, the present application provides a multi-modal generative semantic communication framework, which can extract and transmit a fused semantic segmentation map from visible light and infrared images. At the receiving end, it reconstructs the data into visible light and infrared images according to the semantic segmentation map. By using one-hot encoding and zlib compression, the present application achieves a compression ratio of nearly 200 times for a single semantic segmentation map, significantly improving the success rate of perception data backhaul in emergency environments.

[0031] The present application introduces a latent diffusion model (LDM), which is composed of a pair of autoencoders and a diffusion model. By adding a contrastive learning loss to the autoencoder, the features of visible light and infrared modal data are aligned in the latent space. In addition, the semantic segmentation map and the modal information are used as the conditional parameters of the diffusion model, so that a single diffusion model can reconstruct data of different modalities, effectively reducing the deployment overhead in emergency scenarios.

[0032] According to the present application, a multi-modal generative semantic communication framework is provided. At the sending end, the framework can extract a modal fusion semantic segmentation map from the collected visible light and infrared image data. At the receiving end, the framework can reconstruct the original visible light or infrared image based on the fused semantic segmentation map. Experimental results show that the framework achieves a compression ratio of up to 200 times and exhibits excellent performance in downstream tasks such as target classification and detection.

[0033] Optionally, the multi-modal generative semantic communication method for perception data backhaul according to the present application can be executed by the receiving end as described above, and can also be executed by the sending end (e.g., a UAV device) and the receiving end together. For example, the receiving end (server or terminal) executes the multi-modal generative semantic communication method for perception data backhaul in the present embodiment.

[0034] Figure 1 is a flowchart of the multi-modal generative semantic communication method for perception data backhaul provided by the present application, as shown in Figure 1 The method comprises the following steps.

[0035] Step 101, acquiring compressed image data transmitted by the sending end, wherein the compressed image data is obtained by compressing the one-hot encoded semantic segmentation map.

[0036] In the embodiment of the present application, the sending end can be a drone equipped with an infrared sensing device and a visible light sensing device.

[0037] Considering that the data of different modalities are collected by a single drone at the same time, the information contained in the visible light image and the infrared image is consistent in time and space; the semantic segmentation model in the sending end is used to perform semantic extraction on the visible light image and the infrared image to obtain a semantic segmentation map; and then the semantic segmentation map is one-hot encoded and compressed to obtain compressed image data.

[0038] Here, after the semantic segmentation map is one-hot encoded and converted into binary format data, the zlib method is used for compression. For a semantic segmentation map, the classification (label) of each pixel is usually converted into a one-hot encoding vector, and then the vector is converted into binary format for storage or transmission. Using zlib compression can significantly reduce the size of these data, especially when the number of labels is large or the image resolution is high.

[0039] Among them, zlib is a widely used compression library that provides data compression and decompression functions.

[0040] Through the embodiment of the present application, common semantic segmentation maps can be extracted from two modal images by using a multi-modal fusion semantic segmentation model, thereby avoiding extracting separate semantic segmentation maps for each modality and effectively reducing the calculation and transmission overhead. In addition, the semantic segmentation map can be one-hot encoded, and the transmission data size can be further compressed by combining compression technology.

[0041] Step 102, decompressing the compressed image data to obtain the one-hot encoded semantic segmentation map.

[0042] Step 103, inputting the one-hot encoded semantic segmentation map and the modality category to the pre-trained latent space diffusion model, wherein the modality category includes the visible light modality and the infrared modality.

[0043] In the embodiment of the present application, for the receiving end, in order to reconstruct the original modality from the fused semantic data (one-hot encoded semantic segmentation map), the present application proposes a multi-modal diffusion model based on the LDM model to realize the reconstruction of the visible light and infrared two modal data. Considering that the receiving end receives the one-hot encoded semantic segmentation map after fusion, only inputting this data as the condition of the diffusion model cannot realize the generation of different modalities. Therefore, the present application inputs the modality category as a condition into the diffusion model, and realizes the generation of different modal data by using different modality decoders.

[0044] Step 104, outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model.

[0045] Step 105, outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0046] In the embodiments of the present application, the pre-trained latent space diffusion model is pre-trained, and the visible light decoder and the infrared decoder can recover images (i.e., visible light images and infrared images) from the latent space.

[0047] In the process of decoding and reconstructing the visible light image by using the pre-trained latent space diffusion model (Latent Diffusion Model, LDM), the main concern is to map the latent representation in the latent space back to the original image space. The pre-trained latent space diffusion model usually operates on a high-dimensional latent space and gradually restores the image through a series of denoising steps.

[0048] For example, the decoding process can be to input the one-hot encoded semantic segmentation map and the modal category (visible light or infrared) into the diffusion model as a conditional parameter, use the diffusion model to denoise a Gaussian distribution random feature vector, gradually restore the latent space image features (involving multiple iterations), and reduce noise and enhance feature signals that meet the conditional parameters at each iteration; after completing all denoising steps, the latent representation of the visible light or infrared image is obtained, and the corresponding decoder is used to output and reconstruct the visible light or infrared image.

[0049] Reference Figure 2 , Figure 2 is a structural schematic diagram of a multi-modal generative semantic communication framework provided by the present application, which includes a sending end and a receiving end.

[0050] The sending end includes: collected visible light (image) and infrared (image), inputting the visible light image and the infrared image into a semantic segmentation model to obtain a semantic segmentation map, one-hot encoding the semantic segmentation map to obtain a one-hot encoded semantic segmentation map, and then compressing the one-hot encoded semantic segmentation map.

[0051] The receiving end includes: decompressing the compressed image data to obtain the one-hot encoded semantic segmentation map, inputting the one-hot encoded semantic segmentation map and the modal (category) as conditional parameters into a diffusion model in the latent space to obtain latent variables (z t) corresponding to time steps (t), and inputting the latent variables (z t) corresponding to the time steps (t) into a decoder corresponding to the modal (category) to obtain a reconstructed image corresponding to the modal (category). , , , ), and output the visible light modal latent space feature and the infrared modal latent space feature, and input the visible light modal latent space feature and the infrared modal latent space feature into the infrared decoder and the visible light decoder respectively, and output the reconstructed infrared image and the reconstructed visible light image.

[0052] Through the above steps of the embodiment of the present application, the compressed image data transmitted by the sending end is obtained, wherein the compressed image data is obtained by compressing the one-hot encoded semantic segmentation map; the one-hot encoded semantic segmentation map is decompressed to obtain the one-hot encoded semantic segmentation map; the one-hot encoded semantic segmentation map and the modal category are input into the pre-trained latent space diffusion model, wherein the modal category includes the visible light modal and the infrared modal; the visible light decoder of the pre-trained latent space diffusion model is used to output the reconstructed visible light image; the infrared decoder of the pre-trained latent space diffusion model is used to output the reconstructed infrared image; thereby, the semantic segmentation map and the modal category are used as the conditional parameters of the diffusion model at the same time, so that a single diffusion model can reconstruct data of different modalities, effectively reducing the deployment overhead in an emergency scenario and improving the communication efficiency of perception data in an emergency environment; and further, the technical problem of low communication efficiency in a communication capability limited private network in the related art is solved.

[0053] According to the multi-modal generative semantic communication method for perception data backhaul provided by the present application, before obtaining the compressed image data transmitted by the sending end, the above method further comprises:

[0054] The sending end is called to collect image data to obtain a visible light image and an infrared image, wherein the sending end is a UAV, and the image information of the visible light image and the image information of the infrared image are consistent in time and space dimensions;

[0055] The semantic segmentation model of the sending end is called to perform semantic extraction on the visible light image and the infrared image to obtain a semantic segmentation map, wherein the semantic segmentation map includes common semantic features of the visible light image and the infrared image;

[0056] The sending end is called to perform one-hot encoding on the semantic segmentation map to obtain a one-hot encoded semantic segmentation map;

[0057] The sending end is called to compress the one-hot encoded semantic segmentation map to obtain compressed image data.

[0058] In the embodiment of the present application, for the sending end, considering that the data of different modalities is collected by a single UAV at the same time, the information contained in the visible light image and the infrared image is consistent in time and space, so the multi-modal fusion semantic segmentation model can be used to extract a common semantic segmentation map from the two modal images, thereby avoiding extracting a separate semantic segmentation map for each modality, and effectively reducing the calculation and transmission overhead.

[0059] In addition, the semantic segmentation map can be One-Hot encoded, and the transmission data size can be further compressed by combining compression technology. In particular, since the present application focuses on how to achieve efficient data compression through semantic encoding, reduce the size of the transmission data and achieve high-quality reconstruction at the receiving end, the impact of noise during wireless transmission is not considered. Considering that the transmitted data is small enough, channel coding technology can be further combined to improve the data backhaul efficiency in emergency scenarios.

[0060] In the embodiment of the present application, a multi-modal fusion semantic segmentation model is used to extract fused semantic information from visible light and infrared data. The present application adopts an MFNet model designed for visible light and infrared data, and the model is lightweight and suitable for deployment on the computing device carried by the unmanned aerial vehicle. After processing by the MFNet model, the fused semantic segmentation map is first One-Hot encoded, and the One-Hot encoded data is used as the conditional input of the diffusion model, which can effectively improve the image reconstruction performance. Then, the encoded data is converted into binary format and compressed using the zlib method.

[0061] In the embodiment of the present application, in the semantic segmentation task, each pixel point is assigned a class label indicating which class (such as road, vehicle, pedestrian, etc.) the pixel point belongs to. The One-Hot encoded semantic segmentation map converts the class label of each pixel point into a vector, the length of the vector is equal to the total number of classes, and only one element is 1 (indicating that the pixel point belongs to that class), and the remaining elements are 0.

[0062] According to the multi-modal generative semantic communication method for perception data backhaul provided by the present application, before the One-Hot encoded semantic segmentation map and the modal class are input into the pre-trained latent space diffusion model, the above method further comprises:

[0063] Based on the visible light image sample and the infrared image sample, the visible light autoencoder and the infrared autoencoder of the pre-set latent space diffusion model are trained to obtain a first stage training loss function;

[0064] Based on the One-Hot encoded semantic segmentation map sample and the modal class, the diffusion model of the pre-set latent space diffusion model is trained to obtain a second stage training loss function;

[0065] Based on the first stage training loss function and the second stage training loss function, the pre-set latent space diffusion model is updated to obtain a pre-trained latent space diffusion model.

[0066] The core of the framework of the present application is an LDM model, which can generate a high-quality image from a noisy sample (x, y) and a modal class (c) as input. ) Start, step by step to remove noise and complete feature reconstruction in the latent space. Since the reconstruction is carried out in the latent space, compared with the denoising diffusion probabilistic model (DDPM, Denoising Diffusion Probabilistic Model), the model training and inference efficiency can be significantly improved without reducing the reconstruction quality.

[0067] In addition, the present application extends the two-stage training mechanism of the LDM model, so that the LDM model can support the reconstruction of multi-modal data. Training the LDM model requires training an autoencoder first, where the encoder maps the original image to the latent space, and the decoder can recover the image from the latent space. Then, a diffusion model needs to be trained to realize feature reconstruction in the latent space. In order to realize multi-modal reconstruction, the present application trains an autoencoder for each modal data (visible light modal, infrared modal), and at the same time, the one-hot encoded semantic segmentation map and modal category are input into the diffusion model as conditional input, so as to realize the reconstruction of different modal data.

[0068] According to the multi-modal generative semantic communication method for perception data backhaul provided by the present application, based on the visible light image sample and the infrared image sample, the visible light autoencoder and the infrared autoencoder of the preset latent space diffusion model are trained to obtain a first stage training loss function, which includes:

[0069] Obtain the visible light image sample and the infrared image sample;

[0070] Project the visible light image sample to the latent space through the encoder in the visible light autoencoder of the preset latent space diffusion model to obtain the latent space feature of the visible light image, and then reconstruct the latent space feature of the visible light image into a visible light image through the corresponding visible light decoder to obtain the visible light perception loss generated in the encoding and decoding process;

[0071] Project the infrared image sample to the latent space through the encoder in the infrared autoencoder of the preset latent space diffusion model to obtain the latent space feature of the infrared image, and then reconstruct the latent space feature of the infrared image into an infrared image through the corresponding infrared decoder to obtain the infrared perception loss generated in the encoding and decoding process;

[0072] Determine the contrast learning loss between the latent space feature of the visible light image and the latent space feature of the infrared image;

[0073] Based on the visible light perception loss, the infrared perception loss and the contrast learning loss, determine the first stage training loss function.

[0074] Reference Figure 3 , Figure 3is a flowchart of a first-stage self-encoder training process provided by the present application, which comprises: input images (visible light image samples and infrared image samples), a visible light encoder, an infrared encoder, a visible light decoder, an infrared decoder, a visible light perception loss, an infrared perception loss, and a contrastive learning loss.

[0075] In the embodiment of the present application, in the first-stage training process, the visible light self-encoder and the infrared modal self-encoder are trained before the diffusion model is trained. The visible light self-encoder and the infrared modal self-encoder are intended to project the data of each modal into a unified hidden space. As shown in Figure 3 The present application adds a contrastive learning loss to the training process. Therefore, the loss function definition of the first-stage training can refer to the following formula (1):

[0076] LAE=Lc+LRGB+Linfrared(1)

[0077] Wherein, LAE represents the first-stage training loss function, Lc represents the contrastive learning loss, LRGB represents the visible light perception loss, and Linfrared represents the infrared perception loss. AE C RGB

[0078] In the embodiment of the present application, the perception loss used in the conventional single-modal LDM training process is consistent. By encoding the visible light and infrared data into the same hidden space through the respective encoders, the present application optimizes the performance of the LDM model when processing multi-modal data, and maintains the semantic consistency of the reconstructed data.

[0079] According to the multi-modal generative semantic communication method for perception data backhaul provided by the present application, the diffusion model of the preset hidden space diffusion model is trained based on the one-hot encoded semantic segmentation graph sample and the modal category, and a second-stage training loss function is obtained, which comprises:

[0080] The one-hot encoded semantic segmentation graph sample is obtained, wherein the one-hot encoded semantic segmentation graph sample is based on the visible light image sample and the infrared image sample.

[0081] The one-hot encoded semantic segmentation graph sample and the modal category are simultaneously input into the preset hidden space diffusion model as conditions to obtain the visible light modal hidden space feature and the infrared modal hidden space feature output by the diffusion model of the preset hidden space diffusion model.

[0082] Based on the visible light modal hidden space feature and the visible light modal feature true value, the mean square error loss of the visible light modal is determined.

[0083] ​​​Determine the mean square error loss of the infrared modal based on the infrared modal hidden space feature and the infrared modal feature true value.

[0084] Determine the second stage training loss function based on the mean square error loss of the visible light modal and the mean square error loss of the infrared modal.

[0085] In the embodiment of the present application, after the first stage visible light autoencoder training and infrared autoencoder training are completed, the original visible light and infrared images are input into the respective encoders to obtain the visible light and infrared image hidden space features as the true value in the second stage training, and the mean square error is calculated with the visible light and infrared image hidden space features output by the diffusion model.

[0086] Reference Figure 4 , Figure 4 is a flowchart of the second stage diffusion model training process provided by the present application.

[0087] As Figure 4 shown, the visible light image sample and the infrared image sample are input into the visible light encoder and the infrared encoder respectively to obtain the visible light modal feature true value and the infrared modal feature true value; the semantic segmentation graph sample is One-Hot coded to obtain the One-Hot coded semantic segmentation graph sample, and the modal (category) and the One-Hot coded semantic segmentation graph sample are respectively taken as the condition input of the preset hidden space diffusion model, to obtain the hidden variable (z , , , ) corresponding to the time step, output the visible light modal hidden space feature and the infrared modal hidden space feature, and then combine the visible light modal feature true value and the infrared modal feature true value to determine the mean square error loss (MSE loss) of the visible light modal and the mean square error loss (MSE loss) of the infrared modal, as the second stage training loss function.

[0088] Although the present application introduces a contrast learning loss in the autoencoder to align the features between different modalities as much as possible, the generation of multi-modal data using a single diffusion model still needs further optimization.

[0089] As Figure 4As shown in the figure, the present invention uses the encoded semantic segmentation map and modality category as conditional inputs of the diffusion model so that the diffusion model can generate corresponding latent space features for the visible light and infrared modalities respectively. Because both visible light and infrared modalities use the same semantic segmentation map, in order to ensure the reconstruction accuracy and consistency between the modalities, the loss function used in the second stage of training is the mean square error (MSE) loss between the latent space features generated by each modality and its corresponding true value (obtained by encoding the original image by the corresponding encoder). The MSE losses of the two modalities are then summed to optimize the model training effect, which can be specifically expressed by the following formula (2):

[0090] (2)

[0091] in, Represents the loss function of the second stage training (LDM model), Encode the original image for each modality corresponding to the autoencoder The features obtained. is the reconstruction result of the diffusion model, with the encoded semantic segmentation map and modal category as conditional input, is the time step. For the modality category, 0 represents visible light data and 1 represents infrared data. Using this loss function to train the diffusion model enhances the model's ability to handle multimodal data.

[0092] According to a multimodal generative semantic communication method for perception data backhaul provided by the present invention, the transmitting end is a drone, wherein the drone includes a visible light perception device and an infrared perception device.

[0093] For the sending end, considering that the data of different modalities are collected simultaneously by a single drone, the information contained in the visible light image and the infrared image is consistent in time and space, so the multimodal fusion semantic segmentation model can be used to extract a common semantic segmentation map from the two modal images, thereby avoiding the extraction of a separate semantic segmentation map for each modality, effectively reducing the computational and transmission overhead. In addition, the semantic segmentation map can be one-hot encoded, and the size of the transmitted data can be further compressed in combination with compression technology. In particular, since the present invention focuses on how to achieve efficient data compression through semantic coding, reduce the size of the transmitted data and achieve high-quality reconstruction at the receiving end, the impact of noise during wireless transmission is not considered. Considering that the transmitted data is small enough, it can be further combined with channel coding technology to improve the efficiency of data backhaul in emergency scenarios.

[0094] The following describes an example of a multimodal generative semantic communication method for perception data feedback provided by the present invention in a practical application scenario.

[0095] To evaluate the performance of the proposed framework, the present application uses the traditional indicators LPIPS (learned perceptual image patch similarity) and FID (fréchet inception distance) for image reconstruction quality, on the one hand, and image classification and object detection as downstream tasks, on the other hand, to evaluate the performance of the proposed framework.

[0096] The present application verifies the performance of the proposed scheme through comparative experiments and ablation experiments respectively, and the present application uses the Multi-spectral Semantic Segmentation Dataset published together with the MFNet model, which contains 1,569 visible and infrared images of urban streets. This dataset was originally designed for the semantic segmentation task, providing semantic segmentation maps as ground truth, in which eight common urban street obstacles (cars, pedestrians, bicycles, curves, parking signs, guardrails, colored cones and speed bumps) are labeled. Since the dataset provides visible light images, infrared images and semantic segmentation maps, it is very suitable for training the proposed multi-modal generative semantic communication framework.

[0097] To evaluate the performance of the proposed framework, the present application uses the FID index (a measure of the effectiveness of the generation model) and the LPIPS index (a deep learning-based image similarity evaluation index) in the image reconstruction task. The FID index is closer to human judgment, while the LPIPS score mainly measures perceptual similarity. The lower the value of these two indicators, the higher the similarity between the reconstructed image and the original image, indicating better image reconstruction performance.

[0098] In addition, the present application evaluates the performance of the proposed framework in image classification and object detection tasks. The present application uses a pre-trained yolov8 model to detect pedestrians in images before and after reconstruction, and quantifies the performance of the proposed framework in downstream tasks by calculating the accuracy of the classification label and the average intersection over union (mIoU) of the output pedestrian detection box.

[0099] Reference Figure 5 , Figure 5 is the reconstruction effect diagram of the proposed scheme and GESCO, U-deepSC provided by the present application, which includes the original image, the semantic segmentation map, the proposed scheme (an embodiment of the present application), GESCO and U-deepSC.

[0100] To compare the effectiveness of the embodiments of the present application with the single-modal generative semantic communication method (GESCO) and the multi-modal semantic communication method (U-deepSC), the present application completes image reconstruction, classification and detection experiments of different methods on a single RTX 4090 graphics card, and the image data used is uniformly scaled to 128x128 pixels, and the reconstruction effect is as shown in Figure 5

[0101] Referring to Table 1, Table 1 compares the performance of the three methods in visible light image reconstruction and infrared image reconstruction. Since the main goal of the present application is how to realize multi-modal semantic communication, the experiment does not consider transmission noise. The experimental data compressed using zlib technology show that the average file size of the compressed image is about 700 bytes per image when the present application framework and GESCO transmit 128x128 pixel semantic segmentation maps. In order to maintain the same transmission size, the present application modifies the configuration of the channel encoding module in the U-deepSC method to ensure that the output feature size is consistent with the size of the present application framework and GESCO.

[0102] Table 1 Comparison of the proposed scheme of the present application with U-deepSC and GESCO

[0103]

[0104] The experimental results show that although compared with single-modal methods, the multi-modal method of the present application still benefits from its innovative design in the latent space, and the method of the present application achieves better performance in FID index and downstream tasks of classification and detection. In contrast, the U-deepSC method shows a comparative advantage in LPIPS score, which may be due to its semantic communication based on encoder and decoder. This setting enables the method to transmit relatively complete feature information, thereby bringing better LPIPS score. The method of the present application only transmits part of the semantic features of the object, making these features more prominent in the generated image and achieving better performance in downstream tasks.

[0105] Referring to Table 2, as shown in Table 2, the present application performs an ablation experiment on the proposed scheme. This experiment mainly verifies the effectiveness of two designs: whether to add an additional contrastive learning loss when training the autoencoder, and whether to input the modality category as another condition into the diffusion model. By comparing the results of FID, LPIPS and downstream tasks, it can be proved that adding a contrastive learning loss and inputting the modality category as another condition into the diffusion model both improve the performance of the proposed framework of the present application.

[0106] Table 2 Ablation experiment results of the proposed scheme of the present application

[0107] ​

[0108] To sum up, the multi-modal generative semantic communication method for perception data backhaul provided by the application can realize 200 times compression transmission of a single semantic segmentation map, and after visible light and infrared multi-modal image reconstruction, the method performs well in FID, LPIPS index, and image classification and target detection downstream tasks, and has better performance than existing semantic communication frameworks.

[0109] The multi-modal generative semantic communication device for perception data backhaul provided by the application is described below, and the multi-modal generative semantic communication device for perception data backhaul described below can be referred to each other corresponding to the multi-modal generative semantic communication method for perception data backhaul described above.

[0110] Reference Figure 6 , Figure 6 is a structural schematic diagram of the multi-modal generative semantic communication device for perception data backhaul provided by the application, which includes an acquisition module 601, a compression module 602, an input module 603, a visible light output module 604, and an infrared output module 605.

[0111] The acquisition module 601 is configured to acquire compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a one-hot encoded semantic segmentation map.

[0112] The compression module 602 is configured to decompress the compressed image data to obtain the one-hot encoded semantic segmentation map.

[0113] The input module 603 is configured to input the one-hot encoded semantic segmentation map and a modal category to a pre-trained latent space diffusion model, wherein the modal category includes a visible light modal and an infrared modal.

[0114] The visible light output module 604 is configured to output a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model.

[0115] The infrared output module 605 is configured to output a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0116] Specifically, the multi-modal generative semantic communication device for perception data backhaul provided by the application can realize all method steps implemented by the multi-modal generative semantic communication method for perception data backhaul embodiment, and can achieve the same technical effects. The same parts and beneficial effects in this embodiment as the method embodiment will not be described in detail.

[0117] Figure 7This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call logic instructions in the memory 730 to execute a multimodal generative semantic communication method for sensing data feedback, the method comprising: obtaining compressed image data transmitted by a transmitter, wherein the compressed image data is obtained by compressing a one-hot-encoded semantic segmentation map; decompressing the compressed image data to obtain a one-hot-encoded semantic segmentation map; inputting the one-hot-encoded semantic segmentation map and modality categories into a pre-trained latent space diffusion model, wherein the modality categories include visible light modality and infrared modality; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; and outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0118] Furthermore, the logic instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program is executable by a processor to enable a computer to perform the method of multi-modal generative semantic communication for perception data backhaul, which comprises: obtaining compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a one-hot encoded semantic segmentation map; decompressing the compressed image data to obtain the one-hot encoded semantic segmentation map; inputting the one-hot encoded semantic segmentation map and a modal category into a pre-trained latent space diffusion model, wherein the modal category comprises a visible light modal and an infrared modal; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; and outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0120] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is executable by a processor to implement the method of multi-modal generative semantic communication for perception data backhaul, which comprises: obtaining compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a one-hot encoded semantic segmentation map; decompressing the compressed image data to obtain the one-hot encoded semantic segmentation map; inputting the one-hot encoded semantic segmentation map and a modal category into a pre-trained latent space diffusion model, wherein the modal category comprises a visible light modal and an infrared modal; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; and outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0121] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0122] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0123] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-modal generative semantic communication method for perception data backhauling, characterized in that, The method comprises the following steps: acquiring compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a one-hot encoded semantic segmentation map; decompressing the compressed image data to obtain the one-hot encoded semantic segmentation map; inputting the one-hot encoded semantic segmentation map and a modal category into a pre-trained latent space diffusion model, wherein the modal category comprises a visible light modal and an infrared modal; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model; Before the step of inputting the one-hot encoded semantic segmentation map and the modal category into the pre-trained latent space diffusion model, the method further comprises the following steps: training a visible light autoencoder and an infrared autoencoder of a preset latent space diffusion model based on a visible light image sample and an infrared image sample to obtain a first-stage training loss function; training a diffusion model of the preset latent space diffusion model based on a one-hot encoded semantic segmentation map sample and a modal category to obtain a second-stage training loss function; updating the preset latent space diffusion model based on the first-stage training loss function and the second-stage training loss function to obtain a pre-trained latent space diffusion model; The step of training the diffusion model of the preset latent space diffusion model based on the one-hot encoded semantic segmentation map sample and the modal category to obtain the second-stage training loss function comprises the following steps: acquiring a one-hot encoded semantic segmentation map sample based on the visible light image sample and the infrared image sample; simultaneously inputting the one-hot encoded semantic segmentation map sample and the modal category as condition inputs of the preset latent space diffusion model to obtain visible light modal latent space features and infrared modal latent space features output by the diffusion model of the preset latent space diffusion model; determining a mean square error loss of the visible light modal based on the visible light modal latent space features and visible light modal feature ground truth; determining a mean square error loss of the infrared modal based on the infrared modal latent space features and infrared modal feature ground truth; determining the second-stage training loss function based on the mean square error loss of the visible light modal and the mean square error loss of the infrared modal.

2. The multi-modal generative semantic communication method for perception data backhauling according to claim 1, wherein, Before the step of acquiring the compressed image data transmitted by the sending end, the method further comprises the following steps: calling a sending end to perform image data acquisition to obtain a visible light image and an infrared image, wherein the sending end is a drone, and image information of the visible light image and image information of the infrared image are consistent in time and space dimensions; calling a semantic segmentation model of the sending end to perform semantic extraction on the visible light image and the infrared image to obtain a semantic segmentation map, wherein the semantic segmentation map comprises common semantic features of the visible light image and the infrared image; calling the sending end to one-hot encode the semantic segmentation map to obtain a one-hot encoded semantic segmentation map. The sending end is called to compress the one-hot encoded semantic segmentation map to obtain compressed image data.

3. The multi-modal generative semantic communication method for perception data backhauling according to claim 1, wherein, The visible light self-encoder and the infrared self-encoder of the preset hidden space diffusion model are trained based on the visible light image samples and the infrared image samples to obtain a first-stage training loss function, including: Obtain visible light image samples and infrared image samples; The visible light image samples are projected into the hidden space through an encoder in the visible light self-encoder of the preset hidden space diffusion model to obtain the hidden space features of the visible light image, and the hidden space features of the visible light image are reconstructed into a visible light image through a corresponding visible light decoder to obtain a visible light perception loss generated in the encoding and decoding process; The infrared image samples are projected into the hidden space through an encoder in the infrared self-encoder of the preset hidden space diffusion model to obtain the hidden space features of the infrared image, and the hidden space features of the infrared image are reconstructed into an infrared image through a corresponding infrared decoder to obtain an infrared perception loss generated in the encoding and decoding process; A contrast learning loss between the hidden space features of the visible light image and the hidden space features of the infrared image is determined. The first-stage training loss function is determined based on the visible light perception loss, the infrared perception loss, and the contrast learning loss.

4. The multi-modal generative semantic communication method for perception data backhauling according to claim 1, wherein, The sending end is a UAV, wherein the UAV includes a visible light perception device and an infrared perception device.

5. A multi-modal generative semantic communication device for perception data backhauling, characterized in that, including: An acquisition module is configured to acquire compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a one-hot encoded semantic segmentation map; A compression module is configured to decompress the compressed image data to obtain the one-hot encoded semantic segmentation map; An input module is configured to input the one-hot encoded semantic segmentation map and a modal category into a pre-trained hidden space diffusion model, wherein the modal category includes a visible light modal and an infrared modal; A visible light output module is configured to output a reconstructed visible light image through a visible light decoder of the pre-trained hidden space diffusion model; An infrared output module is configured to output a reconstructed infrared image through an infrared decoder of the pre-trained hidden space diffusion model; The device is further configured to: Before the one-hot encoded semantic segmentation map and the modal category are input into the pre-trained hidden space diffusion model, the visible light self-encoder and the infrared self-encoder of the preset hidden space diffusion model are trained based on visible light image samples and infrared image samples to obtain a first-stage training loss function; The diffusion model of the preset hidden space diffusion model is trained based on one-hot encoded semantic segmentation map samples and modal categories to obtain a second-stage training loss function; The preset hidden space diffusion model is updated based on the first-stage training loss function and the second-stage training loss function to obtain a pre-trained hidden space diffusion model; The diffusion model of the preset hidden space diffusion model is trained based on one-hot encoded semantic segmentation map samples and modal categories to obtain a second-stage training loss function, including: obtain a one-hot encoded semantic segmentation map sample, wherein the one-hot encoded semantic segmentation map sample is based on the visible light image sample and the infrared image sample; input the one-hot encoded semantic segmentation map sample and the modal category into the preset latent space diffusion model as a condition input, to obtain a visible light modal latent space feature and an infrared modal latent space feature of a diffusion model output of the preset latent space diffusion model; determine a mean square error loss of a visible light modal based on the visible light modal latent space feature and a visible light modal feature ground truth; determine a mean square error loss of an infrared modal based on the infrared modal latent space feature and an infrared modal feature ground truth; determine a second stage training loss function based on the mean square error loss of the visible light modal and the mean square error loss of the infrared modal.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the multi-modal generative semantic communication method for perception data backhaul according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the multi-modal generative semantic communication method for perception data backhaul according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the multi-modal generative semantic communication method for perception data backhaul according to any one of claims 1 to 4.

Images