Multi-modal generative semantic communication method and device for sensing data return

Abstract

The invention provides a sensing data return-oriented multi-modal generative semantic communication method and device, and the method comprises the steps: obtaining compressed image data transmitted by a transmitting end, the compressed image data being obtained by compressing a semantic segmentation map after one-hot coding; decompressing the compressed image data to obtain a semantic segmentation map after one-hot coding; the semantic segmentation map after one-hot coding and a modal category are input to a pre-trained hidden space diffusion model, and the modal category comprises a visible light modal and an infrared modal; outputting a reconstructed visible light image through a visible light decoder of a pre-trained hidden space diffusion model; outputting a reconstructed infrared image through an infrared decoder of a pre-trained hidden space diffusion model; according to the invention, the sensing data communication efficiency in the emergency environment can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data transmission technology, and in particular to a multimodal generative semantic communication method and device for perception data feedback. Background Art

[0002] In emergency environments, infrastructure is often affected and taken out of service, resulting in public network interruptions. Drones need to rely on a temporary emergency network to transmit perception data. This has problems such as unstable communication links and limited transmission bandwidth, making it impossible to guarantee the timely transmission of perception data.

[0003] With the continuous advancement of drone technology, the increase in payload capacity and flight time has enabled drones to carry multiple sensing devices of different modes at the same time, such as visible light and infrared sensing devices, which can effectively improve the sensing capabilities of drones in complex emergency environments. However, carrying multiple sensing devices of different modes requires more sensing data to be transmitted back, which brings greater challenges to completing the transmission of sensing data under the conditions of a private network with limited communication capabilities.

[0004] It can be seen that in the related technology, under the conditions of a private network with limited communication capabilities, there is a technical problem of low communication efficiency due to excessive perception data. Summary of the invention

[0005] The present invention provides a multimodal generative semantic communication method and device for perception data feedback, which is used to solve the defect of low communication efficiency due to excessive perception data under the condition of a private network with limited communication capacity in the prior art, and improve the perception data communication efficiency in an emergency environment.

[0006] The present invention provides a multimodal generative semantic communication method for perception data feedback, comprising the following steps: obtaining compressed image data transmitted by a transmitter, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding; decompressing the compressed image data to obtain the semantic segmentation map after one-hot encoding; inputting the semantic segmentation map after one-hot encoding and the modality category into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; and outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0007] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, before obtaining the compressed image data transmitted by the sending end, the method also includes: calling the sending end to collect image data to obtain a visible light image and an infrared image, wherein the sending end is a drone, and the image information of the visible light image is consistent with the image information of the infrared image in the time dimension and the space dimension; calling the semantic segmentation model of the sending end to perform semantic extraction on the visible light image and the infrared image to obtain a semantic segmentation map, wherein the semantic segmentation map includes common semantic features of the visible light image and the infrared image; calling the sending end to perform one-hot encoding on the semantic segmentation map to obtain the one-hot encoded semantic segmentation map; calling the sending end to compress the one-hot encoded semantic segmentation map to obtain compressed image data.

[0008] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, before the semantic segmentation map and modal category after one-hot encoding are input into the pre-trained latent space diffusion model, the method also includes: based on visible light image samples and infrared image samples, training the visible light autoencoder and infrared autoencoder of the preset latent space diffusion model to obtain a first-stage training loss function; based on the semantic segmentation map samples and modal categories after one-hot encoding, training the diffusion model of the preset latent space diffusion model to obtain a second-stage training loss function; based on the first-stage training loss function and the second-stage training loss function, updating the preset latent space diffusion model to obtain a pre-trained latent space diffusion model.

[0009] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, based on visible light image samples and infrared image samples, the visible light autoencoder and the infrared autoencoder of the preset latent space diffusion model are trained to obtain the first stage training loss function, including: obtaining visible light image samples and infrared image samples; projecting the visible light image samples to the latent space through the encoder in the visible light autoencoder of the preset latent space diffusion model to obtain the latent space features of the visible light image, and then reconstructing the latent space features of the visible light image into the visible light image through the corresponding visible light decoder, and obtaining the visible light perception loss generated by the encoding and decoding process; projecting the infrared image samples to the latent space through the encoder in the infrared autoencoder of the preset latent space diffusion model to obtain the latent space features of the infrared image, and then reconstructing the latent space features of the infrared image into the infrared image through the corresponding infrared decoder, and obtaining the infrared perception loss generated by the encoding and decoding process; determining the contrast learning loss between the latent space features of the visible light image and the latent space features of the infrared image; and determining the first stage training loss function based on the visible light perception loss, the infrared perception loss and the contrast learning loss.

[0010] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, the diffusion model of the preset latent space diffusion model is trained based on the semantic segmentation map samples and modality categories after one-hot encoding to obtain the second-stage training loss function, including: obtaining the semantic segmentation map samples after one-hot encoding, wherein the semantic segmentation map samples after one-hot encoding are based on the visible light image samples and the infrared image samples; using the semantic segmentation map samples and modality categories after one-hot encoding as conditional inputs of the preset latent space diffusion model respectively and simultaneously as conditional inputs of the preset latent space diffusion model to obtain the visible light modality latent space features and infrared modality latent space features output by the diffusion model of the preset latent space diffusion model; determining the mean square error loss of the visible light modality based on the visible light modality latent space features and the true value of the visible light modality features; determining the mean square error loss of the infrared modality based on the infrared modality latent space features and the true value of the infrared modality features; determining the second-stage training loss function based on the mean square error loss of the visible light modality and the mean square error loss of the infrared modality.

[0011] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, the sending end is a drone, wherein the drone includes a visible light perception device and an infrared perception device.

[0012] The present invention also provides a multimodal generative semantic communication device for perception data feedback, comprising the following modules: an acquisition module, used to acquire compressed image data transmitted by a transmitter, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding; a compression module, used to decompress the compressed image data to obtain the semantic segmentation map after one-hot encoding; an input module, used to input the semantic segmentation map after one-hot encoding and the modality category into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality; a visible light output module, used to output a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; an infrared output module, used to output a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal generative semantic communication method for perception data feedback as described in any one of the above-mentioned methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal generative semantic communication methods for perception data feedback as described above.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the multimodal generative semantic communication methods for perception data feedback as described above.

[0016] The multimodal generative semantic communication method and device for perception data feedback provided by the present invention obtain compressed image data transmitted by a transmitter, decompress the compressed image data, obtain a semantic segmentation map after one-hot encoding, input the semantic segmentation map after one-hot encoding and the modal category into a pre-trained latent space diffusion model, and then obtain a reconstructed visible light image output by a visible light decoder and a reconstructed infrared image output by an infrared decoder. Thus, the semantic segmentation map and the modal category are used as conditional parameters of the diffusion model at the same time, so that a single diffusion model can reconstruct data of different modalities, effectively reducing deployment overhead in emergency scenarios and improving the communication efficiency of perception data in emergency environments; thereby solving the technical problem of low communication efficiency under the conditions of a private network with limited communication capabilities in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the multimodal generative semantic communication method for perception data feedback provided by the present invention.

[0019] Figure 2 It is a structural diagram of the multimodal generative semantic communication framework provided by the present invention.

[0020] Figure 3 It is a flowchart of the first stage autoencoder training process provided by the present invention.

[0021] Figure 4 It is a flow chart of the second stage diffusion model training process provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the proposed scheme and the reconstruction effects of GESCO and U-deepSC provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of a multimodal generative semantic communication device for perception data feedback provided by the present invention.

[0024] Figure 7 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Timely acquisition of disaster situations at the emergency site is extremely critical for the successful implementation of emergency rescue operations. In complex emergency rescue environments, drones are portable and easy to deploy, and sensing devices are mounted on drones. The drones are quickly deployed to the disaster site to collect on-site information and transmit sensing data back to the rear command center in real time. However, in emergency environments, infrastructure is often affected by disasters and out of service, resulting in public network interruptions. Drones need to rely on temporary emergency networks to transmit sensing data, which has problems such as unstable communication links and limited transmission bandwidth, and cannot guarantee the timely transmission of sensing data. In addition, the visible light sensing devices carried by drones are often blocked by buildings, trees, smoke, etc., and the capabilities of single sensing devices are limited.

[0027] With the continuous advancement of drone technology, the increase in payload capacity and flight time has enabled drones to carry multiple sensing devices of different modes at the same time, such as visible light and infrared sensing devices, which can effectively improve the sensing capabilities of drones in complex emergency environments. However, carrying multiple sensing devices of different modes requires more sensing data to be transmitted back, which brings greater challenges to completing the transmission of sensing data under the conditions of a private network with limited communication capabilities.

[0028] The development of deep learning technology has brought new ideas for the return of perception data. By building a semantic communication framework, deep learning technology is used at the sending end to extract specific semantic information from the collected data of different modalities. By transmitting only the key semantic information related to the task, the demand for bandwidth can be significantly reduced. With the introduction of generative artificial intelligence models such as variational auto-encoders (VAE), generative adversarial networks (GAN), and denoising diffusion probabilistic models (DDPM), the receiving end can reconstruct the original perception data to a certain extent based on the received semantic information, and can further use the reconstructed data for downstream tasks such as image classification and depth estimation. However, existing semantic communication research mainly focuses on single-modal semantic communication, which leads to the need to deploy multiple different semantic communication frameworks at the same time in complex multi-task scenarios, which brings additional overhead. In response to this problem, further research is still needed on the multi-modal generative semantic communication framework, so that a single semantic communication framework can meet multiple application requirements at the same time.

[0029] Therefore, the main problems in building a multimodal generative semantic communication framework are: on the one hand, the traditional semantic communication framework is designed for a single modality. In a complex emergency environment with multiple modalities and multiple tasks, multiple semantic communication frameworks need to be deployed simultaneously, which brings additional resource overhead. On the other hand, the features between different modal data are often not aligned, making it difficult for a single generative model to complete the reconstruction of different modal data.

[0030] In response to the above problems, the present invention proposes a multimodal semantic communication framework that can extract and transmit fused semantic segmentation maps from visible light and infrared images. At the receiving end, it reconstructs the data into visible light and infrared images based on the semantic segmentation map. By using one-hot encoding and zlib compression, the present invention achieves nearly 200 times compression of a single semantic segmentation map, significantly improving the success rate of perception data feedback in emergency environments.

[0031] The present invention introduces a latent diffusion model (LDM), which consists of a pair of autoencoders and a diffusion model. By adding contrastive learning loss to the autoencoder, feature alignment of visible light and infrared modal data in the latent space is achieved. In addition, the semantic segmentation map and modal information are used as conditional parameters of the diffusion model at the same time, so that a single diffusion model can reconstruct data of different modalities, effectively reducing the deployment overhead in emergency scenarios.

[0032] According to the present invention, a multimodal generative semantic communication framework is proposed. At the sending end, the framework can extract a modal fused semantic segmentation map based on the collected visible light and infrared image data. At the receiving end, the framework can reconstruct the original visible light or infrared image based on the fused semantic segmentation map. Experimental results show that this framework achieves a compression ratio of up to 200 times and exhibits excellent performance in downstream tasks such as target classification and detection.

[0033] Optionally, the multimodal generative semantic communication method for perception data feedback of the embodiment of the present application can be executed by the above-mentioned receiving end, or it can be executed jointly by the sending end (for example, drone equipment) and the receiving end, taking the example of the receiving end (server or terminal) executing the multimodal generative semantic communication method for perception data feedback in this embodiment.

[0034] Figure 1 is a flow chart of a multimodal generative semantic communication method for sensing data feedback provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0035] Step 101, obtaining compressed image data transmitted by a transmitting end, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding.

[0036] In an embodiment of the present invention, the transmitting end may be a drone equipped with an infrared sensing device and a visible light sensing device.

[0037] Considering that data of different modalities are collected simultaneously by a single UAV, the information contained in the visible light image and the infrared image is consistent in time and space; the semantic segmentation model in the sending end is used to extract semantics from the visible light image and the infrared image to obtain a semantic segmentation map; the semantic segmentation map is then one-hot encoded and compressed to obtain compressed image data.

[0038] Here, after the semantic segmentation map is one-hot encoded and converted to binary format data, it is compressed using the zlib method. For semantic segmentation maps, the classification (label) of each pixel is usually converted into a one-hot encoding vector, which is then converted to binary format for storage or transmission. Using zlib compression can significantly reduce the size of this data, especially when there are a large number of labels or the image resolution is high.

[0039] Among them, zlib is a widely used compression library that provides data compression and decompression functions.

[0040] Through the embodiments of the present invention, a common semantic segmentation map can be extracted from two modal images using a multimodal fusion semantic segmentation model, thereby avoiding the extraction of a separate semantic segmentation map for each modality, effectively reducing the computation and transmission overhead. In addition, the semantic segmentation map can be One-Hot encoded, and the transmission data size can be further compressed in combination with compression technology.

[0041] Step 102, decompress the compressed image data to obtain a semantic segmentation map after one-hot encoding.

[0042] Step 103, inputting the one-hot encoded semantic segmentation map and the modality category into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality.

[0043] In an embodiment of the present invention, for the receiving end, in order to reconstruct the original modality from the fused semantic data (the semantic segmentation map after one-hot encoding), the present invention proposes a multimodal diffusion model based on the LDM model to realize the reconstruction of visible light and infrared modal data. Considering that the receiving end receives the semantic segmentation map after fusion and one-hot encoding, the generation of different modalities cannot be realized by only using this data as the conditional input of the diffusion model. Therefore, the present invention also inputs the modality category as a condition into the diffusion model, and realizes the generation of different modal data by using decoders of different modalities.

[0044] Step 104, outputting a reconstructed visible light image through a visible light decoder of a pre-trained latent space diffusion model.

[0045] Step 105, outputting the reconstructed infrared image through the infrared decoder of the pre-trained latent space diffusion model.

[0046] In the embodiment of the present invention, the pre-trained latent space diffusion model is pre-trained, and the visible light decoder and the infrared decoder can restore the image (ie, the visible light image and the infrared image) from the latent space.

[0047] In the process of decoding and reconstructing visible light images using a pre-trained latent diffusion model (LDM), the main thing involved is mapping the representation in the latent space back to the original image space. The pre-trained latent diffusion model usually operates on a high-dimensional latent space and gradually restores the image through a series of denoising steps.

[0048] For example, the decoding process can be to input the one-hot encoded semantic segmentation map and the modal category (visible light or infrared) into the diffusion model as conditional parameters, use the diffusion model to denoise a Gaussian distributed random feature vector, and gradually restore the latent space image features (involving multiple iterations). Each iteration reduces the noise and enhances the feature signal that meets the conditional parameters. After completing all denoising steps, the latent space representation of the visible light or infrared image is obtained, and the corresponding decoder output is used to reconstruct it into a visible light or infrared image.

[0049] refer to Figure 2 , Figure 2 It is a structural schematic diagram of the multimodal generative semantic communication framework provided by the present invention, which includes a sending end and a receiving end.

[0050] The sending end includes: collecting visible light (image) and infrared (image), inputting the visible light image and the infrared image into the semantic segmentation model to obtain a semantic segmentation map, performing one-hot encoding on the semantic segmentation map, obtaining the one-hot encoded semantic segmentation map and then compressing it.

[0051] The receiving end includes: decompressing the compressed image data to obtain the semantic segmentation map after one-hot encoding, and inputting the semantic segmentation map after one-hot encoding and the modality (category) as conditional parameters into the diffusion model of the latent space to obtain the latent variable of the corresponding time step ( , , , ), output the visible light modal latent space features and the infrared modal latent space features, and input the visible light modal latent space features and the infrared modal latent space features into the infrared decoder and the visible light decoder respectively, and output the reconstructed infrared image and the reconstructed visible light image.

[0052] Through the above steps of the embodiment of the present invention, compressed image data transmitted by the sending end is obtained, wherein the compressed image data is obtained by compressing the semantic segmentation map after one-hot encoding; the compressed image data is decompressed to obtain the semantic segmentation map after one-hot encoding; the semantic segmentation map after one-hot encoding and the modality category are input into the pre-trained latent space diffusion model, wherein the modality category includes visible light modality and infrared modality; through the visible light decoder of the pre-trained latent space diffusion model, a reconstructed visible light image is output; through the infrared decoder of the pre-trained latent space diffusion model, a reconstructed infrared image is output; thus, the semantic segmentation map and the modality category are used as conditional parameters of the diffusion model at the same time, so that a single diffusion model can reconstruct data of different modalities, effectively reducing the deployment overhead in emergency scenarios and improving the communication efficiency of perceived data in emergency environments; thereby solving the technical problem of low communication efficiency under the conditions of a private network with limited communication capabilities in the related technology.

[0053] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, before obtaining the compressed image data transmitted by the sending end, the method further includes: Call the sending end to collect image data to obtain visible light images and infrared images, where the sending end is a drone, and the image information of the visible light image and the image information of the infrared image are consistent in the time dimension and the space dimension; Calling the semantic segmentation model of the sending end to perform semantic extraction on the visible light image and the infrared image to obtain a semantic segmentation map, wherein the semantic segmentation map includes common semantic features of the visible light image and the infrared image; Call the sender to perform one-hot encoding on the semantic segmentation map to obtain the one-hot encoded semantic segmentation map; The sender is called to compress the semantic segmentation map after one-hot encoding to obtain compressed image data.

[0054] In an embodiment of the present invention, for the sending end, considering that data of different modalities are collected simultaneously by a single drone, the information contained in the visible light image and the infrared image is consistent in time and space. Therefore, a multimodal fusion semantic segmentation model can be used to extract a common semantic segmentation map from the two modal images, thereby avoiding the extraction of a separate semantic segmentation map for each modality, effectively reducing the calculation and transmission overhead.

[0055] In addition, the semantic segmentation map can be One-Hot encoded and combined with compression technology to further compress the size of the transmitted data. In particular, since the present invention focuses on how to achieve efficient data compression through semantic coding, reduce the size of the transmitted data and achieve high-quality reconstruction at the receiving end, the impact of noise during wireless transmission is not considered. Considering that the transmitted data is small enough, it can be further combined with channel coding technology to improve the efficiency of data return in emergency scenarios.

[0056] In an embodiment of the present invention, a multimodal fused semantic segmentation model is used to extract fused semantic information from visible light and infrared data. The present invention adopts the MFNet model, which is designed for visible light and infrared data, and the model is lightweight and suitable for deployment on the computing power equipment carried by the drone. After being processed by the MFNet model, the fused semantic segmentation map is first one-hot encoded, and the one-hot encoded data is used as the conditional input of the diffusion model, which can effectively improve the image reconstruction performance. Then, the encoded data is converted into binary format and compressed using the zlib method.

[0057] In an embodiment of the present invention, in the semantic segmentation task, each pixel is assigned a category label, indicating which category the pixel belongs to (such as road, vehicle, pedestrian, etc.); the semantic segmentation map after one-hot encoding converts the category label of each pixel into a vector whose length is equal to the total number of categories, and only one element is 1 (indicating that the pixel belongs to this category), and the rest of the elements are 0.

[0058] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, before inputting the semantic segmentation map and modality category after one-hot encoding into a pre-trained latent space diffusion model, the method further includes: Based on visible light image samples and infrared image samples, the visible light autoencoder and infrared autoencoder of the preset latent space diffusion model are trained to obtain the first stage training loss function; Based on the semantic segmentation map samples and modal categories after one-hot encoding, the diffusion model of the preset latent space diffusion model is trained to obtain the second-stage training loss function; Based on the first-stage training loss function and the second-stage training loss function, the preset latent space diffusion model is updated to obtain a pre-trained latent space diffusion model.

[0059] The core of the framework of the present invention is an LDM model, which can be derived from a noise sample ( ), gradually removes noise and completes feature reconstruction in the latent space. Since the reconstruction is performed in the latent space, the efficiency of model training and inference can be significantly improved without reducing the reconstruction quality compared to the denoising diffusion probability model (DDPM).

[0060] In addition, the present invention expands the two-stage training mechanism of the LDM model so that the LDM model can support the reconstruction of multimodal data. To train the LDM model, it is necessary to first train an autoencoder, in which the encoder maps the original image to the latent space, and the decoder can restore the image from the latent space. Next, a diffusion model needs to be trained to achieve feature reconstruction in the latent space. In order to achieve multimodal reconstruction, the present invention trains an autoencoder for each modal data (visible light modality, infrared modality), and at the same time, the one-hot encoded semantic segmentation map and modality category are input into the diffusion model as conditions, thereby achieving reconstruction of different modal data.

[0061] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, based on visible light image samples and infrared image samples, a visible light autoencoder and an infrared autoencoder of a preset latent space diffusion model are trained to obtain a first-stage training loss function, including: Obtaining visible light image samples and infrared image samples; The encoder in the visible light autoencoder of the preset latent space diffusion model projects the visible light image sample into the latent space to obtain the latent space features of the visible light image, and then the latent space features of the visible light image are reconstructed into a visible light image through the corresponding visible light decoder to obtain the visible light perception loss generated by the encoding and decoding process; The infrared image sample is projected into the latent space by an encoder in an infrared autoencoder of a preset latent space diffusion model to obtain latent space features of the infrared image, and then the latent space features of the infrared image are reconstructed into an infrared image by a corresponding infrared decoder to obtain infrared perception loss generated by the encoding and decoding processes; determining a contrastive learning loss between latent space features of the visible light image and latent space features of the infrared image; Based on visible light perception loss, infrared perception loss and contrastive learning loss, the first stage training loss function is determined.

[0062] refer to Figure 3 , Figure 3It is a flow chart of the first stage autoencoder training process provided by the present invention, which includes: input image (visible light image sample and infrared image sample), visible light encoder, infrared encoder, visible light decoder, infrared decoder, visible light perception loss, infrared perception loss and contrastive learning loss.

[0063] In the embodiment of the present invention, during the first stage of training, before training the diffusion model, the visible light autoencoder and the infrared modality autoencoder are first trained, and the visible light autoencoder and the infrared modality autoencoder are intended to project the data of each modality into a unified latent space. Figure 3 As shown, the present invention adds contrastive learning loss to the training process. Therefore, the loss function definition of the first stage training can refer to the following formula (1):

[0064] LAE=Lc+LRGB+Linfrared(1) Among them, L AE represents the first stage training loss function, L C represents the contrastive learning loss, L RGB Indicates visible light perception loss, and Linfrared indicates infrared perception loss.

[0065] In the embodiment of the present invention, the perceptual loss used in the traditional single-modal LDM training process is consistent. By encoding the visible light and infrared data into the same latent space through respective encoders, the present invention optimizes the performance of the LDM model to process multimodal data and maintains the semantic consistency of the reconstructed data.

[0066] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, based on the semantic segmentation map samples and modality categories after one-hot encoding, a diffusion model of a preset latent space diffusion model is trained to obtain a second-stage training loss function, including: Obtain a semantic segmentation map sample after one-hot encoding, wherein the semantic segmentation map sample after one-hot encoding is based on a visible light image sample and an infrared image sample; The semantic segmentation map samples and modality categories after one-hot encoding are used as conditional inputs of a preset latent space diffusion model, respectively, to obtain visible light modality latent space features and infrared modality latent space features output by the diffusion model of the preset latent space diffusion model; Based on the latent space features of the visible light modality and the true value of the visible light modality features, the mean square error loss of the visible light modality is determined; Based on the infrared modal latent space features and the true value of the infrared modal features, the mean square error loss of the infrared modal is determined; Based on the mean square error loss of the visible light modality and the mean square error loss of the infrared modality, the second stage training loss function is determined.

[0067] In an embodiment of the present invention, after completing the first stage visible light autoencoder training and infrared autoencoder training, the original visible light and infrared images are input into their respective encoders to obtain visible light and infrared image latent space features, which are used as the true value in the second stage training, and the mean square error is calculated with the visible light and infrared image latent space features output by the diffusion model.

[0068] refer to Figure 4 , Figure 4 It is a flow chart of the second stage diffusion model training process provided by the present invention.

[0069] like Figure 4 As shown, the visible light image samples and the infrared image samples are respectively input into the visible light encoder and the infrared encoder to obtain the true value of the visible light modal feature and the true value of the infrared modal feature; the semantic segmentation map samples are one-hot encoded to obtain the semantic segmentation map samples after one-hot encoding, and the modality (category) and the semantic segmentation map samples after one-hot encoding are respectively used as the conditional input of the diffusion model of the preset latent space diffusion model to obtain the latent variables ( , , , ), output the latent space features of the visible light modality and the latent space features of the infrared modality, and then combine the true values ​​of the visible light modality features and the true values ​​of the infrared modality features to determine the mean square error loss (MSE loss) of the visible light modality and the mean square error loss (MSE loss) of the infrared modality as the second stage training loss function.

[0070] Although the present invention introduces contrastive learning loss in the autoencoder to align features between different modalities as much as possible, generating multimodal data using a single diffusion model still requires further optimization.

[0071] like Figure 4 As shown, the present invention uses the encoded semantic segmentation map and modality category as conditional inputs of the diffusion model at the same time, so that the diffusion model can generate corresponding latent space features for visible light and infrared modalities respectively. Because both visible light and infrared modalities use the same semantic segmentation map, in order to ensure the reconstruction accuracy and consistency between modalities, the loss function used in the second stage of training is the mean square error (MSE) loss between the latent space features generated by each modality and its corresponding true value (obtained by encoding the original image by the corresponding encoder). Then, the MSE losses of the two modalities are summed to optimize the model training effect, which can be specifically expressed by the following formula (2): (2)

[0072] in, represents the loss function of the second stage training (LDM model), Encode the original image for each modality corresponding to the autoencoder The obtained features. is the reconstruction result of the diffusion model, with the encoded semantic segmentation map and modal category as conditional input, is the time step. For the modal category, 0 represents visible light data and 1 represents infrared data. By training the diffusion model with this loss function, the model's ability to process multimodal data can be enhanced.

[0073] According to a multimodal generative semantic communication method for perception data feedback provided by the present invention, the sending end is a drone, wherein the drone includes a visible light perception device and an infrared perception device.

[0074] For the transmitter, considering that data of different modalities are collected simultaneously by a single drone, the information contained in the visible light image and the infrared image is consistent in time and space. Therefore, a multimodal fusion semantic segmentation model can be used to extract a common semantic segmentation map from the two modal images, thereby avoiding the extraction of a separate semantic segmentation map for each modality, effectively reducing the calculation and transmission overhead. In addition, the semantic segmentation map can be one-hot encoded, and the size of the transmitted data can be further compressed in combination with compression technology. In particular, since the present invention focuses on how to achieve efficient data compression through semantic coding, reduce the size of the transmitted data and achieve high-quality reconstruction at the receiving end, the impact of noise during wireless transmission is not considered. Considering that the transmitted data is small enough, it can be further combined with channel coding technology to improve the efficiency of data return in emergency scenarios.

[0075] The following describes an example of a multimodal generative semantic communication method for perception data feedback provided by the present invention in an actual application scenario.

[0076] In order to evaluate the performance of the framework proposed in the present invention, on the one hand, the present invention uses the traditional indicators LPIPS (learned perceptual image patch similarity) and FID (fréchet inception distance) for reflecting the consistency of image information to evaluate the image reconstruction quality. On the other hand, image classification and target detection are used as downstream tasks to evaluate the performance of the framework proposed in the present invention.

[0077] The present invention verifies the performance of the proposed solution through comparative experiments and ablation experiments. The dataset used in the present invention is the Multi-spectral Semantic Segmentation Dataset released together with the MFNet model, which contains 1,569 visible light and infrared images of urban streets. This dataset was originally designed for semantic segmentation tasks and provides semantic segmentation maps as true values. Eight common urban street obstacles (cars, pedestrians, bicycles, curves, stop signs, guardrails, colored cones, and speed bumps) are annotated in these maps. Since the dataset provides visible light images, infrared images, and semantic segmentation maps, it is very suitable for training the multimodal generative semantic communication framework of the present invention.

[0078] In order to evaluate the performance of the framework proposed in this invention, the present invention uses the FID index (a metric used to evaluate the effect of the generative model) and the LPIPS index (an image similarity evaluation index based on deep learning) in the image reconstruction task. The FID index is closer to human judgment, while the LPIPS score mainly measures perceptual similarity. The lower the values ​​of these two indicators, the higher the similarity between the reconstructed image and the original image, indicating better image reconstruction performance.

[0079] In addition, the performance of the proposed framework on image classification and object detection tasks is evaluated. The pre-trained yolov8 model is used to detect pedestrians in the images before and after reconstruction, and the performance of the proposed framework in downstream tasks is quantified by calculating the accuracy of the classification labels and the mean intersection over union (mIoU) of the output pedestrian detection boxes.

[0080] refer to Figure 5 , Figure 5 It is a schematic diagram of the proposed scheme and GESCO and U-deepSC reconstruction effects provided by the present invention, which includes an original image, a semantic segmentation map, the proposed scheme (an embodiment of the present invention), GESCO and U-deepSC.

[0081] In order to compare the effectiveness of the embodiments of the present invention with the single-modal generative semantic communication method (GESCO) and the multi-modal semantic communication method (U-deepSC), the present invention completed the image reconstruction, classification and detection experiments of different methods on a single RTX 4090 graphics card. The image data used was uniformly scaled to 128×128 pixels. The reconstruction effect is as follows: Figure 5 shown.

[0082] Refer to Table 1, which compares the performance of the three methods in visible light image reconstruction and infrared image reconstruction. Since the main goal of the present invention is how to achieve multimodal semantic communication, the experiment does not consider transmission noise. Experimental data compressed using zlib technology show that when the framework of the present invention and GESCO transmit 128×128 pixel semantic segmentation maps, the average file size after compression is approximately 700 bytes per image. In order to maintain the same transmission size, the present invention modifies the configuration of the channel coding module in the U-deepSC method to ensure that the output feature size is consistent with the size of the framework of the present invention and GESCO.

[0083] Table 1 Comparative experimental results of the proposed solution with U-deepSC and GESCO

[0084] The experimental results show that, despite being compared with single-modal methods, the multimodal method of the present invention benefits from its innovative design in the latent space, and the method of the present invention achieves better performance in terms of the FID indicator and classification and detection downstream tasks. In contrast, the U-deepSC method shows a comparative advantage in LPIPS scores, which may be due to its semantic communication based on the encoder and decoder. This setting enables the method to transmit relatively complete feature information, resulting in a better LPIPS score. The method of the present invention only transmits part of the semantic features of the object, making these features more prominent in the generated image and achieving better performance in downstream tasks.

[0085] Referring to Table 2, as shown in Table 2, the present invention conducts an ablation experiment on the proposed scheme. The experiment mainly verifies the effectiveness of two designs: whether to add additional contrastive learning loss when training the autoencoder, and whether to input the modality category as another condition into the diffusion model. By comparing the results of FID, LPIPS and downstream tasks, it can be proved that adding contrastive learning loss and inputting the modality category as another condition into the diffusion model both improve the performance of the framework proposed by the present invention.

[0086] Table 2 Ablation experiment results of the proposed solution

[0087] In summary, the multimodal generative semantic communication method for perception data feedback provided by the present invention not only achieves 200 times compression transmission of a single semantic segmentation map, but also completes the reconstruction of visible light and infrared multimodal images, and performs well in FID, LPIPS indicators, and downstream tasks of image classification and target detection, and has better performance than the existing semantic communication framework.

[0088] The multimodal generative semantic communication device for perceptual data feedback provided by the present invention is described below. The multimodal generative semantic communication device for perceptual data feedback described below and the multimodal generative semantic communication method for perceptual data feedback described above can be referenced to each other.

[0089] refer to Figure 6 , Figure 6 It is a structural diagram of a multimodal generative semantic communication device for perception data feedback provided by the present invention, which includes an acquisition module 601, a compression module 602, an input module 603, a visible light output module 604 and an infrared output module 605.

[0090] An acquisition module 601 is used to acquire compressed image data transmitted by a transmitting end, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding; A compression module 602 is used to decompress the compressed image data to obtain the semantic segmentation map after the one-hot encoding; An input module 603 is used to input the semantic segmentation map and the modality category after the one-hot encoding into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality; A visible light output module 604, configured to output a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; The infrared output module 605 is used to output the reconstructed infrared image through the infrared decoder of the pre-trained latent space diffusion model.

[0091] Specifically, the multimodal generative semantic communication device for perceptual data feedback provided by the present invention can implement all the method steps implemented by the multimodal generative semantic communication method embodiment for perceptual data feedback, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0092] Figure 7 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a multimodal generative semantic communication method for sensing data feedback, the method comprising: obtaining compressed image data transmitted by a transmitter, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding; decompressing the compressed image data to obtain a semantic segmentation map after one-hot encoding; inputting the semantic segmentation map after one-hot encoding and the modality category into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; and outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0093] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal generative semantic communication method for perception data feedback provided by the above methods, and the method includes: obtaining compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding; decompressing the compressed image data to obtain a semantic segmentation map after one-hot encoding; inputting the semantic segmentation map after one-hot encoding and the modality category into a pre-trained latent space diffusion model, wherein the modality category includes visible light modality and infrared modality; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; and outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal generative semantic communication method for perception data feedback provided by the above-mentioned methods, the method comprising: obtaining compressed image data transmitted by a sending end, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding; decompressing the compressed image data to obtain a semantic segmentation map after one-hot encoding; inputting the semantic segmentation map after one-hot encoding and the modality category into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality; outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; and outputting a reconstructed infrared image through an infrared decoder of the pre-trained latent space diffusion model.

[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal generative semantic communication method for sensing data feedback, characterized in that: include: Obtaining compressed image data transmitted by a transmitting end, wherein the compressed image data is obtained by compressing a semantic segmentation map after one-hot encoding; Decompressing the compressed image data to obtain the semantic segmentation map after the one-hot encoding; Inputting the one-hot encoded semantic segmentation map and modality category into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality; Outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; The infrared decoder of the pre-trained latent space diffusion model outputs a reconstructed infrared image.

2. The multimodal generative semantic communication method for perception data feedback according to claim 1 is characterized in that: Before obtaining the compressed image data transmitted by the transmitting end, the method further includes: Calling a sending end to collect image data to obtain a visible light image and an infrared image, wherein the sending end is a drone, and image information of the visible light image and image information of the infrared image are consistent in time dimension and space dimension; Calling the semantic segmentation model of the sending end to perform semantic extraction on the visible light image and the infrared image to obtain a semantic segmentation map, wherein the semantic segmentation map includes common semantic features of the visible light image and the infrared image; Calling the sending end to perform one-hot encoding on the semantic segmentation map to obtain a one-hot encoded semantic segmentation map; The sending end is called to compress the semantic segmentation map after the one-hot encoding to obtain compressed image data.

3. The multimodal generative semantic communication method for perception data feedback according to claim 1, characterized in that: Before inputting the one-hot encoded semantic segmentation map and modality category into the pre-trained latent space diffusion model, the method further includes: Based on visible light image samples and infrared image samples, the visible light autoencoder and infrared autoencoder of the preset latent space diffusion model are trained to obtain the first stage training loss function; Based on the semantic segmentation map samples and modality categories after one-hot encoding, the diffusion model of the preset latent space diffusion model is trained to obtain a second-stage training loss function; Based on the first-stage training loss function and the second-stage training loss function, the preset latent space diffusion model is updated to obtain a pre-trained latent space diffusion model.

4. The multimodal generative semantic communication method for perception data feedback according to claim 3 is characterized in that: The visible light autoencoder and the infrared autoencoder of the preset latent space diffusion model are trained based on the visible light image samples and the infrared image samples to obtain the first stage training loss function, including: Obtaining visible light image samples and infrared image samples; The encoder in the visible light autoencoder of the preset latent space diffusion model projects the visible light image sample into the latent space to obtain the latent space features of the visible light image, and then the latent space features of the visible light image are reconstructed into a visible light image through the corresponding visible light decoder to obtain the visible light perception loss generated by the encoding and decoding processes; The infrared image sample is projected into the latent space by an encoder in an infrared autoencoder of a preset latent space diffusion model to obtain latent space features of the infrared image, and then the latent space features of the infrared image are reconstructed into an infrared image by a corresponding infrared decoder to obtain infrared perception loss generated by the encoding and decoding processes; Determining a contrastive learning loss between latent space features of the visible light image and latent space features of the infrared image; Based on the visible light perception loss, infrared perception loss and contrastive learning loss, a first-stage training loss function is determined.

5. The multimodal generative semantic communication method for perception data feedback according to claim 3 is characterized in that: The diffusion model of the preset latent space diffusion model is trained based on the semantic segmentation map samples and modality categories after one-hot encoding to obtain the second-stage training loss function, including: Obtaining a semantic segmentation map sample after one-hot encoding, wherein the semantic segmentation map sample after one-hot encoding is based on the visible light image sample and the infrared image sample; The semantic segmentation map samples and modality categories after the one-hot encoding are respectively used as conditional inputs of the preset latent space diffusion model to obtain visible light modality latent space features and infrared modality latent space features output by the diffusion model of the preset latent space diffusion model; Determining a mean square error loss of the visible light modality based on the visible light modality latent space features and the visible light modality feature true value; Determining the mean square error loss of the infrared modality based on the infrared modality latent space features and the true value of the infrared modality features; A second-stage training loss function is determined based on the mean square error loss of the visible light modality and the mean square error loss of the infrared modality.

6. The multimodal generative semantic communication method for perception data feedback according to claim 1, characterized in that: The transmitting end is a drone, wherein the drone includes a visible light sensing device and an infrared sensing device.

7. A multimodal generative semantic communication device for sensing data feedback, characterized in that: include: An acquisition module, used for acquiring compressed image data transmitted by a transmitting end, wherein the compressed image data is obtained by compressing the semantic segmentation map after one-hot encoding; A compression module, used for decompressing the compressed image data to obtain the semantic segmentation map after the one-hot encoding; An input module, used for inputting the one-hot encoded semantic segmentation map and modality category into a pre-trained latent space diffusion model, wherein the modality category includes a visible light modality and an infrared modality; A visible light output module, used for outputting a reconstructed visible light image through a visible light decoder of the pre-trained latent space diffusion model; The infrared output module is used to output the reconstructed infrared image through the infrared decoder of the pre-trained latent space diffusion model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal generative semantic communication method for perception data feedback as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal generative semantic communication method for perception data feedback as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal generative semantic communication method for perception data feedback as described in any one of claims 1 to 6 is implemented.

Images