A satellite image semantic encoding and decoding method based on diffusion generation model

Abstract

The present invention discloses a satellite image semantic encoding and decoding method based on a diffusion generative model, in the field of intelligent encoding and decoding technology. This method utilizes a variational autoencoder pre-trained model from the diffusion generative model at the satellite communication transmitter to convert image codes into semantic spatial feature tensors. This is then denoised using a denoising method based on the diffusion generative model. At the receiver, the noisy semantic spatial feature tensors are denoised using text constraints, and the result is input into a variational autodecoder to restore the image. This method significantly compresses the amount of transmitted data while ensuring target recognition accuracy and image transmission quality, enabling it to adapt to satellite communication channels with lower signal-to-noise ratios.

Description

Technical Field

[0001] The present invention relates to the field of intelligent coding and decoding technology, and in particular to a satellite image semantic coding and decoding method based on a diffusion generation model. Background Art

[0002] Satellite communications are a vital component of modern mobile communications. However, due to their long transmission distances, current satellite communication bandwidth is limited, making it difficult for future satellite internet services, in particular, to meet users' real-time transmission needs. Traditional image encoding and decoding algorithms, such as JPEG, use diffusion-based generative models to compress, transmit, and recover images for target recognition tasks. These algorithms have limited data compression capabilities and, in poor signal-to-noise ratio (SNR) channels, experience a cliff effect, resulting in a sharp drop in image quality after decompression. With the development of deep learning algorithms and large models, generative models are now widely applied to image compression, significantly reducing the amount of data required for image transmission. Summary of the Invention

[0003] The purpose of the present invention is to provide a satellite image semantic encoding and decoding method based on a diffusion generation model, which solves the technical problems of the limited data compression capability of the image encoding and decoding algorithms in the prior art and the cliff effect in which the image quality drops sharply after decompression in poor channels with low signal-to-noise ratio. While ensuring the accuracy of target recognition and the quality of image transmission, the method greatly compresses the amount of transmitted data and enables it to adapt to communication channels with lower signal-to-noise ratios without the cliff effect, making it more adaptable to the low-bandwidth and high-signal-to-noise ratio link characteristics of satellite communication.

[0004] In order to achieve the above object, the technical solution of the present invention is as follows:

[0005] The present invention provides a satellite image semantic encoding and decoding method based on a diffusion generation model, comprising the following steps:

[0006] S1. Variational Autoencoder: This pre-trained variational autoencoder model from the diffusion generative model is used to convert image encodings into semantic space feature tensors.

[0007] S2. Noise-adding network: gradually adds noise to the image semantic space feature tensor output by the variational autoencoder, so that the image semantic space feature tensor approaches the semantic space feature tensor of a noisy image;

[0008] S3. Image Target Recognition Network: Determine the evaluation system based on the characteristics and evaluation requirements of the satellite communication system and determine the appropriate evaluation model based on different usage modes and scenarios.

[0009] S4. Text Encoder: Converts the text and bounding box output by the image object recognition network into a tensor;

[0010] S5. Denoising Network: Guided by the text tensor, the noisy image semantic space feature tensor is gradually denoised through noise prediction to restore the noise-free image semantic space feature tensor.

[0011] S6. Variational Autodecoder: The variational autodecoder pre-trained model in the diffusion generative model is used to convert the denoised image semantic space feature tensor back to the pixel space and restore it to the image.

[0012] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S1 includes the following steps:

[0013] S11. Input an image into a deep neural network to convert the image into a tensor in a latent space. The deep neural network uses a variational autoencoder structure.

[0014] S12. The variational autoencoder uses a pre-trained model. If fine-tuning is required, the pre-trained model parameters are used as the initial parameters and trained jointly with the variational autodecoder.

[0015] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S2 includes the following steps:

[0016] S21. The noise-adding network is a pre-trained model of the diffusion generative network, and its output is a semantic space feature tensor.

[0017] S22. Add noise to the semantic space feature tensor, and use the denoising strength to control the amount of noise added.

[0018] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, the S22 includes the following steps:

[0019] S221. If the denoising strength is zero, no noise is added;

[0020] S222. If the denoising strength is 1, a maximum amount of noise is added to make the semantic space feature tensor of the image become a completely random tensor.

[0021] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S3 includes the following steps:

[0022] S31. Input the image into the image recognition network and output the target recognition text result in the image. The image target recognition network adopts the yolov8 network model;

[0023] S32. Encode the vector output by the noise adding network and the image target recognition text and send them into the channel for transmission.

[0024] As one aspect of the satellite image semantic encoding and decoding method based on the diffusion generation model of the present invention, the denoising process of the denoising network in S32 is determined without the intervention of the model; the semantic space feature tensor output by the denoising network and the text output by the image target recognition network are used as data input of the communication system transmitting end.

[0025] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S4 includes the following steps:

[0026] S41. The text encoder is a transformer network that converts text and bounding boxes into a tensor and guides image recovery.

[0027] S42. The text output by the image object recognition network is input to the text encoder to generate a tensor, which is used to guide the semantic space feature tensor through the denoising process of the denoising network.

[0028] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S5 includes the following steps:

[0029] S51. The denoising network uses the diffusion generative model U-Net to perform multi-step denoising on the noisy tensor in the semantic space.

[0030] S52. Text is used as a guide constraint in the denoising process to predict the noise in the semantic space and recover the image semantic space feature tensor.

[0031] By adopting the above technical solution, the present invention has the following advantages:

[0032] This paper provides a satellite image semantic encoding and decoding method based on a diffusion generative model. To address the large data volume required for high-definition image transmission, a variational autoencoder is used to convert the image into a semantic spatial feature tensor. This method compresses the amount of data required for transmission with minimal loss of semantic information, reducing the demand for satellite communication bandwidth. The paper also uses a deep neural network to perform joint source-channel encoding and decoding, achieving coordinated optimization of coding efficiency and coding gain. This method maximizes coding efficiency while ensuring that the image remains within an upper distortion limit. Furthermore, the joint encoding avoids the cliff effect under low signal-to-noise ratio conditions, making it more adaptable to the low-bandwidth, high-signal-to-noise ratio characteristics of satellite communication links. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of satellite image semantic communication based on diffusion generation model;

[0034] Figure 2This is a block diagram of the satellite image semantic encoding and decoding based on the diffusion generation model of the present invention;

[0035] Figure 3 Schematic diagram of the structure of the variational autoencoder and the variational autodecoder in an embodiment of the present invention;

[0036] Figure 4 Schematic diagram of the structure of the noise addition network and the denoising network in an embodiment of the present invention;

[0037] Figure 5 Schematic diagram of the structure of an image object recognition network in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as first and second, etc., are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include," "comprise," or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus.

[0039] A satellite image semantic communication process based on the diffusion generation model is as follows: Figure 1 As shown: the signal source is an image signal source captured in nature. The signal source passes through the image semantic extraction module to complete the sampling quantization and compression of the image data, and converts the image into a semantic space feature tensor, which can reduce data redundancy. The information probability distribution function of the tensor can be used to guide the subsequent source channel joint coding, further improve the coding efficiency, and reduce the number of redundant bits. After channel transmission, the bit data is input into the decoding module to realize the decoding function, and the encoded bit data is decoded into a semantic space feature tensor. The decoded tensor is input into the image semantic recovery module to obtain image information. The image semantic extraction, source channel joint coding, source channel joint decoding module, and image semantic recovery module in the above satellite image semantic communication process are all deep neural networks. Among them, Figure 1 The image semantic extraction and image semantic restoration modules in Figure 2 The variational autoencoder in the source-channel joint encoding and decoding module adopts Figure 2 The denoising and noise-adding networks of the diffusion generative model in

[15] .

[0040] A satellite image semantic encoding and decoding method based on diffusion generation model is as follows: Figure 2 As shown, the following steps are included:

[0041] S1. Variational Autoencoder: This pre-trained variational autoencoder model from the diffusion generative model is used to convert image encodings into semantic space feature tensors.

[0042] S1 includes the following steps:

[0043] S11. Input an image into a deep neural network and convert the image into a tensor in a latent space. The deep neural network uses a variational autoencoder structure as follows: Figure 3 As shown in Figure 2, the variational autoencoder structure and the variational autodecoder structure are symmetrical networks and are trained jointly during training.

[0044] S12. The variational autoencoder uses a pre-trained model. If fine-tuning is required, the pre-trained model parameters are used as the initial parameters and trained jointly with the variational autodecoder.

[0045] S2. Noise-adding network: gradually adds noise to the image semantic space feature tensor output by the variational autoencoder, so that the image semantic space feature tensor approaches the semantic space feature tensor of a noisy image;

[0046] S2 includes the following steps:

[0047] S21. The noise-adding network is a pre-trained model of the diffusion generative network, and its output is a semantic space feature tensor.

[0048] S22. Add noise to the semantic space feature tensor, and use the denoising strength to control the amount of noise added.

[0049] S22 includes the following steps:

[0050] S221. If the denoising strength is zero, no noise is added;

[0051] S222. If the denoising strength is 1, the maximum amount of noise is added to make the semantic space feature tensor of the image become a completely random tensor. In which, both the denoising network and the denoising network use the diffusion generation model network structure as follows Figure 4 shown.

[0052] S3. Image Target Recognition Network: Determine the evaluation system based on the characteristics and evaluation requirements of the satellite communication system and determine the appropriate evaluation model based on different usage modes and scenarios.

[0053] S3 includes the following steps:

[0054] S31. Input the image into the image recognition network and output the target recognition text result in the image. The image target recognition network adopts the neural network yolov8 network model as follows: Figure 5 As shown;

[0055] S32. Encode the vector output by the noise adding network and the image target recognition text and send them into the channel for transmission.

[0056] Among them, the denoising process of the denoising network is deterministic, without the intervention of the model, and its purpose is to produce labels used for training; the semantic space feature tensor output by the denoising network and the text output by the image target recognition network are used as the data input of the transmitter of the communication system.

[0057] S4. Text Encoder: Converts the text and bounding box output by the image object recognition network into a tensor;

[0058] S4 includes the following steps:

[0059] S41. The text encoder is a transformer network that converts text and bounding boxes into a tensor and guides image recovery.

[0060] S42. The text output by the image object recognition network is input to the text encoder to generate a tensor, which is used to guide the semantic space feature tensor through the denoising process of the denoising network.

[0061] S5. Denoising Network: Guided by the text tensor, the noisy image semantic space feature tensor is gradually denoised through noise prediction to restore the noise-free image semantic space feature tensor.

[0062] S5 includes the following steps:

[0063] S51. The denoising network uses the diffusion generative model U-Net to perform multi-step denoising on the noisy tensor in the semantic space.

[0064] S52. Text is used as a guide constraint in the denoising process to predict the noise in the semantic space and recover the image semantic space feature tensor.

[0065] S6. Variational Autodecoder: The variational autodecoder pre-trained model in the diffusion generative model is used to convert the denoised image semantic space feature tensor back to the pixel space and restore it to the image. This is the image received by the receiver after channel transmission.

[0066] Image semantic extraction and encoding compresses images into a smaller latent space without losing information. The reason is that natural images are not random; they have a high degree of regularity: a cat's fur, eyes, and nose have specific spatial relationships, and a dog has four legs with a specific shape. The high dimensionality of images is the result of artificial quantization, making them easily compressible into a smaller semantic space without losing any information. This high amount of information is due to the fact that our representation of images does not exploit their inherent conditional probability distribution. Using deep neural networks can achieve higher encoding efficiency.

[0067] This satellite image semantic encoding and decoding method based on the diffusion generative model uses a pre-trained variational autoencoder model from the diffusion generative model at the satellite communication transmitter to convert the image code into a semantic spatial feature tensor. This is then denoised using a denoising method based on the diffusion generative model. At the receiver, the denoised semantic spatial feature tensor is denoised using text constraints, and the result is input into a variational autodecoder to restore the image. While maintaining target recognition accuracy and image transmission quality, this method significantly compresses the amount of transmitted data and enables it to adapt to satellite communication channels with lower signal-to-noise ratios.

[0068] Finally, it should be pointed out that although the present invention has been described with reference to the current specific embodiments, ordinary technicians in this technical field should realize that the above embodiments are only used to illustrate the present invention and are not used to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, as long as the changes and modifications to the above embodiments are within the scope of the essential spirit of the present invention, they will fall within the scope of the claims of the present invention.

Claims

1. A satellite image semantic encoding and decoding method based on a diffusion generation model, characterized in that: The following steps are involved: S1. Variational Autoencoder: This pre-trained variational autoencoder model from the diffusion generative model is used to convert image encodings into semantic space feature tensors. Said S1 comprises the following steps: S11. Input an image into a deep neural network to convert the image into a tensor in a latent space. The deep neural network uses a variational autoencoder structure. S12. The variational autoencoder uses a pre-trained model. If fine-tuning is required, the pre-trained model parameters are used as the initial parameters for joint training with the variational autodecoder. S2. Noise-adding network: gradually adds noise to the image semantic space feature tensor output by the variational autoencoder, so that the image semantic space feature tensor approaches the semantic space feature tensor of a noisy image; S3. Image Target Recognition Network: Determine the evaluation system based on the characteristics and evaluation requirements of the satellite communication system and determine the appropriate evaluation model based on different usage modes and scenarios. The S3 includes the following steps: S31. Input the image into the image recognition network and output the target recognition text result in the image. The image target recognition network adopts the yolov8 network model; S32. Encode the vector output by the noise network and the image target recognition text and send them to the channel for transmission; S4. Text Encoder: Converts the text and bounding box output by the image object recognition network into a tensor; S5. Denoising Network: Guided by the text tensor, the noisy image semantic space feature tensor is gradually denoised through noise prediction to restore the noise-free image semantic space feature tensor. S6. Variational Autodecoder: The variational autodecoder pre-trained model in the diffusion generative model is used to convert the denoised image semantic space feature tensor back to the pixel space and restore it to the image.

2. The satellite image semantic encoding and decoding method based on the diffusion generation model according to claim 1 is characterized in that: The S2 comprises the following steps: S21. The noise-adding network is a pre-trained model of the diffusion generative network, and its output is a semantic space feature tensor. S22. Add noise to the semantic space feature tensor, and use the denoising strength to control the amount of noise added.

3. The satellite image semantic encoding and decoding method based on the diffusion generation model according to claim 2 is characterized in that: The S22 includes the following steps: S221. If the denoising strength is zero, no noise is added; S222. If the denoising strength is 1, a maximum amount of noise is added to make the semantic space feature tensor of the image become a completely random tensor.

4. The satellite image semantic encoding and decoding method based on the diffusion generation model according to claim 1 is characterized in that: The denoising process of the denoising network in S32 is deterministic without the intervention of a model; the semantic space feature tensor output by the denoising network and the text output by the image target recognition network are used as data input to the transmitting end of the communication system.

5. The satellite image semantic encoding and decoding method based on the diffusion generation model according to claim 1 is characterized in that: The S4 comprises the following steps: S41. The text encoder is a transformer network that converts text and bounding boxes into a tensor and guides image recovery. S42. The text output by the image object recognition network is input to the text encoder to generate a tensor, which is used to guide the semantic space feature tensor through the denoising process of the denoising network.

6. The satellite image semantic encoding and decoding method based on the diffusion generation model according to claim 1 is characterized in that: The S5 comprises the following steps: S51. The denoising network uses the diffusion generative model U-Net to perform multi-step denoising on the noisy tensor in the semantic space. S52. Text is used as a guide constraint in the denoising process to predict the noise in the semantic space and recover the image semantic space feature tensor.

Images