Satellite image semantic coding and decoding method based on diffusion generation model

Abstract

The invention discloses a satellite image semantic coding and decoding method based on a diffusion generation model in the technical field of intelligent coding and decoding. According to the satellite image semantic coding and decoding method based on the diffusion generation model, a variational auto-encoder pre-training model in the diffusion generation model is adopted at a satellite communication sending end, image codes are converted into semantic space feature tensors, and then noise adding is carried out on the semantic space feature tensors based on a noise adding method of the diffusion generation model. And guiding the noise-added semantic space feature tensor to execute a denoising process through text constraint at a receiving end, and inputting a result into a variational self-decoder to recover the result into an image. According to the satellite image semantic coding and decoding method based on the diffusion generation model, under the condition that the target recognition accuracy and the image transmission quality are guaranteed, the transmitted data volume can be greatly compressed, and the method can adapt to a satellite communication channel with a lower signal-to-noise ratio.

Description

Technical Field

[0001] The present invention relates to the field of intelligent coding and decoding technology, and in particular to a satellite image semantic coding and decoding method based on a diffusion generation model. Background Art

[0002] Satellite communication is an important part of modern mobile communication. Due to its long transmission distance, the current satellite communication bandwidth is limited, especially for future satellite Internet services, which are difficult to meet users' real-time transmission needs. Traditional image encoding and decoding algorithms such as JPEG compress, transmit and restore images based on diffusion generation models for image target recognition tasks. The compression capacity of data volume is limited, and in poor channels with low signal-to-noise ratio, there will be a cliff effect where the image quality drops sharply after decompression. With the development of deep learning algorithms and large models, current generation models are widely used in the field of image compression, which can greatly reduce the amount of transmission data required for image transmission. Summary of the invention

[0003] The purpose of the present invention is to provide a satellite image semantic coding and decoding method based on a diffusion generation model, so as to solve the technical problems that the image coding and decoding algorithm in the prior art has limited compression capability for data volume, and in a poor channel with a low signal-to-noise ratio, there will be a cliff effect in which the image quality drops sharply after image decompression. Under the condition of ensuring the target recognition accuracy and image transmission quality, the amount of transmitted data is greatly compressed and can be adapted to a communication channel with a lower signal-to-noise ratio, without the cliff effect, and can be more adapted to the link characteristics of low bandwidth and high signal-to-noise ratio of satellite communication.

[0004] In order to achieve the above object, the technical solution of the present invention is as follows:

[0005] The present invention provides a satellite image semantic encoding and decoding method based on a diffusion generation model, comprising the following steps:

[0006] S1. Variational autoencoder: The variational autoencoder pre-training model in the diffusion generation model is used to convert image encoding into semantic space feature tensors;

[0007] S2. Noise adding network: gradually adding noise to the image semantic space feature tensor output by the variational autoencoder, so that the image semantic space feature tensor approaches the semantic space feature tensor of a noisy image;

[0008] S3. Image target recognition network: Determine the evaluation system based on the characteristics and evaluation requirements of the satellite communication system, and determine the appropriate evaluation model based on different usage modes and scenarios;

[0009] S4.Text encoder: converts the text and bounding box output by the image object recognition network into a tensor;

[0010] S5. Denoising network: Under the guidance of the text tensor, the noisy image semantic space feature tensor is gradually denoised through noise prediction to restore it to a noise-free image semantic space feature tensor;

[0011] S6. Variational self-decoder: The variational self-decoder pre-trained model in the diffusion generation model is used to convert the denoised image semantic space feature tensor back to the pixel space and restore it to the image.

[0012] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S1 comprises the following steps:

[0013] S11. Input an image into a deep neural network to convert the image into a tensor in a latent space. The deep neural network adopts a variational autoencoder structure.

[0014] S12. The variational autoencoder uses a pre-trained model. If fine-tuning is required, the pre-trained model parameters are used as the initial parameters and trained jointly with the variational autodecoder.

[0015] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S2 comprises the following steps:

[0016] S21. The noise-added network is a pre-trained model of the diffusion generation network, and the output is a semantic space feature tensor;

[0017] S22. Add noise to the semantic space feature tensor, and use the denoising strength to control the amount of noise added.

[0018] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, the S22 comprises the following steps:

[0019] S221. If the denoising intensity is zero, no noise is added;

[0020] S222. If the denoising intensity is 1, a maximum amount of noise is added to make the semantic space feature tensor of the image become a completely random tensor.

[0021] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S3 comprises the following steps:

[0022] S31. Input the image into the image recognition network and output the target recognition text result in the image. The image target recognition network adopts the yolov8 network model;

[0023] S32. Encode the vector output by the noise adding network and the image target recognition text and send them into the channel for transmission.

[0024] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, the denoising process of the denoising network in S32 is determined without the intervention of a model; the semantic space feature tensor output by the denoising network and the text output by the image target recognition network are used as data input at the transmitting end of the communication system.

[0025] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S4 comprises the following steps:

[0026] S41. The text encoder is a transformer network that converts text and bounding boxes into a tensor and guides image recovery;

[0027] S42. The text output by the image object recognition network is input to the text encoder to generate a tensor, which is used to guide the semantic space feature tensor through the denoising process of the denoising network.

[0028] As one aspect of a satellite image semantic encoding and decoding method based on a diffusion generation model of the present invention, S5 comprises the following steps:

[0029] S51. The denoising network uses the diffusion generation model U-Net to perform multi-step denoising on the noisy tensor in the semantic space;

[0030] S52. Text is used as a guide constraint in the denoising process to predict the noise in the semantic space and recover the image semantic space feature tensor.

[0031] By adopting the above technical solution, the present invention has the following advantages:

[0032] The present invention provides a satellite image semantic coding and decoding method based on a diffusion generation model. Aiming at the problem of large data volume of high-definition image transmission, a variational autoencoder is used to convert the image into a semantic space feature tensor, which compresses the amount of data to be transmitted with less semantic information loss, and reduces the demand for satellite communication bandwidth. The present invention uses a deep neural network to complete the joint coding and decoding of the source channel, realizes the overall optimization of coding efficiency and coding gain, and maximizes the coding efficiency while ensuring that the image is within the upper limit of distortion. In addition, the joint coding does not have a cliff effect under low signal-to-noise ratio conditions, which can better adapt to the link characteristics of low bandwidth and high signal-to-noise ratio of satellite communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of satellite image semantic communication based on diffusion generation model;

[0034] Figure 2It is a block diagram of the satellite image semantic encoding and decoding based on the diffusion generation model of the present invention;

[0035] Figure 3 Schematic diagram of the structure of a variational autoencoder and a variational autodecoder in an embodiment of the present invention;

[0036] Figure 4 Schematic diagram of the structure of the noise adding network and the noise removing network in the embodiment of the present invention;

[0037] Figure 5 Schematic diagram of the structure of an image object recognition network in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below in conjunction with the drawings in the specification. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0039] A satellite image semantic communication process based on the diffusion generation model is as follows: Figure 1 As shown: the signal source is an image signal source captured in nature. The signal source passes through the image semantic extraction module to complete the sampling quantization and compression of the image data, and converts the image into a semantic space feature tensor, which can reduce data redundancy. The information probability distribution function of the tensor can be used to guide the subsequent source channel joint coding, further improve the coding efficiency, and reduce the number of redundant bits. After channel transmission, the bit data is input into the decoding module to realize the decoding function, and the encoded bit data is decoded into a semantic space feature tensor. The decoded tensor is input into the image semantic recovery module to obtain image information. The image semantic extraction, source channel joint coding, source channel joint decoding module, and image semantic recovery module in the above satellite image semantic communication process are all deep neural networks. Among them, Figure 1 The image semantic extraction and image semantic restoration modules in Figure 2 The variational autoencoder in the source channel joint encoding and decoding module adopts Figure 2 The denoising and denoising networks of the diffusion generative model in .

[0040] A satellite image semantic encoding and decoding method based on diffusion generation model is as follows: Figure 2 As shown, the following steps are included:

[0041] S1. Variational autoencoder: The variational autoencoder pre-training model in the diffusion generation model is used to convert image encoding into semantic space feature tensors;

[0042] S1 includes the following steps:

[0043] S11. Input an image into a deep neural network and convert the image into a tensor in a latent space. The deep neural network uses a variational autoencoder structure as follows: Figure 3 As shown in the figure, the variational autoencoder structure and the variational autodecoder are symmetrical networks and are trained jointly during training;

[0044] S12. The variational autoencoder uses a pre-trained model. If fine-tuning is required, the pre-trained model parameters are used as the initial parameters and trained jointly with the variational autodecoder.

[0045] S2. Noise adding network: gradually adding noise to the image semantic space feature tensor output by the variational autoencoder, so that the image semantic space feature tensor approaches the semantic space feature tensor of a noisy image;

[0046] S2 includes the following steps:

[0047] S21. The noise-added network is a pre-trained model of the diffusion generation network, and the output is a semantic space feature tensor;

[0048] S22. Add noise to the semantic space feature tensor, and use the denoising strength to control the amount of noise added.

[0049] S22 includes the following steps:

[0050] S221. If the denoising intensity is zero, no noise is added;

[0051] S222. If the denoising intensity is 1, add the maximum amount of noise to make the image's semantic space feature tensor become a completely random tensor. The denoising network and the denoising network both use the diffusion generation model network structure as follows: Figure 4 shown.

[0052] S3. Image target recognition network: Determine the evaluation system based on the characteristics and evaluation requirements of the satellite communication system, and determine the appropriate evaluation model based on different usage modes and scenarios;

[0053] S3 includes the following steps:

[0054] S31. Input the image into the image recognition network and output the target recognition text result in the image. The image target recognition network adopts the neural network yolov8 network model as follows: Figure 5 As shown;

[0055] S32. Encode the vector output by the noise adding network and the image target recognition text and send them into the channel for transmission.

[0056] Among them, the denoising process of the denoising network is deterministic, without the intervention of the model, and its purpose is to produce labels used for training; the semantic space feature tensor output by the denoising network and the text output by the image target recognition network are used as the data input of the transmitter of the communication system.

[0057] S4.Text encoder: converts the text and bounding box output by the image object recognition network into a tensor;

[0058] S4 includes the following steps:

[0059] S41. The text encoder is a transformer network that converts text and bounding boxes into a tensor and guides image recovery;

[0060] S42. The text output by the image object recognition network is input to the text encoder to generate a tensor, which is used to guide the semantic space feature tensor through the denoising process of the denoising network.

[0061] S5. Denoising network: Under the guidance of the text tensor, the noisy image semantic space feature tensor is gradually denoised through noise prediction to restore it to a noise-free image semantic space feature tensor;

[0062] S5 includes the following steps:

[0063] S51. The denoising network uses the diffusion generation model U-Net to perform multi-step denoising on the noisy tensor in the semantic space;

[0064] S52. Text is used as a guide constraint in the denoising process to predict the noise in the semantic space and recover the image semantic space feature tensor.

[0065] S6. Variational self-decoder: The variational self-decoder pre-trained model in the diffusion generative model is used to convert the denoised image semantic space feature tensor back to the pixel space and restore it to the image. This is the image received by the receiver after channel transmission.

[0066] Image semantic extraction and encoding compresses images into a smaller latent space without losing information. The reason is that natural images are not random, they have high regularity: a cat's hair, eyes, and nose have a specific spatial relationship, and a dog has four legs and a specific shape. The high dimensionality of the image is the result of artificial quantization, so it is easy to compress it into a smaller semantic space without losing any information. The reason why the amount of information is large is that our representation of the image does not exploit its inherent conditional probability distribution. Using deep neural networks can achieve higher encoding efficiency.

[0067] The satellite image semantic encoding and decoding method based on the diffusion generation model uses the variational autoencoder pre-training model in the diffusion generation model at the satellite communication transmitter to convert the image code into a semantic space feature tensor, and then adds noise to it based on the denoising method of the diffusion generation model. At the receiving end, the semantic space feature tensor after adding noise is guided by text constraints to perform the denoising process, and the result is input into the variational autodecoder to restore it to the image. While ensuring the accuracy of target recognition and the quality of image transmission, the amount of transmitted data can be greatly compressed and can be adapted to satellite communication channels with lower signal-to-noise ratios.

[0068] Finally, it should be pointed out that although the present invention has been described with reference to the current specific embodiments, ordinary technicians in this technical field should realize that the above embodiments are only used to illustrate the present invention and are not used as a limitation of the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, as long as the changes and modifications to the above embodiments are within the essential spirit of the present invention, they will fall within the scope of the claims of the present invention.

Claims

1. A satellite image semantic encoding and decoding method based on a diffusion generation model, characterized in that: The following steps are involved: S1. Variational autoencoder: The variational autoencoder pre-training model in the diffusion generation model is used to convert image encoding into semantic space feature tensors; S2. Noise adding network: gradually adding noise to the image semantic space feature tensor output by the variational autoencoder, so that the image semantic space feature tensor approaches the semantic space feature tensor of a noisy image; S3. Image target recognition network: Determine the evaluation system based on the characteristics and evaluation requirements of the satellite communication system, and determine the appropriate evaluation model based on different usage modes and scenarios; S4.Text encoder: converts the text and bounding box output by the image object recognition network into a tensor; S5. Denoising network: Under the guidance of the text tensor, the noisy image semantic space feature tensor is gradually denoised through noise prediction to restore it to a noise-free image semantic space feature tensor; S6. Variational self-decoder: The variational self-decoder pre-trained model in the diffusion generation model is used to convert the denoised image semantic space feature tensor back to the pixel space and restore it to the image.

2. According to the satellite image semantic encoding and decoding method based on the diffusion generation model according to claim 1, it is characterized in that: The S1 comprises the following steps: S11. Input an image into a deep neural network to convert the image into a tensor in a latent space. The deep neural network adopts a variational autoencoder structure. S12. The variational autoencoder uses a pre-trained model. If fine-tuning is required, the pre-trained model parameters are used as the initial parameters and trained jointly with the variational autodecoder.

3. According to the satellite image semantic encoding and decoding method based on the diffusion generation model of claim 1, it is characterized in that: The S2 comprises the following steps: S21. The noise-added network is a pre-trained model of the diffusion generation network, and the output is a semantic space feature tensor; S22. Add noise to the semantic space feature tensor, and use the denoising strength to control the amount of noise added.

4. According to the satellite image semantic encoding and decoding method based on the diffusion generation model of claim 3, it is characterized in that: The S22 comprises the following steps: S221. If the denoising intensity is zero, no noise is added; S222. If the denoising intensity is 1, a maximum amount of noise is added to make the semantic space feature tensor of the image become a completely random tensor.

5. According to the satellite image semantic encoding and decoding method based on the diffusion generation model according to claim 1, it is characterized in that: The S3 comprises the following steps: S31. Input the image into the image recognition network and output the target recognition text result in the image. The image target recognition network adopts the yolov8 network model; S32. Encode the vector output by the noise adding network and the image target recognition text and send them into the channel for transmission.

6. According to the satellite image semantic encoding and decoding method based on the diffusion generation model of claim 5, it is characterized in that: The denoising process of the denoising network in S32 is determined without the intervention of a model; the semantic space feature tensor output by the denoising network and the text output by the image target recognition network are used as data input to the transmitting end of the communication system.

7. According to the satellite image semantic encoding and decoding method based on the diffusion generation model of claim 1, it is characterized in that: The S4 comprises the following steps: S41. The text encoder is a transformer network that converts text and bounding boxes into a tensor and guides image recovery; S42. The text output by the image object recognition network is input to the text encoder to generate a tensor, which is used to guide the semantic space feature tensor through the denoising process of the denoising network.

8. According to the satellite image semantic encoding and decoding method based on the diffusion generation model of claim 1, it is characterized in that: The S5 comprises the following steps: S51. The denoising network uses the diffusion generation model U-Net to perform multi-step denoising on the noisy tensor in the semantic space; S52. Text is used as a guide constraint in the denoising process to predict the noise in the semantic space and recover the image semantic space feature tensor.

Images