A multi-scale information generative remote sensing image inter-frame compression coding method

Abstract

The application discloses a multi-scale information generation type remote sensing image interframe compression coding method, and belongs to the field of remote sensing image compression and video coding. The method adopts a frame intertemporal information generation strategy at the decoding end, and adopts a simple transformation mode at the encoding end to convert video frames into compact latent representations. After quantization and entropy coding, the video frames are reobtained at the decoding end. Preliminary temporal features are extracted from the previous two reconstructed frames, and a diffusion model is used to generate temporal feature representations with richer semantics. A multi-scale temporal feature mining module further obtains multi-scale temporal semantic features containing different semantic information. The multi-scale temporal semantic features further guide the diffusion model to generate more robust reconstructed feature representations, and serve as semantic prior information in the inverse transformation process of the reconstructed feature representations, so that high-fidelity reconstructed video frames are obtained. The purpose of simple encoding and joint decoding at the decoding end by means of a generation model is achieved, and the method is suitable for remote sensing image video compression.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image compression and video encoding and decoding, and more specifically, relates to a multi-scale information generation remote sensing image inter-frame compression and encoding / decoding method. Background Technology

[0002] Remote sensing satellite imaging has wide applications in natural disaster monitoring and resource exploration, and is a core technology for effective Earth observation. With the continuous development of remote sensing imaging technology, Earth observation satellites have evolved from traditional single-satellite operation to a satellite constellation system with multiple satellites working together. Under this constellation system, the amount of spatiotemporal and spectral data in remote sensing imaging has increased dramatically. The severely limited computing power and storage resources of the onboard platform, along with the limited transmission bandwidth of the satellite-to-ground link, hinder the application of remote sensing imaging. Video compression technology can effectively remove spatiotemporal redundancy in sequential remote sensing images and alleviate the bandwidth pressure on the satellite-to-ground transmission link, making it a key technology for the efficient application of massive sequential remote sensing imaging data. Traditional video compression methods use modules such as motion vector estimation and motion compensation at the encoding end to remove inter-frame redundancy. However, these modules greatly increase the computing power and storage resources required at the encoding end, posing a significant challenge to the onboard platform. At the same time, the cumbersome encoding process at the encoding end greatly reduces compression efficiency.

[0003] Chinese patent application CN200610080881.8 proposes an inter-frame coding method based on macroblock spatiotemporal correlation. However, this method cannot effectively reduce spatiotemporal redundancy in remote sensing video sequences, has low compression efficiency, and cannot effectively resolve the contradiction between massive constellation data and limited transmission bandwidth. Chinese patent application 202311011782.4 employs a deep network model, improving video coding performance through enhanced motion information estimation and probabilistic entropy modeling. However, the motion estimation module at the encoding end significantly increases computational load and memory consumption, making it unsuitable for satellite applications. Chinese patent application 202510797399.9 decomposes and maps the video signal to three latent subspaces, encoding each subspace separately to generate three sets of corresponding coded data. However, the motion estimation module at the encoding end also significantly increases computational load and memory consumption.

[0004] It is evident that existing methods employ motion estimation and motion compensation modules at the encoding end, resulting in significant latency overhead during the encoding process. This is unacceptable for resource-constrained satellite systems. Furthermore, the information used at the encoding end is limited, leading to poor quality of the reconstructed video. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding method, thereby solving the problems of huge latency overhead and poor video quality in existing satellite constellation scene remote sensing image compression methods during the encoding process.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding system is provided, comprising an encoding end and a decoding end: The encoding end includes a feature transformation module, a quantization module, and an arithmetic encoder; The feature transformation module is used to transform the first feature in the source view video. Frame Image Performing nonlinear transformation operations yields a continuous latent representation. The quantization module is used for... Quantization yields a discretized latent representation. ; The arithmetic encoder is used for... Compress the data to obtain a compressed bitstream; The decoding end includes an arithmetic decoder, a first diffusion model, a second diffusion model, a temporal feature extraction module, a multi-scale temporal feature mining module, and a feature inverse transformation module; The arithmetic decoder is used to restore the compressed bitstream to... The temporal feature extraction module is used to extract the first... Frame, First Frame Reconstruction Image , The temporal characteristics as Temporal characteristics The first diffusion model is used for... Enhancement processing is performed to obtain enhanced temporal features. The multi-scale temporal feature mining module is used to... and Generate multi-scale temporal features The second diffusion model is used for multi-scale features. Under the guidance of , Generate reconstructed feature representation F; the inverse feature transformation module is used to... Under guidance, a nonlinear inverse transform operation is performed on F to obtain the first... Frame Reconstruction Image ; This represents the number of multi-scale features.

[0007] According to a second aspect of the present invention, a training method for a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding system as described in the first aspect is provided, characterized in that it includes: The system is trained by inputting sample images from the source video and minimizing the difference between the reconstructed image output by the system and the sample images. The system's feature transformation module, quantization module, temporal feature extraction module, multi-scale temporal feature mining module, and feature inverse transformation module are trained.

[0008] According to a third aspect of the present invention, a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding method is provided, comprising: The image to be processed in the video from the source's perspective is input into the system described in the first aspect to obtain the corresponding reconstructed image.

[0009] According to a fourth aspect of the present invention, an electronic device is provided, comprising: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect or the second invention.

[0010] According to a fifth aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first aspect or the second invention.

[0011] According to a sixth aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method described in the first aspect or the second invention.

[0012] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: The multi-scale information-generative remote sensing image inter-frame compression encoding and decoding method provided by this invention leverages the powerful generative capabilities of generative models. At the satellite encoding end, a simplified processing method can be used to obtain a compressed bitstream containing key information of video frames, thereby improving compression efficiency and alleviating the transmission problem caused by limited bandwidth in satellite-to-ground links. At the ground decoding end, the generative model is used to obtain high-quality, high-fidelity reconstructed images. By enhancing the semantic information of the reconstructed features at the decoding end, the visual fidelity of the reconstructed video frames is improved. This method is suitable for inter-frame compression tasks of remote sensing images in satellite constellation scenarios and has the following advantages: (1) By employing a simplified pixel shuffling downsampling as a nonlinear transformation operation at the encoding end, the source view video frame is transformed into a compact latent representation, achieving decoupling from the high-dimensional pixel domain to the low-dimensional latent feature domain. At the same time, this operation effectively removes spatial redundancy while retaining key data in the source view video frame. In addition, this operation can achieve parameterless and non-multiplicative accumulation lightweight downsampling, completing the dimensionality transformation only through tensor rearrangement, with extremely low computational and memory overhead, making it very suitable for spaceborne platforms with limited computing power and memory. Furthermore, removing the motion estimation and motion compensation modules at the encoding end greatly reduces the computational and storage resources required for the coding segment, making it suitable for resource-constrained satellite ends. At the decoding end, a generative model (i.e., the first and second extension modules) is used to generate rich inter-frame information, thereby ensuring the quality of the reconstructed video frame.

[0013] (2) The multi-scale information generation remote sensing image frame compression encoding and decoding method proposed in this invention first extracts preliminary temporal semantic features through the feature extraction module at the decoding end. However, due to the quantization operation in the encoding process, some information in the reconstructed video frame may be lost. Therefore, the extracted preliminary temporal semantic features are used to generate enhanced temporal semantic feature representations through the first diffusion model. At the same time, the multi-scale temporal feature mining module used at the decoding end can mine the rich semantic representations of temporal features at multiple scales, further ensuring the effective use of temporal information.

[0014] (3) The multi-scale information generation remote sensing image inter-frame compression encoding and decoding method proposed in this invention uses a diffusion model to generate robust reconstruction feature representations at the decoding end. With the powerful generation capability of the diffusion model, at the decoding end, this invention uses the potential representation of the source view video frame, the enhanced temporal semantic features, and the multi-scale temporal semantic features as prior conditions and inputs them into the diffusion model to generate a more semantically complete reconstruction feature representation, thereby ensuring a higher fidelity reconstruction image at the inverse transformation module. At the same time, multi-scale temporal semantic features are also introduced in the inverse transformation process to further ensure the reconstruction quality. Attached Figure Description

[0015] Figure 1 This is a structural diagram of a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding system provided in an embodiment of the present invention; Figure 2 This is a structural diagram of the multi-scale temporal feature mining module in the multi-scale information-generating remote sensing image inter-frame compression encoding and decoding system provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0017] This invention provides a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding system, such as... Figure 1 As shown, it includes the encoding end and the decoding end; The encoding end includes a feature transformation module, a quantization module, and an arithmetic encoder; The feature transformation module is used to transform the first feature in the source view video. Frame Image Performing nonlinear transformation operations yields a continuous latent representation. The quantization module is used for... Quantization yields a discretized latent representation. ; The arithmetic encoder is used for... Compress the data to obtain a compressed bitstream; The decoding end includes an arithmetic decoder, a first diffusion model, a second diffusion model, a temporal feature extraction module, a multi-scale temporal feature mining module, and a feature inverse transformation module; The arithmetic decoder is used to restore the compressed bitstream to... The temporal feature extraction module is used to extract the first time sequence feature from the video source's perspective. Frame, First Frame Image , Corresponding reconstructed image , The temporal characteristics as Temporal characteristics The first diffusion model is used for... Enhancement processing is performed to obtain enhanced temporal features. The multi-scale temporal feature mining module is used to... and Generate multi-scale temporal features The second diffusion model is used for multi-scale features. Under the guidance of , Generate reconstructed feature representation F; the inverse feature transformation module is used to... Under guidance, a nonlinear inverse transform operation is performed on F to obtain the result. The corresponding number Frame Reconstruction Image , The number of multi-scale features, .

[0018] Specifically, the video from the source perspective acquired from the satellite will be divided into multiple groups of video frames, and each video frame in each group will be analyzed separately. Compression and encoding / decoding processes are performed. For example, suppose the video from the source viewpoint acquired from the satellite is divided into N groups of video frames, and each group of video frames includes K video frames, then i=1,2,…,N×K.

[0019] At the encoding end, the feature transformation module performs... A nonlinear transformation operation is performed to transform the pixel-domain information image into a compact, continuous latent representation. The quantization module will convert the continuous latent representation Quantization is performed on the latent representation of the continuous reading. Quantization into an encodeable discretized latent representation , The compressed bitstream is then obtained through an arithmetic encoder.

[0020] At the decoding end, the arithmetic decoder restores the compressed bitstream back to a compact latent representation. ; The temporal feature extraction module extracts the current source view video frames from the buffer. The previous two reconstructed video frames , The temporal feature information is used as the current video frame. Temporal characteristic information ; The time-series feature information The data is further input into the first diffusion model, which utilizes... Generate enhanced temporal feature information with richer semantic information To fully utilize the temporal feature information, the temporal feature information and enhanced temporal feature information The input is fed into the multi-scale temporal feature mining module, which outputs multi-scale feature information representations with rich temporal semantic information. ; The second diffusion model utilizes a reduced, compact latent representation. and enhanced temporal feature information and in multi-scale temporal features Guided by the algorithm, a robust and semantically richer reconstructed feature representation F is generated; the feature inverse transformation module is used for multi-scale temporal features. Guided by this principle, a nonlinear inverse transform operation is performed on F to restore the feature domain representation back to the high-fidelity pixel domain for video frame reconstruction. By introducing multi-scale features Prior knowledge can further ensure the acquisition of high-quality reconstructed frames.

[0021] Preferably, the multi-scale temporal feature mining module includes: The first upsampling module is connected in sequence. , A second upsampling module connected in sequence , A series of connected downsampling modules , Feature extraction module and Convolutional layers ; in, , Enter them separately , ; and The features formed by splicing the output As The input, at the same time, through After processing and The output fusion, as Input, ; and The features formed by splicing the output through After processing, it is also used as , Input; Used to output the first Features at multiple scales, i.e., multi-scale features ; The first and second upsampling modules each include a residual block and an upsampling layer connected in sequence; the downsampling module includes a residual block and a downsampling layer connected in sequence; and the feature extraction module includes a convolutional layer and a residual block connected in sequence.

[0022] by For example, the structure of the multi-scale temporal feature mining module is as follows: Figure 2 As shown.

[0023] It is understood that the feature transformation module can be constructed using any sequentially connected method. Each downsampling layer reduces the dimensionality of high-dimensional pixel-domain video frames to a compact, continuous latent representation. This embodiment of the invention does not impose a unique limitation on this method.

[0024] Preferably, the feature transformation module comprises sequentially connected... A mixed-wash downsampling layer.

[0025] Accordingly, the feature inverse transformation module can be constructed using any sequentially connected method. The upsampling layer, and the first The inputs of each upsampling layer also correspond one-to-one with the following: The embodiments of the present invention do not limit this to uniqueness.

[0026] Preferably, the feature inverse transformation module comprises sequentially connected... The first mixed-wash upsampling layer, and the second The inputs of each shuffled upsampling layer also correspond one-to-one with the following: .

[0027] by For example, after the first-level shuffling upsampling layer of the F input feature inverse transform module, the first-level upsampling features are obtained; the first-level upsampling features and The fused features obtained from the splicing and fusion are input into the second-level shuffling upsampling layer to obtain the second-level upsampling features; the second-level upsampling features and The fused features obtained from the splicing and fusion are input into the third-level shuffling upsampling layer to obtain the third-level features; the third-level upsampling features and multi-scale feature information are represented. The fused features obtained from splicing and fusion are input into the fourth-level shuffling upsampling layer to obtain the final high-fidelity source view reconstruction frame. .

[0028] The temporal feature extraction module can adopt any existing structure capable of temporal feature extraction, and this embodiment of the invention does not limit its uniqueness. As an example, the temporal feature extraction module includes multiple residual blocks connected in sequence.

[0029] The first and second diffusion models can adopt the structure of any existing diffusion model, and the embodiments of the present invention do not limit the uniqueness of them.

[0030] by For example, both the first and second diffusion models can include an encoder with four downsampling layers and a decoder with four upsampling layers, with skip connections between the encoder and decoder. The features output from the encoder and the first-level downsampling layer are spliced ​​and fused along the channel dimension and then input into the second-level downsampling layer of the encoder to obtain the second-level features. After being concatenated and fused with the second-level features in the channel dimension, the third-level features are input into the third-level downsampling layer of the encoder to obtain the third-level features; After being spliced ​​and fused with the third-level features in the channel dimension, the fourth-level features are input into the fourth-level downsampling layer of the encoder to obtain the fourth-level features; the fourth-level features are then input into the decoder to obtain the reconstructed feature representation F.

[0031] This invention provides a training method for a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding system as described in any of the above embodiments, comprising: The system is trained by inputting sample images from the source video and minimizing the difference between the reconstructed image output by the system and the sample images. The system's feature transformation module, quantization module, temporal feature extraction module, multi-scale temporal feature mining module, and feature inverse transformation module are trained.

[0032] Preferably, uniform noise is added during training to simulate quantization and ensure the backpropagation of gradient information in the network model; during inference, sampling and rounding are directly performed to realize the quantization operation of the quantization module.

[0033] This invention provides a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding method, comprising: The image to be processed in the video from the source's perspective is input into the system described in any of the above embodiments to obtain the corresponding reconstructed image.

[0034] This invention provides an electronic device, including: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the training method or encoding / decoding method as described in any of the above embodiments.

[0035] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to execute the training method or encoding / decoding method as described in any of the above embodiments.

[0036] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the training method or encoding / decoding method as described in any of the above embodiments.

[0037] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-scale information-generating remote sensing image inter-frame compression encoding and decoding system, characterized in that, Including the encoding and decoding ends: The encoding end includes a feature transformation module, a quantization module, and an arithmetic encoder; The feature transformation module is used to transform the first feature in the source view video. Frame Image Performing nonlinear transformation operations yields a continuous latent representation. The quantization module is used for... Quantization yields a discretized latent representation. ; The arithmetic encoder is used for... Compress the data to obtain a compressed bitstream; The decoding end includes an arithmetic decoder, a first diffusion model, a second diffusion model, a temporal feature extraction module, a multi-scale temporal feature mining module, and a feature inverse transformation module; The arithmetic decoder is used to restore the compressed bitstream to... The temporal feature extraction module is used to extract the first... Frame, First Frame Reconstruction Image , The temporal characteristics as Temporal characteristics The first diffusion model is used for... Enhancement processing is performed to obtain enhanced temporal features. The multi-scale temporal feature mining module is used to... and Generate multi-scale temporal features The second diffusion model is used for multi-scale features. Under the guidance of , Generate reconstructed feature representation F; the inverse feature transformation module is used to... Under guidance, a nonlinear inverse transform operation is performed on F to obtain the first... Frame Reconstruction Image ; This represents the number of multi-scale features.

2. The system as described in claim 1, characterized in that, The multi-scale temporal feature mining module includes The first upsampling module is connected in sequence. , A second upsampling module connected in sequence , A series of connected downsampling modules , Feature extraction module and Convolutional layers ; in, , Enter them separately , ; and The features formed by splicing the output As The input, at the same time, through After processing and The output fusion, as Input, ; and The features formed by splicing the output through After processing, it is also used as , Input; Used to output the first Features at multiple scales, i.e., multi-scale features ; The first and second upsampling modules each include a residual block and an upsampling layer connected in sequence; the downsampling module includes a residual block and a downsampling layer connected in sequence; and the feature extraction module includes a convolutional layer and a residual block connected in sequence.

3. The system as described in claim 1, characterized in that, The feature transformation module includes sequentially connected... A mixed-wash downsampling layer.

4. The system as described in claim 1, characterized in that, The feature inverse transformation module includes sequentially connected... The first mixed-wash upsampling layer, and the second The inputs of each shuffled upsampling layer also correspond one-to-one with the following: .

5. A training method for a multi-scale information-generating remote sensing image inter-frame compression encoding and decoding system as described in any one of claims 1-4, characterized in that, include: The system is trained by inputting sample images from the source video and minimizing the difference between the reconstructed image output by the system and the sample images. The system's feature transformation module, quantization module, temporal feature extraction module, multi-scale temporal feature mining module, and feature inverse transformation module are trained.

6. A multi-scale information-generating remote sensing image inter-frame compression encoding and decoding method, characterized in that, include: The image to be processed in the video from the source's perspective is input into the system as described in any one of claims 1-4 to obtain the corresponding reconstructed image.

7. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in claim 5 or 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in claim 5 or 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the method as described in claim 5 or 6.

Images