A method and device for enhancing the quality of compressed mars image based on semantic prior

Abstract

The method comprises the following steps: acquiring a compressed Mars image to be processed; performing matching on the compressed Mars image to be processed by a semantic matching module to obtain a reference image pair; inputting the compressed Mars image to be processed and the reference image pair into a trained Mars image quality enhancement model to obtain an enhanced Mars image. The above technical solution can solve the technical problem that the existing quality enhancement technology is insufficient to improve the quality of the Mars image. Through quantitative and qualitative analysis of the characteristics of the Mars image and design of the network structure and the module, the characteristics of the Mars image are fully utilized to assist the quality enhancement process, which can effectively enhance the quality of the compressed Mars image and save the bandwidth resources of the ground-to-Mars data transmission.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method and apparatus for enhancing the quality of compressed Mars images based on semantic priors. Background Technology

[0002] Among related technologies, deep learning-based methods, by employing a network structure with early exit mechanisms to balance network performance and resource consumption, can effectively enhance the quality of compressed Earth images while maximizing computational resource savings. This technology is designed for Earth image databases containing various natural and man-made scenes, such as plants, artificial objects, and cityscapes. However, Mars images differ from Earth images due to the diversity in texture patterns and semantics exhibited by Earth images. Therefore, existing quality enhancement techniques designed for Earth images are insufficient to improve the quality of Mars images. Summary of the Invention

[0003] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a method and apparatus for enhancing the quality of compressed Mars images based on semantic priors.

[0004] This disclosure provides a method for enhancing the quality of compressed Mars images based on semantic priors, the method comprising:

[0005] Acquire compressed Mars images to be processed;

[0006] The semantic-based matching module matches the compressed Mars image to be processed to obtain reference image pairs;

[0007] The compressed Mars image to be processed and the reference image are used to process the pre-trained Mars image quality enhancement model to obtain an enhanced Mars image.

[0008] Optionally, the semantic-based matching module matches the compressed Mars image to be processed to obtain reference image pairs, including:

[0009] The compressed Mars image to be processed is segmented to obtain multiple Mars image blocks;

[0010] The compressed Mars image to be processed is classified based on a preset Mars image segmentation network, and the classification results of the multiple Mars image blocks are obtained according to their corresponding positions.

[0011] Based on each classification result, a match is made in the semantic class dictionary to obtain multiple candidate lossless reference image blocks, and a candidate compressed reference image block corresponding to each candidate lossless reference image block is obtained.

[0012] Based on the minimum divergence between the local binary mode (LBP) features of each candidate compressed reference image block and the LBP features of each Mars image block, a target lossless reference image block is determined from the plurality of candidate lossless reference image blocks, and a target compressed reference image block is determined from the plurality of candidate compressed reference image blocks.

[0013] Multiple target lossless reference image blocks are stitched together to form a lossless reference image, and multiple target compressed reference image blocks are stitched together to form a lossy reference image. The lossless reference image and the lossy reference image are then combined to form the reference image pair.

[0014] Optionally, the method for enhancing the quality of compressed Mars images based on semantic priors further includes:

[0015] Acquire compressed Mars image samples to be processed;

[0016] The semantic matching module matches the compressed Mars image samples to be processed to obtain reference image pairs.

[0017] The Mars image quality enhancement model is trained based on the compressed Mars image samples to be processed and the reference image samples to obtain the trained Mars image quality enhancement model.

[0018] Optionally, the step of processing the compressed Mars image to be processed and the reference image into a pre-trained Mars image quality enhancement model to obtain an enhanced Mars image includes:

[0019] The compressed Mars image to be processed and the reference image are input into a convolutional network to obtain the first compressed Mars image features, the first lossless reference image features, and the first compressed reference image features.

[0020] The first compressed Mars image feature to be processed, the first lossless reference image feature, and the first compressed reference image feature are input into the first texture extraction module to obtain the first transferable feature.

[0021] After sequentially inputting the first compressed Mars image feature to be processed, the first lossless reference image feature, and the first compressed reference image feature into the downsampling layer and the convolutional layer, the second compressed Mars image feature to be processed, the second lossless reference image feature, and the second compressed reference image feature are obtained and then input into the second texture extraction module to obtain the second transferable feature.

[0022] The second compressed Mars image feature to be processed, the second lossless reference image feature, and the second compressed reference image feature are sequentially input into the downsampling layer and the convolutional layer to obtain the third compressed Mars image feature to be processed, the third lossless reference image feature, and the third compressed reference image feature. These are then input into the third texture extraction module to obtain the third transferable feature.

[0023] The third compressed Mars image feature to be processed, the third lossless reference image feature, and the third compressed reference image feature are sequentially input into the downsampling layer, the convolutional layer, and the upsampling layer, and then added to the third compressed Mars image feature to be processed and input into the first texture fusion module. At the same time, the third transferable feature is input into the first texture fusion module to obtain the third fused feature.

[0024] The third fusion feature is upsampled and added to the second compressed Mars image feature to be processed, and then input into the second texture fusion module. At the same time, the second transferable feature is input into the second texture fusion module to obtain the second fusion feature.

[0025] The second fusion feature is upsampled and added to the features of the compressed Mars image to be processed, and then input into the third texture fusion module. At the same time, the first transferable feature is input into the third texture fusion module to obtain the first fusion feature, which is then input into the supervised attention module to obtain the preliminary enhanced image as the enhanced Mars image.

[0026] Optionally, the method for enhancing the quality of compressed Mars images based on semantic priors further includes:

[0027] Based on the first fusion feature and input into the supervised attention module, attention-enhanced features are obtained;

[0028] The compressed Mars image to be processed is input into a convolutional layer to obtain the fourth compressed Mars image feature to be processed. After merging the channel dimension with the attention enhancement feature, it is input into the first quality enhancement module to obtain the first encoding feature.

[0029] The first compressed Mars image features to be processed and the first fused features are respectively input into a convolutional layer, added together, and then added to the first encoded features. After downsampling, they are input into the second quality enhancement module to obtain the second encoded features.

[0030] The second compressed Mars image features to be processed and the second fused features are respectively input into a convolutional layer, added together, and then added to the second encoded features. After downsampling, they are input into the third quality enhancement module to obtain the third encoded features.

[0031] The third compressed Mars image feature to be processed and the third fusion feature are respectively input into the convolutional layer, added together, and then added to the second encoded feature. After downsampling, they are input into the nonlocal module and then upsampled to obtain the fourth encoded feature.

[0032] The fourth coding feature is added to the third coding feature and then input into the fourth quality enhancement module and the upsampling layer to obtain the fifth coding feature;

[0033] The fifth coding feature is added to the second coding feature and then input into the fifth quality enhancement module and the upsampling layer to obtain the sixth coding feature;

[0034] The sixth encoded feature is added to the first encoded feature and then input into the sixth quality enhancement module and the upper convolutional layer to obtain the final enhanced image as the enhanced Mars image.

[0035] Optionally, the features of the compressed Mars image to be processed, the features of the lossless reference image, and the features of the compressed reference image are input into the texture extraction module to obtain transferable features, including:

[0036] The features of the compressed Mars image to be processed, the features of the lossless reference image, and the features of the compressed reference image are expanded according to a preset size to obtain the feature blocks of the compressed Mars image to be processed, the feature blocks of the lossless reference image, and the feature blocks of the compressed reference image.

[0037] Based on a preset mathematical formula, the feature blocks of the compressed Mars image to be processed, the feature blocks of the lossless reference image, and the feature blocks of the compressed reference image are calculated to obtain transferable feature blocks;

[0038] All the described transferable feature blocks are merged to obtain the transferable feature.

[0039] Optionally, the upsampled fused features are added to the features of the compressed Mars image to be processed as the first input to the texture fusion module. Simultaneously, transferable features are input to the texture fusion module as the second input to obtain the target fused features, including:

[0040] The intermediate features are obtained by processing the second input with a pre-defined convolutional layer and then subtracting the first input.

[0041] After the channel attention layer processes the first input, it performs an element-wise inner product operation with the intermediate features, adds the intermediate features to the first input, and then processes the input through the pre-set size convolutional layer to obtain the target fusion features.

[0042] This disclosure also provides a semantic prior-based compressed Mars image quality enhancement device, the device comprising:

[0043] The acquisition module is used to acquire compressed Mars images to be processed;

[0044] The matching module is used to match the compressed Mars image to be processed based on the semantic matching module to obtain reference image pairs;

[0045] The processing module is used to process the compressed Mars image to be processed and the reference image into a pre-trained Mars image quality enhancement model to obtain an enhanced Mars image.

[0046] This disclosure also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the semantic prior-based compressed Mars image quality enhancement method provided in this disclosure.

[0047] This disclosure also provides a computer-readable storage medium storing a computer program for performing a semantic prior-based compressed Mars image quality enhancement method as provided in this disclosure.

[0048] Compared with existing technologies, the technical solution provided in this disclosure has the following advantages: The semantic prior-based compressed Mars image quality enhancement scheme provided in this disclosure acquires a compressed Mars image to be processed, and a semantic matching module matches the compressed Mars image to be processed to obtain a reference image pair; the compressed Mars image to be processed and the reference image pair are input into a trained Mars image quality enhancement model for processing to obtain an enhanced Mars image. By adopting the above technical solution, the technical problem that existing quality enhancement technologies are insufficient to improve the quality of Mars images can be solved. Through quantitative and qualitative analysis of Mars image characteristics and the design of network structures and modules to fully utilize the prior characteristics of Mars images to assist in the quality enhancement process, the quality of compressed Mars images can be effectively enhanced, saving bandwidth resources for data transmission between Earth and Mars. Attached Figure Description

[0049] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0050] Figure 1 A flowchart illustrating a method for enhancing the quality of compressed Mars images based on semantic priors, provided in an embodiment of this disclosure;

[0051] Figure 2A flowchart illustrating a semantic prior-based compressed Mars image quality enhancement network structure processing method provided in this disclosure embodiment;

[0052] Figure 3 This is a flowchart illustrating a TEM module processing method provided in an embodiment of the present disclosure.

[0053] Figure 4 This disclosure provides a flowchart example of TFM module processing.

[0054] Figure 5 This disclosure provides a flowchart example of QEM module processing.

[0055] Figure 6 A flowchart illustrating another method for enhancing the quality of compressed Mars images based on semantic prior is provided for embodiments of this disclosure.

[0056] Figure 7 This is a structural example diagram of a semantic prior-based compressed Mars image quality enhancement device provided in an embodiment of this disclosure. Detailed Implementation

[0057] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0058] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0059] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0060] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0061] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0062] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0063] Specifically, Mars images differ from Earth images, possessing the following unique characteristics: significant intra- and inter-image similarities and more compact texture representations. Existing quality enhancement techniques designed for Earth images are insufficient to improve the quality of Mars images.

[0064] Furthermore, a Mars image database was constructed based on NASA's database, and a Mars image compression network was then designed based on the non-local self-similarity prior of Mars images. This network employs non-local modules to utilize both local and non-local priors within Mars images, effectively compressing them and thus saving bitrate. This technique, based on the local self-similarity prior of Mars images, designs a Mars image compression network to achieve effective compression. However, in addition to intra-image similarity, Mars images also exhibit inter-image similarity and compact texture representation; therefore, this technique does not fully utilize these characteristics and still has room for optimization. Moreover, this technique is geared towards Mars image compression tasks, which differs somewhat from the image quality enhancement task of this technique.

[0065] The disclosed technology (MarsQE) employs deep learning methods and designs a network specifically based on the characteristics of Mars images, which can effectively enhance the quality of compressed Mars images and save bandwidth resources for data transmission between Earth and Mars.

[0066] Therefore, this disclosure proposes a method for enhancing the quality of compressed Mars images based on semantic priors. It designs a quality enhancement network for compressed Mars images using deep learning. Since existing techniques do not fully utilize the characteristics of Mars images, this disclosure first quantitatively and qualitatively analyzes these characteristics and designs network structures and modules to fully utilize these prior characteristics to assist the quality enhancement process, thereby improving both the objective and subjective results of the enhanced quality.

[0067] Specifically, Figure 1 This is a flowchart illustrating a semantic prior-based compressed Mars image quality enhancement method provided in an embodiment of this disclosure. This method can be executed by a semantic prior-based compressed Mars image quality enhancement device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0068] Step 101: Obtain the compressed Mars image to be processed.

[0069] Step 102: The semantic matching module matches the compressed Mars image to be processed to obtain reference image pairs.

[0070] In this embodiment of the disclosure, the compressed Mars image to be processed refers to the compressed Mars image to be enhanced.

[0071] In some embodiments, the semantic-based matching module matches the compressed Mars image to be processed to obtain reference image pairs, including: segmenting the compressed Mars image to be processed to obtain multiple Mars image blocks; classifying the compressed Mars image to be processed based on a preset Mars image segmentation network, and obtaining classification results of multiple Mars image blocks according to their corresponding positions; matching each classification result in a semantic class dictionary to obtain multiple candidate lossless reference image blocks, and obtaining a candidate compressed reference image block corresponding to each candidate lossless reference image block; determining a target lossless reference image block and a target compressed reference image block from multiple candidate lossless reference image blocks based on the minimum divergence between the local binary pattern (LBP) features of each candidate compressed reference image block and the LBP features of each Mars image block; stitching multiple target lossless reference image blocks into a lossless reference image, and stitching multiple target compressed reference blocks into a lossy reference image, and combining the lossless reference image and the lossy reference image into the reference image pairs for subsequent processing.

[0072] Specifically, through systematic analysis of a Mars image compression dataset, two main features of Mars images were identified: significant intra- and inter-image similarity and more compact texture representation. Based on these two main features, a Mars image quality enhancement technique, named MarsQE, was subsequently proposed. Specifically, MarsQE employs a two-stage quality enhancement network that fully leverages the significantly higher pixel similarity in Mars images compared to Earth images, specifically combining... Figure 2 Provide a detailed description.

[0073] In the first stage, such as Figure 2 The network shown is divided into two parts: a reference image matching part ( Figure 2 Midpoint wireframe section) and quality enhancement section ( Figure 2(Upper part). Specifically, this disclosure proposes a semantic-based matching module (SMM) in the reference image matching part, which uses the compact semantic representation of Mars images to match reference images with similar textures; in the quality enhancement part, a novel quality enhancement network architecture is constructed, which can use high-quality reference images to assist the quality enhancement process.

[0074] Specifically, in the reference image patch matching part, the SMM module adopts a reference image mode based on semantic matching: first, a concise but comprehensive reference dictionary is constructed, and then reference images with similar textures are matched based on the reference dictionary.

[0075] More specifically, SMM first segments the original lossless image, i.e., the Mars image to be compressed, into non-overlapping S×S image patches. Then, using an existing Mars image segmentation network, these image patches are classified into K semantic categories. Finally, for each semantic category, a certain number of image patches are selected to construct a reference dictionary. For the k-th category, N is selected. k A dictionary is constructed using image patches, where k is in the range of 1 to K. As an example, the hyperparameters S, K, and N are... k They are fixed at 128, 4, and 6 respectively.

[0076] Furthermore, in the process of matching the reference image based on the dictionary, the input compressed Mars image I to be processed is similarly used. comp The Mars image is divided into multiple non-overlapping S×S resolution patches*P comp + and classify according to the Mars image segmentation network. For each Mars image patch P comp The matching process involves matching the results from the corresponding semantic class dictionary. This matching process can be represented mathematically as follows:

[0077]

[0078] Among them, D KL (x|y) and LBP(·) represent the KL divergence and LBP feature descriptor, respectively. represent The corresponding compressed image. Thus, we obtain the compressed input image block P. comp Similar reference image patches and By matching and stitching together each image block, a reference image pair corresponding to the entire compressed image can be obtained.

[0079] Step 103: Process the compressed Mars image to be processed and the reference image into the pre-trained Mars image quality enhancement model to obtain the enhanced Mars image.

[0080] In some embodiments, a compressed Mars image sample to be processed is obtained, a semantic matching module matches the compressed Mars image sample to be processed to obtain a reference image pair sample, and a Mars image quality enhancement model is trained based on the compressed Mars image sample to be processed and the reference image pair sample to obtain a trained Mars image quality enhancement model.

[0081] Specifically, to obtain a network that can effectively enhance compressed Mars images, training data can be selected and the network trained. The details are as follows: A Mars image compression dataset is selected, whose image data originates from the original images collected by NASA's Perseverance rover. Compressed original images are used to obtain compressed-original Mars image pairs, which are then used to train the network.

[0082] For example, the network is trained using gradient descent, with the following details: The Adam optimizer is used, with an initial learning rate of 10e-4, and periodic cosine annealing is employed to gradually decrease the learning rate. The network is trained under eight compression settings (JPEG compression parameters QF = 20, 30, 40, and 50; BPG compression parameters QP = 27, 32, 37, and 42). The resulting converged model allows the network to enhance images from compressed Mars images.

[0083] After training convergence, extensive performance validation experiments were conducted on a Mars image dataset to verify the state-of-the-art performance of the MarsQE technique. Robustness validation was also performed on multiple Mars mission datasets, and the experimental results demonstrate that the network effectively improves the quality of the returned compressed images of the Martian surface.

[0084] In some embodiments, the process of processing a pre-trained Mars image quality enhancement model with a Mars image to be processed and a reference image to obtain an enhanced Mars image includes: inputting the Mars image to be processed and the reference image into a convolutional network to obtain a first Mars image to be processed compressed feature, a first lossless reference image feature, and a first compressed reference image feature; inputting the first Mars image to be processed compressed feature, the first lossless reference image feature, and the first compressed reference image feature into a first texture extraction module to obtain a first transferable feature; sequentially inputting the first Mars image to be processed compressed feature, the first lossless reference image feature, and the first compressed reference image feature into a downsampling layer and a convolutional layer to obtain a second Mars image to be processed compressed feature, a second lossless reference image feature, and a second compressed reference image feature, and inputting these into a second texture extraction module to obtain a second transferable feature; and sequentially inputting the second Mars image to be processed compressed feature, the second lossless reference image feature, and the second compressed reference image feature into a downsampling layer and a convolutional layer... The process involves obtaining the third compressed Mars image features to be processed, the third lossless reference image features, and the third compressed reference image features, which are then input into the third texture extraction module to obtain the third transferable feature. These features are then sequentially input into a downsampling layer, a convolutional layer, and an upsampling layer, and added to the third compressed Mars image features before being input into the first texture fusion module. Simultaneously, the third transferable feature is also input into the first texture fusion module to obtain the third fused feature. The third fused feature is then upsampled and added to the second compressed Mars image features to be processed before being input into the second texture fusion module. Simultaneously, the second transferable feature is input into the second texture fusion module to obtain the second fused feature. Finally, the second fused feature is upsampled and added to the compressed Mars image features to be processed before being input into the third texture fusion module. Simultaneously, the first transferable feature is input into the third texture fusion module to obtain the first fused feature, which is then input into the supervised attention module to obtain the preliminary enhanced image as the enhanced Mars image.

[0085] Specifically, such as Figure 2 As shown, in the Semantic-informed QualityEnhancement part, an L-layer encoder-decoder architecture network is constructed, and a Texture Extraction Module (TEM) and a Texture Fusing Module (TFM) are designed to utilize similar reference images matched by SMM to assist the quality enhancement process. Specifically, in the encoder part of the network ( Figure 2 (Top left) First, the TEM module extracts transferable features at multiple scales from the encoder features, and then in the decoder part ( Figure 2(Top right) These transferable features are fused with decoder features using the TFM module. For example, L is set to 4.

[0086] In some embodiments, the compressed Mars image features to be processed, the lossless reference image features, and the compressed reference image features are input into the texture extraction module to obtain transferable features, including: expanding the compressed Mars image features to be processed, the lossless reference image features, and the compressed reference image features according to a preset size to obtain compressed Mars image feature blocks to be processed, lossless reference image feature blocks, and compressed reference image feature blocks; calculating the compressed Mars image feature blocks to be processed, the lossless reference image feature blocks, and the compressed reference image feature blocks based on a preset mathematical formula to obtain transferable feature blocks; and merging all transferable feature blocks to obtain transferable features.

[0087] Specifically, such as Figure 3 The TEM module processing procedure shown in the diagram first involves processing the three input encoder features F... comp , and F ref Expanded into a feature block of size d×d and *B ref +. For each feature block Transferable feature blocks can be obtained. Using the following mathematical formula:

[0088]

[0089] Among them, D i S represents the position coordinates. i This represents the degree of similarity. The final transferable feature block *B trans + is recombined into transferable feature F trans The output TEM module enables the extraction of transferable features from the encoder features. In this disclosure, for example, the hyperparameter d can be set to 4.

[0090] In some embodiments, the fused features are upsampled and added to the features of the compressed Mars image to be processed as the first input of the texture fusion module. At the same time, the transferable features are input to the texture fusion module as the second input to obtain the target fused features. This includes: processing the second input through a convolutional layer of a pre-set size and then subtracting the first input to obtain intermediate features; processing the first input through a channel attention layer and then performing an element-wise inner product operation with the intermediate features, adding the intermediate features to the first input, and then processing the intermediate features through a convolutional layer of a pre-set size to obtain the target fused features.

[0091] Specifically, such as Figure 4 The TFM module processing procedure shown in the diagram involves processing the input encoder features. And the transferable features F extracted above trans The fusion is performed to obtain the fused feature F. fused The specific fusion method can be expressed by the following mathematical formula:

[0092]

[0093] Among them, C 3×3 (·) represents a convolutional layer with a kernel size of 3×3, and CA(·) represents a channel attention layer. For example... Figure 2 As shown, at the output terminal The images are stitched together to the size of the input image and then output through the SAM module.

[0094] Specifically, the SAM module can selectively propagate useful features F. sam Entering the second stage, it also includes a simple and universal output terminal that can decode features. Converted to residual and compared with the original compressed image I comp Adding them together, we finally obtain the preliminary enhanced image I. enh .

[0095] In some embodiments, the method further includes: obtaining attention-enhancing features based on a first fusion feature and inputting it into the supervised attention module; inputting the compressed Mars image to be processed into a convolutional layer to obtain a fourth compressed Mars image feature to be processed, merging it with the attention-enhancing feature by channel dimension, and then inputting it into a first quality enhancement module to obtain a first encoding feature; inputting the first compressed Mars image feature to be processed and the first fusion feature into a convolutional layer respectively, adding them together, adding them together with the first encoding feature, and then downsampling and inputting them into a second quality enhancement module to obtain a second encoding feature; inputting the second compressed Mars image feature to be processed and the second fusion feature into a convolutional layer respectively, adding them together, adding them together with the second encoding feature, and then downsampling. The sample is input into the third quality enhancement module to obtain the third encoded feature; the third compressed Mars image feature to be processed and the third fusion feature are input into the convolutional layer and added together, and then added to the second encoded feature. After downsampling, the sample is input into the nonlocal module and then upsampled to obtain the fourth encoded feature; the fourth encoded feature is added to the third encoded feature and then input into the fourth quality enhancement module and then upsampled to obtain the fifth encoded feature; the fifth encoded feature is added to the second encoded feature and then input into the fifth quality enhancement module and then upsampled to obtain the sixth encoded feature; the sixth encoded feature is added to the first encoded feature and then input into the sixth quality enhancement module and then upconvolutional layer to obtain the final enhanced image as the enhanced Mars image.

[0096] like Figure 2 In the second stage shown, the network further utilizes intra-image similarity to recover texture details, resulting in an image that is significantly improved compared to the initial enhanced image I. enh Higher quality images More specifically, a Quality Enhancement Module (QEM) is designed to restore image texture. An existing Non-Local Module (NLM) (derived from the module in the paper Non-local Neural Networks) is used to reduce block artifacts by utilizing similar texture features in different regions within the image.

[0097] Specifically, such as Figure 5 The processing procedure of the QEM module is shown below. The QEM module can be represented by the following mathematical formula:

[0098] X i =C 3×3 (CA(L LN (F i )))+F i (Formula 7)

[0099] F i+1 =C 1×1 (L GELU (C 1×1 (L LN (X i ))))+X i (Formula 8)

[0100] Among them, F i and F i+1 Represents the inputs and outputs of the QEM module, C 1×1 (·), CA(·), L LN (·) and L GELU (·) represent a convolutional layer with a kernel size of 1×1, a channel attention layer, layer normalization, and the GELU activation function, respectively.

[0101] Specifically, the second-stage network is also designed as an encoder-decoder architecture: such as Figure 2 As shown in the lower part, the input compressed Mars image is progressively encoded using an encoder consisting of convolutional layers, a QEM module, and downsampling layers to obtain encoded features at different scales. At each scale, these features are fused with the encoded and decoded features from the first-stage quality enhancement network. The deepest features are processed by an NLM module to enhance them using intra-image similarity. At the decoding end, the enhanced features are progressively decoded by a decoder consisting of QEM, convolutional layers, and upsampling layers, and fused with the encoded features transmitted through skip connections. Finally, the enhanced image is output.

[0102] As an example of a scenario, such as Figure 6As shown, the steps are as follows: Step 6.1: Martian surface image data collection; Step 6.2: Martian image data characteristic analysis; Step 6.3: Training set samples; Step 6.4: Test set samples; Step 6.5: Model training; Step 6.6: Martian image quality enhancement model; Step 6.7: Output of enhanced high-quality Martian images; Step 6.8: Verification and analysis of model effectiveness and robustness.

[0103] Specifically, the technology disclosed herein can effectively enhance the quality of compressed Mars images. On a test set of a Mars image compression dataset, for example, using the recognized objective metric PSNR to measure the superiority of the technology disclosed herein: the technology disclosed herein can improve the PSNR by an average of 2.26, 2.17, 2.19, and 2.25 dB when the compression parameter QF is 20, 30, 40, and 50. Compared to the prior art, the technology disclosed herein can further improve the PSNR by 0.30, 0.33, 0.30, and 0.31 dB when the compression parameter QF is 20, 30, 40, and 50. Furthermore, the present disclosure also uses the recognized standard of rate-distortion curves to measure the superiority of the technology disclosed herein: for Mars images of the same quality, the technology disclosed herein can save 38.94% and 35.93% of the bitrate compared to JPEG and BPG, while the prior art can only save 34.20% and 30.30% of the bitrate. Furthermore, the robustness of the method disclosed herein is verified: without fine-tuning the network, this disclosure uses a model trained on a Mars image compression dataset with QF=50 to test data returned by the Curiosity rover mission. On the Curiosity rover datasets (i.e., AI4Mars and Mars32K), this disclosure uses the recognized non-reference metrics BRISQUE and PI to measure the enhanced results, finding that the technique disclosed herein can effectively enhance compressed images from previous deep space exploration missions and can be applied to Mars images from multiple missions.

[0104] Therefore, a quality enhancement network is designed based on semantic similarity and compact texture representation. The two steps of semantic reference matching and semantic quality enhancement effectively utilize texture similarity in Mars images to assist the quality enhancement process. Furthermore, extensive experiments have been conducted to verify that the MarsQE technique disclosed herein outperforms other quality enhancement algorithms for Earth images, achieving the best enhancement results both subjectively and objectively. Without fine-tuning the network, this disclosure also verifies the generalization ability of MarsQE on Mars images from multiple Mars exploration missions, thus demonstrating the robustness of the technique.

[0105] Figure 7 This is a schematic diagram of a semantic prior-based compressed Mars image quality enhancement device provided in an embodiment of this disclosure. The device can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 7 As shown, the device includes:

[0106] Acquisition module 201 is used to acquire compressed Mars images to be processed;

[0107] The matching module 202 is used to match the compressed Mars image to be processed based on the semantic matching module to obtain a reference image pair;

[0108] The processing module 203 is used to process the compressed Mars image to be processed and the reference image into the input trained Mars image quality enhancement model to obtain an enhanced Mars image.

[0109] Optionally, the matching module 202 is specifically used for:

[0110] The compressed Mars image to be processed is segmented to obtain multiple Mars image blocks;

[0111] The compressed Mars image to be processed is classified based on a preset Mars image segmentation network, and the classification results of the multiple Mars image blocks are obtained according to their corresponding positions.

[0112] Based on each classification result, a match is made in the semantic class dictionary to obtain multiple candidate lossless reference image blocks, and a candidate compressed reference image block corresponding to each candidate lossless reference image block is obtained.

[0113] Based on the minimum divergence between the local binary mode (LBP) features of each candidate compressed reference image block and the LBP features of each Mars image block, a target lossless reference image block is determined from the plurality of candidate lossless reference image blocks, and a target compressed reference image block is determined from the plurality of candidate compressed reference image blocks.

[0114] Multiple target lossless reference image blocks are stitched together to form a lossless reference image, and multiple target compressed reference image blocks are stitched together to form a lossy reference image. The lossless reference image and the lossy reference image are then combined to form the reference image pair.

[0115] Optionally, the semantic prior-based compressed Mars image quality enhancement device further includes:

[0116] The sample acquisition module is used to acquire compressed Mars image samples to be processed.

[0117] The image matching module is used to match the compressed Mars image samples to be processed with the semantic matching module to obtain reference image pairs.

[0118] The training module is used to train the Mars image quality enhancement model based on the compressed Mars image samples to be processed and the reference image samples, so as to obtain the trained Mars image quality enhancement model.

[0119] Optionally, the processing module 203 is specifically used for:

[0120] The compressed Mars image to be processed and the reference image are input into a convolutional network to obtain the first compressed Mars image features, the first lossless reference image features, and the first compressed reference image features.

[0121] The first compressed Mars image feature to be processed, the first lossless reference image feature, and the first compressed reference image feature are input into the first texture extraction module to obtain the first transferable feature.

[0122] After sequentially inputting the first compressed Mars image feature to be processed, the first lossless reference image feature, and the first compressed reference image feature into the downsampling layer and the convolutional layer, the second compressed Mars image feature to be processed, the second lossless reference image feature, and the second compressed reference image feature are obtained and then input into the second texture extraction module to obtain the second transferable feature.

[0123] The second compressed Mars image feature to be processed, the second lossless reference image feature, and the second compressed reference image feature are sequentially input into the downsampling layer and the convolutional layer to obtain the third compressed Mars image feature to be processed, the third lossless reference image feature, and the third compressed reference image feature. These are then input into the third texture extraction module to obtain the third transferable feature.

[0124] The third compressed Mars image feature to be processed, the third lossless reference image feature, and the third compressed reference image feature are sequentially input into the downsampling layer, the convolutional layer, and the upsampling layer, and then added to the third compressed Mars image feature to be processed and input into the first texture fusion module. At the same time, the third transferable feature is input into the first texture fusion module to obtain the third fused feature.

[0125] The third fusion feature is upsampled and added to the second compressed Mars image feature to be processed, and then input into the second texture fusion module. At the same time, the second transferable feature is input into the second texture fusion module to obtain the second fusion feature.

[0126] The second fusion feature is upsampled and added to the features of the compressed Mars image to be processed, and then input into the third texture fusion module. At the same time, the first transferable feature is input into the third texture fusion module to obtain the first fusion feature, which is then input into the supervised attention module to obtain the preliminary enhanced image as the enhanced Mars image.

[0127] Optionally, the processing module 203 is further configured to:

[0128] Based on the first fusion feature and input into the supervised attention module, attention-enhanced features are obtained;

[0129] The compressed Mars image to be processed is input into a convolutional layer to obtain the fourth compressed Mars image feature to be processed. After merging the channel dimension with the attention enhancement feature, it is input into the first quality enhancement module to obtain the first encoding feature.

[0130] The first compressed Mars image features to be processed and the first fused features are respectively input into a convolutional layer, added together, and then added to the first encoded features. After downsampling, they are input into the second quality enhancement module to obtain the second encoded features.

[0131] The second compressed Mars image features to be processed and the second fused features are respectively input into a convolutional layer, added together, and then added to the second encoded features. After downsampling, they are input into the third quality enhancement module to obtain the third encoded features.

[0132] The third compressed Mars image feature to be processed and the third fusion feature are respectively input into the convolutional layer, added together, and then added to the second encoded feature. After downsampling, they are input into the nonlocal module and then upsampled to obtain the fourth encoded feature.

[0133] The fourth coding feature is added to the third coding feature and then input into the fourth quality enhancement module and the upsampling layer to obtain the fifth coding feature;

[0134] The fifth coding feature is added to the second coding feature and then input into the fifth quality enhancement module and the upsampling layer to obtain the sixth coding feature;

[0135] The sixth encoded feature is added to the first encoded feature and then input into the sixth quality enhancement module and the upper convolutional layer to obtain the final enhanced image as the enhanced Mars image.

[0136] Optionally, the features of the compressed Mars image to be processed, the features of the lossless reference image, and the features of the compressed reference image are input into the texture extraction module to obtain transferable features, including:

[0137] The features of the compressed Mars image to be processed, the features of the lossless reference image, and the features of the compressed reference image are expanded according to a preset size to obtain the feature blocks of the compressed Mars image to be processed, the feature blocks of the lossless reference image, and the feature blocks of the compressed reference image.

[0138] Based on a preset mathematical formula, the feature blocks of the compressed Mars image to be processed, the feature blocks of the lossless reference image, and the feature blocks of the compressed reference image are calculated to obtain transferable feature blocks;

[0139] All the described transferable feature blocks are merged to obtain the transferable feature.

[0140] Optionally, the upsampled fused features are added to the features of the compressed Mars image to be processed as the first input to the texture fusion module. Simultaneously, transferable features are input to the texture fusion module as the second input to obtain the target fused features, including:

[0141] The intermediate features are obtained by processing the second input with a pre-defined convolutional layer and then subtracting the first input.

[0142] After the channel attention layer processes the first input, it performs an element-wise inner product operation with the intermediate features, adds the intermediate features to the first input, and then processes the input through the pre-set size convolutional layer to obtain the target fusion features.

[0143] The semantic prior-based compressed Mars image quality enhancement system provided in this disclosure can execute the semantic prior-based compressed Mars image quality enhancement method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method.

[0144] This disclosure also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the semantic prior-based compressed Mars image quality enhancement method provided in any embodiment of this disclosure.

[0145] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0146] processor;

[0147] Memory used to store the processor's executable instructions;

[0148] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the semantic prior-based compressed Mars image quality enhancement method as provided in any of the present disclosure.

[0149] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program for performing a semantic prior-based compressed Mars image quality enhancement method as described in any of the present disclosure.

[0150] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0151] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0152] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for semantic-prior based compressed Mars image quality enhancement, characterized in that, include: Acquire compressed Mars image samples to be processed; The semantic matching module matches the compressed Mars image samples to be processed to obtain reference image pairs. The Mars image quality enhancement model is trained based on the compressed Mars image samples to be processed and the reference image samples to obtain the trained Mars image quality enhancement model. Acquire compressed Mars images to be processed; The semantic-based matching module matches the compressed Mars image to be processed to obtain reference image pairs; The compressed Mars image to be processed and the reference image are used to process the trained Mars image quality enhancement model to obtain an enhanced Mars image. The step of processing the compressed Mars image to be processed and the reference image into a pre-trained Mars image quality enhancement model to obtain an enhanced Mars image includes: The compressed Mars image to be processed and the reference image are input into a convolutional network to obtain the first compressed Mars image features, the first lossless reference image features, and the first compressed reference image features. The first compressed Mars image feature to be processed, the first lossless reference image feature, and the first compressed reference image feature are input into the first texture extraction module to obtain the first transferable feature. After sequentially inputting the first compressed Mars image feature to be processed, the first lossless reference image feature, and the first compressed reference image feature into the downsampling layer and the convolutional layer, the second compressed Mars image feature to be processed, the second lossless reference image feature, and the second compressed reference image feature are obtained and then input into the second texture extraction module to obtain the second transferable feature. The second compressed Mars image feature to be processed, the second lossless reference image feature, and the second compressed reference image feature are sequentially input into the downsampling layer and the convolutional layer to obtain the third compressed Mars image feature to be processed, the third lossless reference image feature, and the third compressed reference image feature. These are then input into the third texture extraction module to obtain the third transferable feature. The third compressed Mars image feature to be processed, the third lossless reference image feature, and the third compressed reference image feature are sequentially input into the downsampling layer, the convolutional layer, and the upsampling layer, and then added to the third compressed Mars image feature to be processed and input into the first texture fusion module. At the same time, the third transferable feature is input into the first texture fusion module to obtain the third fused feature. The third fusion feature is upsampled and added to the second compressed Mars image feature to be processed, and then input into the second texture fusion module. At the same time, the second transferable feature is input into the second texture fusion module to obtain the second fusion feature. The second fusion feature is upsampled and added to the features of the compressed Mars image to be processed, and then input into the third texture fusion module. At the same time, the first transferable feature is input into the third texture fusion module to obtain the first fusion feature, which is then input into the supervised attention module to obtain the preliminary enhanced image as the enhanced Mars image.

2. The method for enhancing the quality of compressed Mars images based on semantic prior as described in claim 1, characterized in that, The semantic-based matching module matches the compressed Mars image to be processed to obtain reference image pairs, including: The compressed Mars image to be processed is segmented to obtain multiple Mars image blocks; The compressed Mars image to be processed is classified based on a preset Mars image segmentation network, and the classification results of the multiple Mars image blocks are obtained according to their corresponding positions. Based on each classification result, a match is made in the semantic class dictionary to obtain multiple candidate lossless reference image blocks, and a candidate compressed reference image block corresponding to each candidate lossless reference image block is obtained. Based on the minimum divergence between the local binary mode (LBP) features of each candidate compressed reference image block and the LBP features of each Mars image block, a target lossless reference image block is determined from the plurality of candidate lossless reference image blocks, and a target compressed reference image block is determined from the plurality of candidate compressed reference image blocks. Multiple target lossless reference image blocks are stitched together to form a lossless reference image, and multiple target compressed reference image blocks are stitched together to form a lossy reference image. The lossless reference image and the lossy reference image are then combined to form the reference image pair.

3. The method for enhancing the quality of compressed Mars images based on semantic prior as described in claim 1, characterized in that, Also includes: Based on the first fusion feature and input into the supervised attention module, attention-enhanced features are obtained; The compressed Mars image to be processed is input into a convolutional layer to obtain the fourth compressed Mars image feature to be processed. After merging the channel dimension with the attention enhancement feature, it is input into the first quality enhancement module to obtain the first encoding feature. The first compressed Mars image features to be processed and the first fused features are respectively input into a convolutional layer, added together, and then added to the first encoded features. After downsampling, they are input into the second quality enhancement module to obtain the second encoded features. The second compressed Mars image features to be processed and the second fused features are respectively input into a convolutional layer, added together, and then added to the second encoded features. After downsampling, they are input into the third quality enhancement module to obtain the third encoded features. The third compressed Mars image feature to be processed and the third fusion feature are respectively input into the convolutional layer, added together, and then added to the second encoded feature. After downsampling, they are input into the nonlocal module and then upsampled to obtain the fourth encoded feature. The fourth coding feature is added to the third coding feature and then input into the fourth quality enhancement module and the upsampling layer to obtain the fifth coding feature; The fifth coding feature is added to the second coding feature and then input into the fifth quality enhancement module and the upsampling layer to obtain the sixth coding feature; The sixth encoded feature is added to the first encoded feature and then input into the sixth quality enhancement module and the upper convolutional layer to obtain the final enhanced image as the enhanced Mars image.

4. The method for enhancing the quality of compressed Mars images based on semantic priors according to claim 1, characterized in that, The features of the compressed Mars image to be processed, the features of the lossless reference image, and the features of the compressed reference image are input into the texture extraction module to obtain transferable features, including: The features of the compressed Mars image to be processed, the features of the lossless reference image, and the features of the compressed reference image are expanded according to a preset size to obtain the feature blocks of the compressed Mars image to be processed, the feature blocks of the lossless reference image, and the feature blocks of the compressed reference image. Based on a preset mathematical formula, the feature blocks of the compressed Mars image to be processed, the feature blocks of the lossless reference image, and the feature blocks of the compressed reference image are calculated to obtain transferable feature blocks; All the described transferable feature blocks are merged to obtain the transferable feature.

5. The method for enhancing the quality of compressed Mars images based on semantic priors according to claim 1, characterized in that, After upsampling, the fused features are added to the features of the compressed Mars image to be processed, serving as the first input to the texture fusion module. Simultaneously, transferable features are input to the texture fusion module as the second input to obtain the target fused features, including: The intermediate features are obtained by processing the second input with a pre-defined convolutional layer and then subtracting the first input. After the channel attention layer processes the first input, it performs an element-wise inner product operation with the intermediate features, adds the intermediate features to the first input, and then processes the input through the pre-set size convolutional layer to obtain the target fusion features.

6. A device for enhancing the quality of compressed Mars images based on semantic priors, characterized in that, include: The sample acquisition module is used to acquire compressed Mars image samples to be processed. The image matching module is used to match the compressed Mars image samples to be processed with the semantic matching module to obtain reference image pairs. The training module is used to train the Mars image quality enhancement model based on the compressed Mars image samples to be processed and the reference image samples, so as to obtain the trained Mars image quality enhancement model. The acquisition module is used to acquire compressed Mars images to be processed; The matching module is used to match the compressed Mars image to be processed based on the semantic matching module to obtain reference image pairs; The processing module is used to process the compressed Mars image to be processed and the reference image into the pre-trained Mars image quality enhancement model to obtain an enhanced Mars image; The processing module is specifically used for: The compressed Mars image to be processed and the reference image are input into a convolutional network to obtain the first compressed Mars image features, the first lossless reference image features, and the first compressed reference image features. The first compressed Mars image feature to be processed, the first lossless reference image feature, and the first compressed reference image feature are input into the first texture extraction module to obtain the first transferable feature. After sequentially inputting the first compressed Mars image feature to be processed, the first lossless reference image feature, and the first compressed reference image feature into the downsampling layer and the convolutional layer, the second compressed Mars image feature to be processed, the second lossless reference image feature, and the second compressed reference image feature are obtained and then input into the second texture extraction module to obtain the second transferable feature. The second compressed Mars image feature to be processed, the second lossless reference image feature, and the second compressed reference image feature are sequentially input into the downsampling layer and the convolutional layer to obtain the third compressed Mars image feature to be processed, the third lossless reference image feature, and the third compressed reference image feature. These are then input into the third texture extraction module to obtain the third transferable feature. The third compressed Mars image feature to be processed, the third lossless reference image feature, and the third compressed reference image feature are sequentially input into the downsampling layer, the convolutional layer, and the upsampling layer, and then added to the third compressed Mars image feature to be processed and input into the first texture fusion module. At the same time, the third transferable feature is input into the first texture fusion module to obtain the third fused feature. The third fusion feature is upsampled and added to the second compressed Mars image feature to be processed, and then input into the second texture fusion module. At the same time, the second transferable feature is input into the second texture fusion module to obtain the second fusion feature. The second fusion feature is upsampled and added to the features of the compressed Mars image to be processed, and then input into the third texture fusion module. At the same time, the first transferable feature is input into the third texture fusion module to obtain the first fusion feature, which is then input into the supervised attention module to obtain the preliminary enhanced image as the enhanced Mars image.

7. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the semantic prior-based compressed Mars image quality enhancement method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the semantic prior-based compressed Mars image quality enhancement method as described in any one of claims 1-5.

Images