High-resolution remote sensing image color correction method and system based on style transfer

Abstract

This invention discloses a method and system for color correction of high-resolution remote sensing images based on style transfer. The method includes: downsampling a high-resolution remote sensing image using a deterministic partitioning sampling approach to obtain multiple low-resolution images; constructing and training a multi-scale style transfer network to perform color correction on the low-resolution images; synthesizing the corrected low-resolution images into a color-corrected high-resolution image through the inverse operation of deterministic partitioning sampling; and applying a color consistency voting strategy to the synthesized image to further improve the color correction effect and enhance the visual coherence of the image. The deterministic partitioning sampling method proposed in this invention can effectively reduce the hardware burden in the high-resolution image processing process and ensure structural integrity; the color consistency voting strategy can improve the overall color consistency and naturalness of the corrected image. This invention improves upon the problems of color inhomogeneity and structural distortion existing in traditional methods.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing, and particularly relates to a technique for color correction of high-resolution remote sensing images using style transfer, which is mainly used in the fields of style transfer and color correction of high-resolution remote sensing images. Background Technology

[0002] Remote sensing imagery is widely used in various fields such as land resource surveys, environmental monitoring, urban planning, and agricultural assessment. However, due to the influence of various factors such as imaging sensor type, imaging time, meteorological conditions, and atmospheric interference, images acquired at different times or on different platforms exhibit significant differences in color, brightness, and contrast, severely impacting the comprehensive utilization and analytical accuracy of multi-temporal and multi-source remote sensing data. Therefore, effectively performing color correction on remote sensing images to maintain consistency in their visual and statistical characteristics is one of the key issues in remote sensing image preprocessing.

[0003] Traditional color correction methods mainly include histogram matching, spectral adjustment, and atmospheric correction based on physical models. These methods often rely on certain assumptions, such as consistent lighting conditions and stable surface reflectance characteristics, making them difficult to adapt to complex and ever-changing real-world remote sensing scenarios. Furthermore, when processing remote sensing images from different sensors or with significant stylistic differences, these methods often struggle to preserve the structural information and realism of the images.

[0004] In recent years, style transfer, as a cutting-edge technology in computer vision, can effectively transfer stylistic features such as color and texture from one image to another while preserving the original structural information, demonstrating good versatility and expressive power. This technology has achieved significant results in areas such as image style transfer and augmented reality. Introducing style transfer methods into the color correction task of remote sensing imagery is expected to overcome the limitations of traditional methods based on statistical matching or physical modeling, achieving more natural and consistent color correction effects, and improving the comparability and fusion analysis capabilities of multi-temporal and multi-source remote sensing images.

[0005] However, the direct application of style transfer technology in remote sensing imagery still faces two key challenges: First, remote sensing images typically have extremely high spatial resolution, and directly performing style transfer processing on the entire image will be limited by hardware resources such as video memory and system memory. While image segmentation can alleviate the hardware burden, it easily produces obvious color discontinuities or seam artifacts in the stitching areas, affecting the overall correction effect. Second, existing style transfer methods mostly focus on artistic style transfer, emphasizing the reproduction of color and texture styles, often neglecting the preservation of image structural information. Directly applying these methods to remote sensing images can easily cause problems such as blurred ground feature boundaries and distorted spatial relationships, which is detrimental to subsequent remote sensing applications such as ground feature identification and change detection. Therefore, there is an urgent need for an efficient color correction method that balances style feature preservation and structural information integrity, and is suitable for high-resolution remote sensing images, to better meet the practical needs of remote sensing image analysis and processing. Summary of the Invention

[0006] This invention aims to address the problems of high hardware resource consumption and easy loss of image structural information in the color correction process of high-resolution remote sensing images in existing technologies. It proposes a style transfer-based method for color correction of remote sensing images. This method is applicable to remote sensing images of any resolution, effectively reducing computational resource overhead while maintaining the integrity of the image structure and achieving natural consistency correction of color features.

[0007] Specifically, this invention designs a deterministic partitioning sampling mechanism to regularly divide high-resolution remote sensing images according to a set scale, and extracts pixels based on fixed offset positions in each partitioned region to generate multiple low-resolution sub-images. This reduces the overall processing resolution and alleviates hardware processing pressure while maintaining full image coverage information. Subsequently, a multi-resolution style transfer network structure is proposed, which achieves high-fidelity style transfer and color correction of remote sensing images by fusing color and structural information from different sampling results. Finally, a color consistency voting strategy is designed to ensure the visual consistency of the final color-corrected remote sensing image. Compared with existing methods, this invention can handle the color correction problem of images at any resolution while preserving good structural information. Therefore, this invention has significant practical value and broad application prospects in remote sensing image processing.

[0008] This invention provides a high-resolution remote sensing image color correction method based on style transfer, the method comprising the following steps:

[0009] Step 1: Perform deterministic partitioning sampling on the high-resolution remote sensing image to obtain multiple downsampled low-resolution remote sensing images;

[0010] Step 2: Construct a multi-scale style transfer network for color correction of low-resolution images and train it to converge.

[0011] Step 3: Use a multi-scale style transfer network to perform color correction on the low-resolution image, and obtain a color-corrected high-resolution image through the inverse operation of deterministic partition sampling.

[0012] Step 4: Perform color consistency voting on the color-corrected high-resolution image to further enhance the color correction effect and make its visual effect more consistent.

[0013] Furthermore, the specific implementation method of step 1 is as follows:

[0014] Step 1.1: Determine the resolution of the downsampled low-resolution remote sensing image, and denote the input remote sensing image as having a size of [size missing]. Image Set the downsampling factor to be Divide the image into Non-overlapping image patches, where Represents positive integers;

[0015] Step 1.2, for each image patch, from a fixed offset position Sample a pixel value; this pixel is located at the [number]th [number]th [pixel] in the image patch. Okay, number List;

[0016] Step 1.3: Concatenate all selected pixels in their original spatial order to form a structure with a resolution of [resolution value missing]. A low-resolution sub-image;

[0017] Step 1.4, repeat steps 1.2 to 1.3, to obtain a total of Each low-resolution sub-image corresponds to a fixed offset position combination.

[0018] Furthermore, the multi-scale style transfer network in step 2 consists of three parts: a feature extraction module, a style mapping module, and a style transfer decoder.

[0019] Furthermore, the feature extraction module is a VGG19 network, used to extract depth features at different scales from the input image.

[0020] Furthermore, the style mapping module's processing procedure is as follows:

[0021] First, the depth features at different scales extracted by the feature extraction module are analyzed. and Standardization process is required to obtain and ,in , It refers to the number of layers with features at different scales; then for Calculate its corresponding Gram matrix. The style mapping module is specifically defined as follows:

[0022]

[0023] in, and These are learnable matrix parameters used to adjust the feature size transformation when mapping style features to structural features. That is The mean.

[0024] Furthermore, the style transfer decoder consists of N upsampling modules. Each upsampling module comprises two convolutions, two activation functions, and one upsampling operation, in the order of convolution, activation, upsampling, convolution, activation. The input to the upsampling module is the feature layer at each scale. The first upsampling module's input is only the concatenation of the output from the previous upsampling module. The final layer undergoes a convolution operation, and the output of the style transfer decoder is related to the structure diagram. Stylized result image of uniform size That is, the input structure diagram According to the target color style map The result of color correction.

[0025] Furthermore, the process of training a multi-scale style transfer network is as follows:

[0026] First, the existing dataset is used as the structure map and style map to initialize the multi-scale style transfer network. Then, remote sensing imagery is used as the structure map and style map to optimize the multi-scale style transfer network. Each training process adopts a two-step training strategy: first, two different images are used as the structure map. and style diagram The input is fed into a multi-scale style transfer network to obtain a stylized image. Computational stylization of images and Structural loss Stylized images and Style loss ; then use Simultaneously, the structure map and style map are input into a multi-scale style transfer network to obtain a stylized image. Computational stylization of images and Structural loss Stylized images and Style loss and stylized images and L1 regularization loss The total loss is Structural loss and Style loss and L1 regularization loss The definition is as follows:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] in , , , These represent the extraction of stylized images using the VGG19 network. Stylized images Style diagram and structural diagram Corresponding different scale layers depth features, The number of layers representing features at different scales. and This indicates the calculation of the mean and variance.

[0033] Furthermore, the specific steps in step 3 are as follows:

[0034] Step 3.1: Use the trained multi-scale style transfer network to process the data obtained in Step 1. Style transfer is performed on low-resolution sub-images to achieve color correction of a single ground-resolution image. The input to the multi-scale style transfer network is... The low-resolution sub-image is used as the structure map, the target color remote sensing image is used as the style map, and the output is color correction. A low-resolution sub-image;

[0035] Step 3.2, for the color-corrected The inverse operation of deterministic partitioning sampling is performed on the low-resolution subimage, based on its corresponding offset index. According to the spatial mapping rules, the images are repositioned to their corresponding positions to form a complete high-resolution image. .

[0036] Furthermore, the principle of the color consistency voting strategy in step 4 is as follows: pixels with the same color before style transfer should maintain the same color after style transfer; the color that appears most frequently is selected as the color after style transfer. The specific steps of the high-resolution image color consistency voting strategy are as follows:

[0037] Step 4.1, Construct color consistency groups: Scan the original image and construct color consistency groups using pixel color as the key value; that is, for pixels with the same color value in the original image, establish a set of corresponding pixel locations. The set is denoted as:

[0038]

[0039] in Indicates color value, Represents the original high-resolution image The position of the middle pixel;

[0040] Step 4.2, perform color voting on the style-transferred images: for each group Pixels in the image are extracted and their values ​​are reflected in the style-transferred image. The color results in the image; statistical analysis of this group of pixels in the image. From the color histogram in the image, the color that appears most frequently is selected as the candidate uniform color for that group. ;

[0041] Step 4.3, Color Replacement and Blending: This unifies the candidate color value... Replace Image The middle belongs to the set of pixel positions The color of the pixel, or its current color, is used to enhance color consistency in the color migration result;

[0042] Step 4.4, High-resolution remote sensing image color correction: Repeat steps 4.1 to 4.3 until the image... The final color-corrected image is obtained by fully correcting all color values. .

[0043] The present invention also provides a high-resolution remote sensing image color correction system based on style transfer, including a memory and a processor communicatively connected to the memory. The memory stores computer program instructions, and when the processor executes the computer program instructions, it implements a high-resolution remote sensing image color correction method based on style transfer as described above.

[0044] The beneficial effects of this invention are:

[0045] 1) The multi-scale style transfer network constructed in this invention can achieve color correction of remote sensing images while preserving good structural integrity.

[0046] 2) The deterministic partitioning sampling mechanism proposed in this invention can perform style transfer on remote sensing images by region to achieve high-resolution color correction.

[0047] The color consistency voting strategy proposed in this invention can further enhance the visual effect after color correction. Attached Figure Description

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0049] Figure 1 This is a general flowchart of an embodiment of the present invention;

[0050] Figure 2 The multi-scale style transfer network framework constructed for this invention is shown in the diagram.

[0051] Figure 3 This is a schematic diagram of the multi-scale style transfer network training strategy of the present invention;

[0052] Figure 4 This is a qualitative comparison result of the color correction of remote sensing images in this invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] like Figure 1 As shown, the high-resolution remote sensing image color correction method based on style transfer provided in this embodiment of the invention includes the following steps:

[0055] Step 1: Perform deterministic partitioning sampling on the high-resolution remote sensing image to obtain multiple downsampled low-resolution remote sensing images;

[0056] Step 2: Construct a multi-scale style transfer network for color correction of low-resolution images and train it to converge.

[0057] Step 3: Use a multi-scale style transfer network to perform color correction on the low-resolution image, and obtain a color-corrected high-resolution image through the inverse operation of deterministic partition sampling.

[0058] Step 4: Perform color consistency voting on the color-corrected high-resolution image to further enhance the color correction effect and make its visual effect more consistent.

[0059] Furthermore, the specific implementation method of step 1 is as follows:

[0060] Step 1.1: Determine the resolution of the downsampled low-resolution remote sensing image, and denote the input remote sensing image as having a size of [size missing]. Image Set the downsampling factor to be Divide the image into Non-overlapping image patches, where Represents positive integers;

[0061] Step 1.2, for each image patch, from a fixed offset position Sample a pixel value; this pixel is located at the [number]th [number]th [pixel] in the image patch. Okay, number Column, for example, the top left corner is The bottom right corner is ;

[0062] Step 1.3: Concatenate all selected pixels in their original spatial order to form a structure with a resolution of [resolution value missing]. A low-resolution sub-image;

[0063] Step 1.4, repeat steps 1.2 to 1.3, to obtain a total of Each low-resolution sub-image corresponds to a fixed offset position combination.

[0064] Furthermore, the multi-scale style transfer network in step 2 consists of three parts: feature extraction, style mapping module, and style transfer decoder. The specific implementation method for network construction and training is as follows:

[0065] Step 2.1: Extract style maps using the VGG19 network. and structural diagram Corresponding depth features at different scales and ,in , It represents the number of layers with features at different scales, and the structure diagram. The input image requires color correction; style map It is the target image that references the color style;

[0066] Step 2.2: Construct the style mapping module, with the style map features extracted in Step 2.1 as input. and structural features The output is the stylized features. First of all and Standardization process is required to obtain and Then to Calculate its corresponding Gram matrix. The style mapping module is specifically defined as follows:

[0067]

[0068] in, and These are learnable matrix parameters used to adjust the feature size transformation when mapping style features to structural features. That is The mean.

[0069] Step 2.3: Construct the style transfer decoder. The decoder consists of N upsampling modules. Each upsampling module comprises two convolutions, two activation functions, and one upsampling operation, in the order of convolution, activation, upsampling, convolution, activation. The input to the upsampling module is the feature layer at each scale. corresponding The first upsampling module's input is only the concatenation of the output from the previous upsampling module. The final layer undergoes a convolution operation before outputting. The output of the style transfer decoder is then compared with the structure diagram. Stylized result image of uniform size That is, the input structure diagram According to the target color style map The result of color correction.

[0070] Step 2.4: Train the multi-scale style transfer network. First, initialize the multi-scale style transfer network using the COCO dataset as structure and style maps. Then, optimize the multi-scale style transfer network using remote sensing imagery as structure and style maps. Each training process employs a two-step training strategy: first, use two different images as structure maps... and style diagram The input is fed into a multi-scale style transfer network to obtain a stylized image. Computational stylization of images and Structural loss Stylized images and Style loss ; then use Simultaneously, the structure map and style map are input into a multi-scale style transfer network to obtain a stylized image. Computational stylization of images and Structural loss Stylized images and Style loss and stylized images and L1 regularization loss The total loss is Structural loss and Style loss and L1 regularization loss The definition is as follows:

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] in , , , These represent the extraction of stylized images using the VGG19 network. Stylized images Style diagram and structural diagram Corresponding different scale layers depth features, The number of layers representing features at different scales. and This indicates the calculation of the mean and variance.

[0077] Furthermore, the specific steps in step 3 are as follows:

[0078] Step 3.1: Use the trained multi-scale style transfer network to process the data obtained in Step 1. Style transfer is performed on low-resolution sub-images to achieve color correction of a single ground-resolution image. The input to the multi-scale style transfer network is... The low-resolution sub-image is used as the structure map, the target color remote sensing image is used as the style map, and the output is color correction. A low-resolution sub-image.

[0079] Step 3.2, for the color-corrected The inverse operation of deterministic partitioning sampling is performed on the low-resolution subimage, based on its corresponding offset index. According to the spatial mapping rules, the images are repositioned to their corresponding positions to form a complete high-resolution image. .

[0080] Furthermore, the principle of the color consistency voting strategy in step 4 is as follows: pixels with the same color before style transfer should maintain the same color after style transfer; the color that appears most frequently is selected as the color after style transfer. The specific steps of the high-resolution image color consistency voting strategy are as follows:

[0081] Step 4.1: Construct color consistency groups. Scan the original image and construct color consistency groups using pixel color as the key. That is, for pixels with the same color value in the original image, establish a set of corresponding pixel locations. The set is denoted as:

[0082]

[0083] in Indicates color value, Represents the original high-resolution image The position of the middle pixel.

[0084] Step 4.2: Perform color voting on the style-transferred images. For each group... Pixels in the image are extracted and their values ​​are reflected in the style-transferred image. The color results in the image; statistical analysis of this group of pixels in the image. From the color histogram in the image, the color that appears most frequently is selected as the candidate uniform color for that group. .

[0085] Step 4.3, Color Replacement and Blending. This involves unifying the candidate color value. Replace Image The middle belongs to the set of pixel positions The color of the pixel is used, or it is blended with its current color (such as by weighted averaging), thereby enhancing the color consistency in the color migration result.

[0086] Step 4.4, High-resolution remote sensing image color correction. Repeat steps 4.1 to 4.3 until the image... The final color-corrected image is obtained by fully correcting all color values. .

[0087] Figure 4 This presents a qualitative comparison of the color correction results for remote sensing images according to the present invention. Figure 4 In the image, (a) represents the original image without color correction. Figure 4 (b) in the text represents the classic style transfer method WCT2. [1] The results of color correction for remote sensing images, Figure 4 In the figure, (c) represents the result of color correction of remote sensing images by the method of the present invention.

[0088] [1] Yoo, J., Uh, Y., Chun, S., Kang, B. and Ha, JW, 2019. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE / CVF international conference on computer vision (pp. 9036-9045).

[0089] Table 1 shows the quantitative comparison results of the color correction of remote sensing images by the present invention, where the larger the SSIM and PSNR values, the better the effect.

[0090] Table 1. Quantitative Comparison Results of Color Correction for Remote Sensing Images Based on the Invention

[0091]

[0092] On the other hand, embodiments of the present invention also provide a high-resolution remote sensing image color correction system based on style transfer, including a memory and a processor communicatively connected to the memory. The memory stores computer program instructions, and when the processor executes the computer program instructions, it implements a high-resolution remote sensing image color correction method based on style transfer as described in the above technical solution.

[0093] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the invention.

Claims

1. A color correction method for high-resolution remote sensing images based on style transfer, characterized in that, Includes the following steps: Step 1: Perform deterministic partitioning sampling on the high-resolution remote sensing image to obtain multiple downsampled low-resolution remote sensing images; The specific implementation method of step 1 is as follows: Step 1.1: Determine the resolution of the downsampled low-resolution remote sensing image, and denote the input remote sensing image as having a size of [size missing]. Image Set the downsampling factor to be Divide the image into Non-overlapping image patches, where Represents positive integers; Step 1.2, for each image patch, from a fixed offset position Sample a pixel value; this pixel is located at the [number]th [number]th [pixel] in the image patch. Okay, number List; Step 1.3: Concatenate all selected pixels in their original spatial order to form a structure with a resolution of [resolution value missing]. A low-resolution sub-image; Step 1.4, repeat steps 1.2 to 1.3, to obtain a total of Each low-resolution sub-image corresponds to a fixed offset position combination; Step 2: Construct a multi-scale style transfer network for color correction of low-resolution images and train it to converge. Step 3: Use a multi-scale style transfer network to perform color correction on the low-resolution image, and obtain a color-corrected high-resolution image through the inverse operation of deterministic partition sampling. The specific steps in step 3 are as follows: Step 3.1: Use the trained multi-scale style transfer network to process the data obtained in Step 1. Style transfer is performed on low-resolution sub-images to achieve color correction of a single ground-resolution image. The input to the multi-scale style transfer network is... The low-resolution sub-image is used as the structure map, the target color remote sensing image is used as the style map, and the output is color correction. A low-resolution sub-image; Step 3.2, for the color-corrected The inverse operation of deterministic partitioning sampling is performed on the low-resolution subimage, based on its corresponding offset index. According to the spatial mapping rules, the images are repositioned to their corresponding positions to form a complete high-resolution image. ; Step 4: Perform color consistency voting on the color-corrected high-resolution image to further enhance the color correction effect and make its visual effect more consistent.

2. The high-resolution remote sensing image color correction method based on style transfer as described in claim 1, characterized in that: The multi-scale style transfer network in step 2 consists of three parts: a feature extraction module, a style mapping module, and a style transfer decoder.

3. The high-resolution remote sensing image color correction method based on style transfer as described in claim 2, characterized in that: The feature extraction module is a VGG19 network, used to extract depth features at different scales from the input image.

4. The high-resolution remote sensing image color correction method based on style transfer as described in claim 2, characterized in that: The style mapping module processes the process as follows: First, the depth features at different scales extracted by the feature extraction module are analyzed. and Standardization process is required to obtain and ,in , It refers to the number of layers with features at different scales; then for Calculate its corresponding Gram matrix. The style mapping module is specifically defined as follows: in, and These are learnable matrix parameters used to adjust the feature size transformation when mapping style features to structural features. That is The mean.

5. The high-resolution remote sensing image color correction method based on style transfer as described in claim 4, characterized in that: The style transfer decoder consists of N upsampling modules. Each upsampling module comprises two convolutions, two activation functions, and one upsampling operation, in the order of convolution, activation, upsampling, convolution, activation. The input to each upsampling module is a feature layer at each scale. The first upsampling module's input is only the concatenation of the output from the previous upsampling module. The final layer undergoes a convolution operation, and the output of the style transfer decoder is related to the structure diagram. Stylized result image of uniform size That is, the input structure diagram According to the target color style map The result of color correction.

6. The high-resolution remote sensing image color correction method based on style transfer as described in claim 1, characterized in that: The process of training a multi-scale style transfer network is as follows: First, the existing dataset is used as the structure map and style map to initialize the multi-scale style transfer network. Then, remote sensing imagery is used as the structure map and style map to optimize the multi-scale style transfer network. Each training process adopts a two-step training strategy: first, two different images are used as the structure map. and style diagram The input is fed into a multi-scale style transfer network to obtain a stylized image. Computational stylization of images and Structural loss Stylized images and Style loss ; then use Simultaneously, the structure map and style map are input into a multi-scale style transfer network to obtain a stylized image. Computational stylization of images and Structural loss Stylized images and Style loss and stylized images and L1 regularization loss The total loss is Structural loss and Style loss and L1 regularization loss The definition is as follows: in , , , These represent the extraction of stylized images using the VGG19 network. Stylized images Style diagram and structural diagram Corresponding different scale layers depth features, The number of layers representing features at different scales. and This indicates the calculation of the mean and variance.

7. The high-resolution remote sensing image color correction method based on style transfer as described in claim 1, characterized in that: The principle of the color consistency voting strategy in step 4 is as follows: pixels with the same color before style transfer should maintain the same color after style transfer; the color that appears most frequently is selected as the color after style transfer. The specific steps of the high-resolution image color consistency voting strategy are as follows: Step 4.1, Construct color consistency groups: Scan the original image and construct color consistency groups using pixel color as the key; that is, for pixels with the same color value in the original image, establish a set of corresponding pixel locations. The set is denoted as: in Indicates color value, Represents the original high-resolution image The position of the middle pixel; Step 4.2, perform color voting on the style-transferred images: for each group Pixels in the image are extracted and their values ​​are reflected in the style-transferred image. The color results in the image; statistical analysis of this group of pixels in the image. From the color histogram in the image, the color that appears most frequently is selected as the candidate uniform color for that group. ; Step 4.3, Color Replacement and Blending: This unifies the candidate color value... Replace Image The middle belongs to the set of pixel positions The color of the pixel, or its current color, is used to enhance color consistency in the color migration result; Step 4.4, High-resolution remote sensing image color correction: Repeat steps 4.1 to 4.3 until the image... The final color-corrected image is obtained by fully correcting all color values. .

8. A high-resolution remote sensing image color correction system based on style transfer, characterized in that: The method includes a memory and a processor communicatively connected to the memory. The memory stores computer program instructions, and the processor executes the computer program instructions to implement a high-resolution remote sensing image color correction method based on style transfer as described in any one of claims 1-7.

Images