High-resolution remote sensing image color correction method and system based on style migration

Abstract

The invention discloses a high-resolution remote sensing image color correction method and system based on style migration, and the method comprises the steps: carrying out the down-sampling of a high-resolution remote sensing image through employing a deterministic partition sampling mode, and obtaining a plurality of low-resolution images; constructing and training a multi-scale style migration network, and performing color correction on the low-resolution image; synthesizing the corrected low-resolution image into a high-resolution image after color correction through inverse operation of deterministic partition sampling; and a color consistency voting strategy is applied to the synthesized image, so that the color correction effect is further improved, and the visual coherence of the image is enhanced. According to the deterministic partition sampling method provided by the invention, the hardware burden in a high-resolution image processing process can be effectively reduced, and the structural integrity is ensured; the color consistency voting strategy can improve the overall color consistency and naturalness of the corrected image. According to the invention, the problems of non-uniform color and structure distortion existing in the traditional method are improved.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing, and in particular relates to a technology for realizing color correction of high-resolution remote sensing images by utilizing style migration, and is mainly used in the fields of style migration, color correction of high-resolution remote sensing images, and the like. Background Art

[0002] Remote sensing imagery is widely used in fields such as land and resources surveys, environmental monitoring, urban planning, and agricultural assessment. However, because remote sensing image acquisition is affected by multiple factors, including sensor type, imaging time, meteorological conditions, and atmospheric interference, images acquired at different times or from different platforms exhibit significant differences in color, brightness, and contrast. This significantly impacts the comprehensive utilization and analytical accuracy of multi-temporal and multi-source remote sensing data. Therefore, effectively color correcting remote sensing images to maintain consistent visual and statistical characteristics is a key issue in remote sensing image preprocessing.

[0003] Traditional color correction methods primarily include histogram matching, spectral adjustment, and atmospheric correction based on physical models. These methods often rely on certain assumptions, such as consistent lighting conditions and stable surface reflectance, making them difficult to adapt to the complex and ever-changing real-world remote sensing scenarios. Furthermore, these methods often struggle to preserve image structure and realism when processing remote sensing imagery from multiple sensors or with significant style differences.

[0004] In recent years, style transfer, a cutting-edge technology in computer vision, has demonstrated excellent versatility and expressiveness by effectively transferring stylistic features such as color and texture from one image to another while preserving the original structural information. This technology has achieved remarkable results in areas such as image style transfer and augmented reality. Introducing style transfer methods into the color correction task of remote sensing imagery is expected to overcome the limitations of traditional methods based on statistical matching or physical modeling, achieving more natural, coordinated, and consistent color correction results, and enhancing the comparability and fusion analysis capabilities of multi-temporal and multi-source remote sensing imagery.

[0005] However, the direct application of style transfer technology to remote sensing imagery still faces the following two key challenges: First, remote sensing images typically have extremely high spatial resolution, and directly performing style transfer on the entire image will face hardware resource limitations such as video memory and main memory. While using image block processing can alleviate the hardware burden, it is prone to produce obvious color discontinuities or seam artifacts in the stitched areas, affecting the overall correction effect. Second, existing style transfer methods mainly focus on artistic style transfer, focusing on the reproduction of color and texture styles while often ignoring the preservation of image structural information. Direct application to remote sensing imagery can easily lead to problems such as blurred object boundaries and distorted spatial relationships, which is not conducive to subsequent remote sensing application tasks such as object recognition and change detection. Therefore, there is an urgent need for an efficient color correction method that balances the preservation of style features and the integrity of structural information and is applicable to high-resolution remote sensing imagery to better meet the practical needs of remote sensing image analysis and processing. Summary of the Invention

[0006] This paper aims to address existing issues with high-resolution remote sensing image color correction, such as high hardware resource consumption and the loss of image structural information. It proposes a remote sensing image color correction method based on style transfer. This method, applicable to remote sensing images of any resolution, effectively reduces computational resource consumption while maintaining the integrity of the image structure and achieving natural, consistent correction of color features.

[0007] Specifically, the present invention designs a deterministic partition sampling mechanism for dividing high-resolution remote sensing images into regular divisions according to a set scale, and extracting pixels based on fixed offset positions in each divided area to generate multiple low-resolution sub-images, thereby reducing the overall processing resolution and alleviating hardware processing pressure while maintaining the full image coverage information. Subsequently, a multi-resolution style transfer network structure is proposed to achieve high-fidelity style transfer and color correction of remote sensing images by fusing color and structural information from different sampling results. Finally, a color consistency voting strategy is designed to ensure the consistency of the visual effects of the final color-corrected remote sensing images. Compared with existing methods, the present invention can handle the color correction problem of images of arbitrary resolution and retain better structural information. Therefore, the present invention has very important practical value and broad application prospects in the fields related to remote sensing image processing.

[0008] The present invention provides a method for color correction of high-resolution remote sensing images based on style migration, which comprises the following steps: Step 1: Deterministically partition the high-resolution remote sensing image to obtain multiple downsampled low-resolution remote sensing images; Step 2: Construct a multi-scale style transfer network for low-resolution image color correction and train it to converge; Step 3: Use the multi-scale style transfer network to perform color correction on the low-resolution image, and obtain the color-corrected high-resolution image through the inverse operation of deterministic partition sampling; Step 4: Perform color consistency voting on the high-resolution image after color correction to further enhance the color correction effect and make the visual effect more consistent.

[0009] Furthermore, the specific implementation method of step 1 is: Step 1.1, determine the resolution of the downsampled low-resolution remote sensing image, and record the input remote sensing image as Image , set the downsampling ratio to , divide the image into non-overlapping image blocks, where represents a positive integer; Step 1.2: For each image block, start from the fixed offset position Sample a pixel value, the pixel is located in the image block Row, No. List; Step 1.3: Concatenate all selected pixels in their original spatial order to form a pixel with a resolution of A low-resolution sub-image of Step 1.4, repeat steps 1.2 to 1.3, and obtain A low-resolution sub-image, one for each fixed offset position combination.

[0010] Furthermore, the multi-scale style transfer network in step 2 includes three parts: a feature extraction module, a style mapping module, and a style transfer decoder.

[0011] Furthermore, the feature extraction module is a VGG19 network, which is used to extract depth features of different scales of the input image.

[0012] Furthermore, the processing of the style mapping module is as follows: First, the depth features of different scales extracted by the feature extraction module are and Standardized processing to obtain and ,in , is the number of layers of features at different scales; then Calculate its corresponding Gram matrix Gram( ), the style mapping module is specifically defined as follows:

[0013] in, and It is a learnable matrix parameter used to adjust the feature size transformation of style features mapped to structural features. is The mean of .

[0014] Furthermore, the style transfer decoder consists of N upsampling modules, each of which consists of 2 convolutions, 2 activation functions, and an upsampling operation. The order is convolution, activation, upsampling, convolution, activation; the input of the upsampling module is each scale feature layer The input of the first upsampling module is only concatenated with the output of the previous upsampling module. , the last layer is output through convolution operation, and the output of the style transfer decoder is the same as the structure diagram Stylized result images of consistent size , that is, the structural diagram of the input According to the target color style map The result of color correction.

[0015] Furthermore, the process of training the multi-scale style transfer network is as follows: First, use the existing dataset as the structure map and style map to initialize the multi-scale style transfer network, and then use the remote sensing image as the structure map and style map to optimize the multi-scale style transfer network; each training process adopts a two-step training strategy: first use two different images as the structure map and style map Input to the multi-scale style transfer network to obtain a stylized image , calculate the stylized image and Structural loss , stylized images and Style loss ; Reuse At the same time, it is used as the structure map and style map to input into the multi-scale style transfer network to obtain the stylized image , calculate the stylized image and Structural loss , stylized images and Style loss , and stylized images and L1 regularization loss , the total loss is ; Structural loss and , style loss and , L1 regularization loss The definition is as follows:

[0016]

[0017]

[0018]

[0019]

[0020] in 、 、 、 Respectively represent the use of VGG19 network to extract stylized images , stylized images , style map and structure diagram Corresponding different scale layers The deep features of The number of layers representing features of different scales, and Calculates the mean and variance.

[0021] Furthermore, the specific steps in step 3 are: Step 3.1: Use the trained multi-scale style transfer network to transfer the style obtained in step 1. The low-resolution sub-image is used for style transfer to achieve single-resolution image color correction. The input of the multi-scale style transfer network is The low-resolution sub-image is used as the structure map, the target color remote sensing image is used as the style map, and the output is the color correction A low-resolution sub-image; Step 3.2, after color correction The low-resolution sub-image performs the inverse operation of deterministic partition sampling, according to its corresponding offset index , repositioned to the corresponding position according to the spatial mapping rules to form a complete high-resolution image .

[0022] Furthermore, the principle of the color consistency voting strategy in step 4 is: pixels with the same color before style transfer should also maintain the same color after style transfer. The color that appears the most times is selected as the color after style transfer. The specific steps of the high-resolution image color consistency voting strategy are as follows: Step 4.1, build color consistency groups: Scan the original image and build color consistency groups using pixel color as the key value; that is, for pixels with the same color value in the original image, establish a corresponding pixel position set , the set is recorded as:

[0023] in Represents the color value, Represents the original high-resolution image The position of the pixel in Step 4.2, perform color voting on the style-transferred images: for each group , extract the pixels in the image after style transfer The color result in the image; statistics of this group of pixels The color histogram in , select the color with the highest frequency as the unified color candidate value of the group ; Step 4.3, color replacement and fusion: the unified color candidate value Replace Image belongs to the set of pixel positions The color of the pixel being transferred, or blended with its current color, thereby enhancing the color consistency in the color migration result; Step 4.4, color correction of high-resolution remote sensing images: Repeat steps 4.1 to 4.3 until the image All color values ​​in the image are fully corrected to obtain the final color-corrected image. .

[0024] The present invention also provides a high-resolution remote sensing image color correction system based on style transfer, comprising a memory and a processor communicatively connected to the memory, wherein the memory stores computer program instructions, and when the processor executes the computer program instructions, it implements a high-resolution remote sensing image color correction method based on style transfer as described in the above technical solution.

[0025] The beneficial effects produced by the present invention are: 1) The multi-scale style transfer network constructed in this paper can achieve color correction of remote sensing images while retaining good structural integrity.

[0026] 2) The deterministic partition sampling mechanism proposed in this paper can perform style transfer on remote sensing images by dividing them into regions, achieving high-resolution color correction.

[0027] The color consistency voting strategy proposed in this invention can further enhance the visual effect after color correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which: Figure 1 is an overall flow chart of an embodiment of the present invention; Figure 2 A diagram of the multi-scale style transfer network framework constructed for the present invention; Figure 3 Schematic diagram of the multi-scale style transfer network training strategy of the present invention; Figure 4 This is the qualitative comparison result of the remote sensing image color correction by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] like Figure 1 As shown, the high-resolution remote sensing image color correction method based on style migration provided by the embodiment of the present invention includes the following steps: Step 1: Deterministically partition the high-resolution remote sensing image to obtain multiple downsampled low-resolution remote sensing images; Step 2: Construct a multi-scale style transfer network for low-resolution image color correction and train it to converge; Step 3: Use the multi-scale style transfer network to perform color correction on the low-resolution image, and obtain the color-corrected high-resolution image through the inverse operation of deterministic partition sampling; Step 4: Perform color consistency voting on the high-resolution image after color correction to further enhance the color correction effect and make the visual effect more consistent.

[0031] Furthermore, the specific implementation method of step 1 is: Step 1.1, determine the resolution of the downsampled low-resolution remote sensing image, and record the input remote sensing image as Image , set the downsampling ratio to , divide the image into non-overlapping image blocks, where represents a positive integer; Step 1.2: For each image block, start from the fixed offset position Sample a pixel value, the pixel is located in the image block Row, No. Column, for example, the upper left corner is , the lower right corner is ; Step 1.3: Concatenate all selected pixels in their original spatial order to form a pixel with a resolution of A low-resolution sub-image of Step 1.4, repeat steps 1.2 to 1.3, and obtain A low-resolution sub-image, one for each fixed offset position combination.

[0032] Furthermore, the multi-scale style transfer network in step 2 consists of three parts: feature extraction, style mapping module, and style transfer decoder. The specific implementation method of network construction and training is as follows: Step 2.1: Extract style map using VGG19 network and structure diagram Corresponding depth features of different scales and ,in , is the number of layers of features at different scales, the structure diagram is the input image that needs color correction, style map is the target image with reference color style; Step 2.2: Build a style mapping module, the input is the style map features extracted in step 2.1 and structural features , the output is the stylized feature , first of all and Standardized processing to obtain and , then Calculate its corresponding Gram matrix Gram( ), the style mapping module is specifically defined as follows:

[0033] in, and It is a learnable matrix parameter used to adjust the feature size transformation of style features mapped to structural features. is The mean of .

[0034] Step 2.3, build the style transfer decoder. The decoder consists of N upsampling modules, each of which consists of 2 convolutions, 2 activation functions, and an upsampling operation. The order is convolution, activation, upsampling, convolution, activation. The input of the upsampling module is the feature layer of each scale. Corresponding The input of the first upsampling module is only concatenated with the output of the previous upsampling module. , the last layer is output through convolution operation. The output of the style transfer decoder is the same as the structure diagram Stylized result images of consistent size , that is, the structural diagram of the input According to the target color style map The result of color correction.

[0035] Step 2.4: Train the multi-scale style transfer network. First, use the COCO dataset as the structure map and style map to initialize the multi-scale style transfer network, and then use the remote sensing image as the structure map and style map to optimize the multi-scale style transfer network. Each training process adopts a two-step training strategy: first, use two different images as the structure map and style map Input to the multi-scale style transfer network to obtain a stylized image , calculate the stylized image and Structural loss , stylized images and Style loss ; Reuse At the same time, it is used as the structure map and style map to input into the multi-scale style transfer network to obtain the stylized image , calculate the stylized image and Structural loss , stylized images and Style loss , and stylized images and L1 regularization loss , the total loss is ; Structural loss and , style loss and , L1 regularization loss The definition is as follows:

[0036]

[0037]

[0038]

[0039]

[0040] in 、 、 、 Respectively represent the use of VGG19 network to extract stylized images , stylized images , style map and structure diagram Corresponding different scale layers The deep features of The number of layers representing features of different scales, and Calculates the mean and variance.

[0041] Furthermore, the specific steps in step 3 are: Step 3.1: Use the trained multi-scale style transfer network to transfer the style obtained in step 1. The input of the multi-scale style transfer network is The low-resolution sub-image is used as the structure map, the target color remote sensing image is used as the style map, and the output is the color correction A low-resolution sub-image.

[0042] Step 3.2, after color correction The low-resolution sub-image performs the inverse operation of deterministic partition sampling, according to its corresponding offset index , repositioned to the corresponding position according to the spatial mapping rules to form a complete high-resolution image .

[0043] Furthermore, the principle of the color consistency voting strategy in step 4 is: pixels with the same color before style transfer should also maintain the same color after style transfer, and the color that appears the most times is selected as the color after style transfer. The specific steps of the high-resolution image color consistency voting strategy are: Step 4.1: Construct color consistency groups. Scan the original image and construct color consistency groups using pixel colors as key values. That is, for pixels with the same color value in the original image, establish a corresponding pixel position set. , the set is recorded as:

[0044] in Represents the color value, Represents the original high-resolution image The position of the pixel in .

[0045] Step 4.2, perform color voting on the style-transferred images. , extract the pixels in the image after style transfer The color result in the image; statistics of this group of pixels The color histogram in , select the color with the highest frequency as the unified color candidate value of the group .

[0046] Step 4.3, color replacement and fusion. Replace Image belongs to the set of pixel positions The color of the pixel being transferred can be changed or merged with its current color (such as weighted average) to enhance the color consistency in the color migration result.

[0047] Step 4.4, color correction of high-resolution remote sensing images. Repeat steps 4.1 to 4.3 until the image All color values ​​in the image are fully corrected to obtain the final color-corrected image. .

[0048] Figure 4 This is the qualitative comparison result of the remote sensing image color correction of the present invention. Figure 4 (a) in the figure represents the original image without color correction. Figure 4 (b) represents the classic style transfer method WCT2 [1] The results of remote sensing image color correction are: Figure 4 (c) in FIG. 5 shows the result of color correction of remote sensing images by the method of the present invention.

[0049] [1] Yoo, J., Uh, Y., Chun, S., Kang, B. and Ha, JW, 2019. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE / CVF international conference on computer vision (pp. 9036-9045). Table 1 shows the quantitative comparison results of the remote sensing image color correction by the present invention, wherein the larger the SSIM and PSNR values ​​are, the better the effect is.

[0050] Table 1 Quantitative comparison results of remote sensing image color correction by the present invention

[0051] On the other hand, an embodiment of the present invention also provides a high-resolution remote sensing image color correction system based on style transfer, including a memory and a processor communicatively connected to the memory, wherein the memory stores computer program instructions, and when the processor executes the computer program instructions, it implements a high-resolution remote sensing image color correction method based on style transfer as described in the above technical solution.

[0052] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent replacements and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention should be included in the scope of protection of the invention.

Claims

1. A high-resolution remote sensing image color correction method based on style transfer, characterized in that: The steps include: Step 1: Deterministically partition the high-resolution remote sensing image to obtain multiple downsampled low-resolution remote sensing images; Step 2: Construct a multi-scale style transfer network for low-resolution image color correction and train it to converge; Step 3: Use the multi-scale style transfer network to perform color correction on the low-resolution image, and obtain the color-corrected high-resolution image through the inverse operation of deterministic partition sampling; Step 4: Perform color consistency voting on the high-resolution image after color correction to further enhance the color correction effect and make the visual effect more consistent.

2. The high-resolution remote sensing image color correction method based on style transfer according to claim 1, characterized in that: The specific implementation method of step 1 is: Step 1.1, determine the resolution of the downsampled low-resolution remote sensing image, and record the input remote sensing image as Image , set the downsampling ratio to , divide the image into non-overlapping image blocks, where represents a positive integer; Step 1.2: For each image block, start from the fixed offset position Sample a pixel value, the pixel is located in the image block Row, No. List; Step 1.3: Concatenate all selected pixels in their original spatial order to form a pixel with a resolution of A low-resolution sub-image of Step 1.4, repeat steps 1.2 to 1.3, and obtain A low-resolution sub-image, one for each fixed offset position combination.

3. The high-resolution remote sensing image color correction method based on style transfer according to claim 1, characterized in that: The multi-scale style transfer network in step 2 consists of three parts: feature extraction module, style mapping module, and style transfer decoder.

4. The high-resolution remote sensing image color correction method based on style transfer according to claim 3, characterized in that: The feature extraction module is a VGG19 network, which is used to extract depth features of different scales of the input image.

5. The high-resolution remote sensing image color correction method based on style transfer according to claim 3, characterized in that: The processing of the style mapping module is as follows: First, the depth features of different scales extracted by the feature extraction module are and Standardized processing to obtain and ,in , is the number of layers of features at different scales; then Calculate its corresponding Gram matrix Gram( ), the style mapping module is specifically defined as follows: in, and It is a learnable matrix parameter used to adjust the feature size transformation of style features mapped to structural features. is The mean of .

6. The high-resolution remote sensing image color correction method based on style transfer according to claim 5, characterized in that: The style transfer decoder consists of N upsampling modules, each of which consists of 2 convolutions, 2 activation functions, and an upsampling operation. The order is convolution, activation, upsampling, convolution, activation; the input of the upsampling module is each scale feature layer The input of the first upsampling module is only concatenated with the output of the previous upsampling module. , the last layer is output through convolution operation, and the output of the style transfer decoder is the same as the structure diagram Stylized result images of consistent size , that is, the structural diagram of the input According to the target color style map The result of color correction.

7. The high-resolution remote sensing image color correction method based on style transfer according to claim 1, characterized in that: The process of training a multi-scale style transfer network is as follows: First, use the existing dataset as the structure map and style map to initialize the multi-scale style transfer network, and then use the remote sensing image as the structure map and style map to optimize the multi-scale style transfer network; each training process adopts a two-step training strategy: first use two different images as the structure map and style map Input to the multi-scale style transfer network to obtain a stylized image , calculate the stylized image and Structural loss , stylized images and Style loss ; Reuse At the same time, it is used as the structure map and style map to input into the multi-scale style transfer network to obtain the stylized image , calculate the stylized image and Structural loss , stylized images and Style loss , and stylized images and L1 regularization loss , the total loss is ; Structural loss and , style loss and , L1 regularization loss The definition is as follows: in 、 、 、 Respectively represent the use of VGG19 network to extract stylized images , stylized images , style map and structure diagram Corresponding different scale layers The deep features of The number of layers representing features of different scales, and Calculates the mean and variance.

8. The high-resolution remote sensing image color correction method based on style transfer according to claim 1, characterized in that: The specific steps in step 3 are: Step 3.1: Use the trained multi-scale style transfer network to transfer the style obtained in step 1. The low-resolution sub-image is used for style transfer to achieve single-resolution image color correction. The input of the multi-scale style transfer network is The low-resolution sub-image is used as the structure map, the target color remote sensing image is used as the style map, and the output is the color correction A low-resolution sub-image; Step 3.2, after color correction The low-resolution sub-image performs the inverse operation of deterministic partition sampling, according to its corresponding offset index , repositioned to the corresponding position according to the spatial mapping rules to form a complete high-resolution image .

9. The high-resolution remote sensing image color correction method based on style transfer according to claim 1, characterized in that: The principle of the color consistency voting strategy in step 4 is: pixels with the same color before style transfer should also maintain the same color after style transfer. The color that appears the most times is selected as the color after style transfer. The specific steps of the color consistency voting strategy for high-resolution images are as follows: Step 4.1, build color consistency groups: Scan the original image and build color consistency groups using pixel color as the key value; that is, for pixels with the same color value in the original image, establish a corresponding pixel position set , the set is recorded as: in Represents the color value, Represents the original high-resolution image The position of the pixel in Step 4.2, perform color voting on the style-transferred images: for each group , extract the pixels in the image after style transfer The color result in the image; statistics of this group of pixels The color histogram in , select the color with the highest frequency as the unified color candidate value of the group ; Step 4.3, color replacement and fusion: the unified color candidate value Replace Image belongs to the set of pixel positions The color of the pixel being transferred, or blended with its current color, thereby enhancing the color consistency in the color migration result; Step 4.4, color correction of high-resolution remote sensing images: Repeat steps 4.1 to 4.3 until the image All color values ​​in the image are fully corrected to obtain the final color-corrected image. .

10. A high-resolution remote sensing image color correction system based on style transfer, characterized by: The invention comprises a memory and a processor in communication with the memory, wherein the memory stores computer program instructions, and when the processor executes the computer program instructions, a high-resolution remote sensing image color correction method based on style migration according to any one of claims 1 to 9 is implemented.

Images