Method and device for automatic color uniformity of remote sensing images based on conditional adversarial neural network

Abstract

The present invention provides a method and device for automatic color equalization of remote sensing images based on a conditional adversarial neural network, which relates to the technical field of image processing, and includes the following steps: obtaining a remote sensing image to be color equalized and a color template image, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color equalized; based on geographic coordinates, cropping the remote sensing image to be color equalized and the color template image according to preset sizes to obtain a remote sensing image block to be color equalized and a color template image block; inputting the remote sensing image block to be color equalized and the color template image block into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color equalized, wherein the target remote sensing image block is a remote sensing image block to be color equalized that has been completed; and splicing the target remote sensing image blocks to obtain a remote sensing image that has been completed. The method solves the technical problems of uneven radiation accuracy within the image or oversaturation of brighter areas in existing automatic color equalization technologies for remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for automatic color uniformity of remote sensing images based on a conditional adversarial neural network. Background Art

[0002] Existing automatic image color grading algorithms are all based on template images or standard color libraries. By statistically analyzing the mean, variance, histogram, and other information of the template image and the image to be grading, and performing mathematical model assimilation processing, the color of the image to be grading can be closer to the color of the template image after processing. However, existing conventional color grading methods mostly use linear models, which are suitable for images with relatively uniform internal radiation. For images with uneven internal radiation, traditional automatic color grading methods usually do not achieve ideal results. Secondly, some ground objects in remote sensing images, such as white roofs, have certain texture information. After statistical calculation and use of the color grading coefficient, traditional color grading methods will cause the highlighted areas to be oversaturated, losing the original texture information of the remote sensing image and are irreversible.

[0003] For the above problems, no effective solutions have been proposed yet. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method and device for automatic color equalization of remote sensing images based on conditional adversarial neural network, so as to alleviate the technical problems of existing automatic color equalization technology of remote sensing images, such as uneven radiation accuracy within the image or oversaturation of brighter areas.

[0005] In a first aspect, an embodiment of the present invention provides a method for automatic color equalization of remote sensing images based on a conditional adversarial neural network, comprising: obtaining a remote sensing image to be color equalized and a color template image, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color equalized; based on geographic coordinates, cropping the remote sensing image to be color equalized and the color template image according to preset sizes to obtain a remote sensing image block to be color equalized and a color template image block; inputting the remote sensing image block to be color equalized and the color template image block into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color equalized, wherein the target remote sensing image block is a remote sensing image block to be color equalized that has been completed; and splicing the target remote sensing image blocks to obtain a remote sensing image with completed color equalization.

[0006] Furthermore, before the remote sensing image to be uniformly colored and the color template image are cropped according to preset sizes based on the geographic coordinates to obtain the remote sensing image block to be uniformly colored and the color template image block, the method also includes: determining whether there is a deviation between the geographic coordinates of the remote sensing image to be uniformly colored and the geographic coordinates of the color template image; if so, performing image set registration processing on the remote sensing image to be uniformly colored and the color template image.

[0007] Furthermore, the conditional adversarial neural network model is a conditional adversarial neural network model constructed based on Pix2PixHD.

[0008] Furthermore, the conditional adversarial neural network model includes: a first generator, a second generator, a first discriminator and a second discriminator, and the remote sensing image block to be uniformed and the color template image block are input into the conditional adversarial neural network model to obtain the target remote sensing image block corresponding to the remote sensing image block to be uniformed, including: inputting the initial remote sensing image block to be uniformed into the first generator to obtain an initial remote sensing image block, wherein the initial remote sensing image block to be uniformed is any one of the remote sensing image blocks to be uniformed, the initial remote sensing image block is the initial remote sensing image block to be uniformed that has completed uniformity and the resolution of the initial remote sensing image block is less than the resolution of the initial remote sensing image block to be uniformed; inputting the initial remote sensing image block to be uniformed and the initial remote sensing image block into the first discriminator to obtain a first prediction probability value, wherein the first prediction probability value is used to characterize the initial remote sensing image block to be uniformed. Whether the remote sensing image block to be uniformed and the initial remote sensing image block are a pair of real images; if the first prediction probability value is greater than a first preset threshold, the initial remote sensing image block to be uniformed and the initial remote sensing image block are input into the second generator to obtain an intermediate remote sensing image block, wherein the resolution of the intermediate remote sensing image block is equal to the resolution of the initial remote sensing image block to be uniformed; the intermediate remote sensing image block and the initial color template image block are input into the second discriminator to obtain a second prediction probability value, wherein the initial color template image block is a color template image block with the same geographic coordinates as the initial remote sensing image block to be uniformed, and the second prediction probability value is used to characterize whether the intermediate remote sensing image block and the initial color template image block are a pair of real images; if the second prediction probability value is greater than a second preset threshold, the intermediate remote sensing image block is determined as the target remote sensing image block.

[0009] Furthermore, before splicing the target remote sensing image block to obtain a remote sensing image with uniform color, the method also includes: calculating target parameters between the target remote sensing image block and the color template image block corresponding to the target remote sensing image block, wherein the target parameters include: the mean, variance and standard deviation of each band, and the peak signal-to-noise ratio; based on the target parameters, determining whether to perform image color correction on the target remote sensing image block.

[0010] In a second aspect, an embodiment of the present invention further provides an automatic color equalization device for remote sensing images based on a conditional adversarial neural network, comprising: an acquisition unit, a cropping unit, a color equalization unit and a splicing unit, wherein the acquisition unit is used to acquire the remote sensing image to be color equalized and the color template image, wherein the area range corresponding to the color template image includes the area range corresponding to the remote sensing image to be color equalized; the cropping unit is used to crop the remote sensing image to be color equalized and the color template image according to preset sizes based on geographic coordinates to obtain a remote sensing image block to be color equalized and a color template image block; the color equalization unit is used to input the remote sensing image block to be color equalized and the color template image block into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color equalized, wherein the target remote sensing image block is a remote sensing image block to be color equalized that has completed color equalization; the splicing unit is used to splice the target remote sensing image blocks to obtain a remote sensing image with completed color equalization.

[0011] Furthermore, the device also includes: a registration unit, which is used to: determine whether there is a deviation between the geographical coordinates of the remote sensing image to be uniformly colored and the geographical coordinates of the color template image; if so, perform image set registration processing on the remote sensing image to be uniformly colored and the color template image.

[0012] Furthermore, the device also includes: a correction unit, which is used to: calculate target parameters between the target remote sensing image block and the color template image block corresponding to the target remote sensing image block, wherein the target parameters include: the mean, variance and standard deviation of each band, and the peak signal-to-noise ratio; based on the target parameters, determine whether to perform image color correction on the target remote sensing image block.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the method described in the first aspect above, and the processor is configured to execute the program stored in the memory.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored.

[0015] In an embodiment of the present invention, a remote sensing image to be color-leveled and a color template image are obtained, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color-leveled; based on geographic coordinates, the remote sensing image to be color-leveled and the color template image are cropped according to preset sizes to obtain a remote sensing image block to be color-leveled and a color template image block; the remote sensing image block to be color-leveled and the color template image block are input into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color-leveled, wherein the target remote sensing image block is a remote sensing image block to be color-leveled that has completed color-leveling; the target remote sensing image blocks are spliced ​​to obtain a remote sensing image that has completed color-leveling, thereby achieving the purpose of automatically color-leveling the remote sensing image. Furthermore, the existing automatic color-leveling technology for remote sensing images has the technical problems of uneven radiation accuracy within the image or oversaturation of brighter areas, thereby achieving the technical effect of improving the quality of automatic color-leveling of remote sensing images.

[0016] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The objectives and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a method for automatic color uniformity of remote sensing images based on a conditional adversarial neural network provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of an encoder-decoder structure provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the U-Net network structure provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the ResNet network block structure provided by an embodiment of the present invention;

[0023] Figure 5A schematic diagram of the ConvNext network block structure provided by an embodiment of the present invention;

[0024] Figure 6 A schematic diagram of an automatic color grading device for remote sensing images based on a conditional adversarial neural network provided by an embodiment of the present invention;

[0025] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] Example 1:

[0028] According to an embodiment of the present invention, an embodiment of a method for automatic color uniformity of remote sensing images based on a conditional adversarial neural network is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 FIG. 1 is a flow chart of a method for automatic color grading of remote sensing images based on a conditional adversarial neural network according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0030] Step S102, obtaining a remote sensing image to be color-leveled and a color template image, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color-leveled;

[0031] Step S104: based on the geographic coordinates, the remote sensing image to be color-leveled and the color template image are cropped according to preset sizes to obtain a remote sensing image block to be color-leveled and a color template image block;

[0032] Step S106: Inputting the remote sensing image block to be color-leveled and the color template image block into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color-leveled, wherein the target remote sensing image block is the remote sensing image block to be color-leveled that has been completed.

[0033] It should be noted that the above-mentioned conditional adversarial neural network model is a conditional adversarial neural network model built based on Pix2PixHD.

[0034] Step S108 : stitching the target remote sensing image blocks to obtain a remote sensing image with uniform color.

[0035] In an embodiment of the present invention, a remote sensing image to be color-leveled and a color template image are obtained, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color-leveled; based on geographic coordinates, the remote sensing image to be color-leveled and the color template image are cropped according to preset sizes to obtain a remote sensing image block to be color-leveled and a color template image block; the remote sensing image block to be color-leveled and the color template image block are input into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color-leveled, wherein the target remote sensing image block is a remote sensing image block to be color-leveled that has completed color-leveling; the target remote sensing image blocks are spliced ​​to obtain a remote sensing image that has completed color-leveling, thereby achieving the purpose of automatically color-leveling the remote sensing image. Furthermore, the existing automatic color-leveling technology for remote sensing images has the technical problems of uneven radiation accuracy within the image or oversaturation of brighter areas, thereby achieving the technical effect of improving the quality of automatic color-leveling of remote sensing images.

[0036] In an embodiment of the present invention, before step S104, the method further includes:

[0037] Determining whether there is a deviation between the geographic coordinates of the remote sensing image to be color-uniformed and the geographic coordinates of the color template image;

[0038] If so, image set registration processing is performed on the remote sensing image to be uniformly colored and the color template image.

[0039] In an embodiment of the present invention, the data preprocessing that is usually done before deep learning sample training includes data cropping, data enhancement, data augmentation, etc. In order to meet the demand for automatic color matching of remote sensing images, it is necessary to form several pairs of one-to-one corresponding image samples. Unlike the demand for automatic extraction of ground object information, there is no need to perform data enhancement and data augmentation to expand the diversity of samples. In addition, the amount of data in remote sensing images themselves is usually relatively large. Therefore, data preprocessing only requires data cropping. Unlike ordinary pictures, remote sensing images have geographic coordinates. During the cropping process, they need to be cropped according to geographic coordinates rather than pixel coordinates. According to the needs of deep learning training and the amount of video memory of the hardware GPU, the image to be color matched and the color template image are usually cropped into several 256*256 image block pairs. It is worth noting that there should be an overlap of about 10 pixels between the cropped image blocks to facilitate the subsequent splicing of the final results.

[0040] Since cropping is based on geographic coordinates, if the geographic locations of the image to be uniformly colored and the color template image deviate, geometric registration preprocessing is required before image cropping can be performed. Geometric registration primarily utilizes automatic matching to search for key points between the image to be uniformly colored and the color template image. Using these key points, a polynomial correction model or a spline correction model is used to geometrically register the image to be uniformly colored. Automatic matching of key points utilizes a pyramid-based hierarchical correlation coefficient matching method, which achieves both high matching efficiency and high matching accuracy. The primary metric used in automatic matching is the correlation coefficient. The correlation coefficient matching method uses the correlation coefficient (standardized covariance) as a similarity measure. In statistics, the correlation coefficient is used to represent the correlation between two random variables. Extended to image matching, it can be used to represent the degree of similarity between two images of the same size.

[0041]

[0042] R(X,Y) is called the correlation coefficient between the two images. E(X) and E(Y) are the grayscale means of the two images, D(X) and D(Y) are the variances of the two images, and E(XY) is the mean of the multiplication of corresponding points in the two images. Their definitions are the same as those in general statistical theory. After matching a number of points with the same name, the remote sensing image to be color-leveled is resampled using a geometric polynomial correction model to obtain the registered image. The registered image maintains the same geographic coordinate accuracy as the color template image. Subsequently, the sample data can be automatically cropped in batches based on the geographic coordinates to obtain the remote sensing image blocks to be color-leveled and the color template image blocks.

[0043] The conditional adversarial neural network model is introduced below.

[0044] Conventional deep learning tasks such as image classification, object detection, and semantic or instance segmentation all boil down to prediction. Image classification predicts a single category, object detection predicts both range and category, and semantic or instance segmentation predicts the category of each pixel. Adversarial neural networks, on the other hand, generate a new object, such as an image.

[0045] The task of image processing is usually to convert the input image into another image after a certain transformation. The traditional automatic color matching task of images requires the statistical calculation of the mean and variance of each band of the image to be matched and the color template image to establish a mathematical conversion model. The established mathematical conversion model aims to reduce the color error between the image to be matched and the color template image. This is the same as the main purpose of traditional automatic color matching of images. The same is true for the color matching method based on deep learning. The difference is that the established mathematical model is replaced by a more complex neural network model with more parameters.

[0046] Image-to-image translation is a very important application direction of GAN. Image-to-image translation is the process of obtaining the desired output image based on an input image. It can be regarded as a mapping between images. Common image restoration and super-resolution are actually examples of image-to-image translation. It also includes processes such as generating from labels to images and generating from image edges to images. Pix2Pix is ​​an algorithm that applies GAN to supervised image-to-image translation. Supervision means that the training data is paired. The network can train on one-to-one input samples and learn the mapping between input images and output images. It can handle and output a series of complex application scenarios, such as day to night, grayscale to color, vector to raster, contour to entity, etc., and achieves good results. However, Pix2Pix will produce blurring of the synthesized details of high-resolution images. Later, scholars proposed the pix2pixHD method to solve the problem of high resolution and image quality, which also achieved good results. Therefore, the present invention attempts to use the Pix2PixHD network model to complete the automatic color grading function of remote sensing images.

[0047] Since the advent of convolutional neural networks, various traditional image processing algorithms have been experimenting with and adapting to deep learning. While convolutional neural networks can automatically learn certain things, they still require human experience and design. Designing an appropriate loss function is one such approach. For image color grading tasks, it's crucial to tell the convolutional neural network what to learn. If a convolutional neural network is taught an inappropriate loss function, the results will be unsatisfactory.

[0048] The loss function of the pix2pix algorithm based on the conditional adversarial generation network model is usually:

[0049] L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x,z)))]

[0050] Among them, x is the input image to be uniformly colored, y is the color template image, z is random noise, G represents the generator, and D represents the discriminator. x needs to be input into G and D, the input of G is (x, z), the output is the generated image G(x, z), and D needs to distinguish between (x, G(x, z)) and (x, y). If the random noise z is not used, the adversarial network can still learn to map from x to y, but it will produce deterministic output. The generator G hopes that the discriminator will regard the pictures it generates as real, while the discriminator D hopes to correctly distinguish between real samples and samples generated by G. The optimization goal of the generator G is to make L cGAN(G, D) is the smallest, and the optimization goal of the discriminator D is to make L cGAN (G, D) is the largest. This is the meaning and significance of adversarial training.

[0051] In order to ensure the similarity between the uniform color output image and the color template image, it is necessary to minimize the distance between the two. After a large amount of data experiments, it is shown that combining the C-GAN loss function with the traditional L1 or L2 loss function can bring better results. The L1 loss function can increase the clarity of the output, so the L1 loss function is selected. The L1 loss function is as follows:

[0052] L L1 (G)=E x,y,z [‖yG(x,z)‖1]

[0053] Adding the adversarial loss and the L1 loss gives us the final overall loss function:

[0054]

[0055] To distinguish between real and synthetic images at high resolution, the discriminator must have a large receptive field, which requires a deeper network or larger convolution kernels. Both options increase network capacity, making the network more prone to overfitting and increasing the storage space required for training. The loss function of the Pix2pixHD algorithm optimizes the discriminator at a pyramid level based on the loss function of pix2pix. Taking into account multiple pyramid levels, the overall loss function is as follows: Here, three discriminators (D1, D2, and D3) are used to process images of different scales. They have the same network structure:

[0056]

[0057] Where k represents the different levels of the pyramid. Based on the above loss function, a feature matching loss function is added to stabilize the model training because the model structure has multiple scales. Features are extracted from multiple layers of D to learn the intermediate representations that match the real image and the synthetic image. The multi-layer feature matching loss function is defined as:

[0058]

[0059] Where T is the total number of extracted feature layers, three layers are used here; N i Indicates the number of elements in each layer; Denotes the discriminator D k The final target loss function combines the GAN loss function and the feature matching loss function as follows:

[0060]

[0061] λ is used to adjust the weights of these two components, D k As a feature extractor only, it does not maximize the feature matching loss function L FM .

[0062] Generally speaking, a GAN consists of two types of networks, G and D. G is the generator, responsible for generating images. Specifically, after receiving a random code z as input, it outputs a fake image G(z) automatically generated by a neural network. The other network, D, is the discriminator, responsible for judging whether the image is real or fake. A real image is output as 1, while a fake image is output as 0. As the two networks compete with each other, their capabilities increase: G's generated images become increasingly realistic, and D becomes increasingly adept at judging the authenticity of images. At this point, D is discarded and G is used as the image generator. This involves minimizing D's ability to judge G while maximizing its capabilities—a min-max problem. Pix2pix has improved the network structures of both the generator and discriminator of a CGAN.

[0063] The task of the generator network is to restore an image with the same color as the color template image from the image to be uniformed without reducing the clarity.

[0064] The most commonly used generator structure in traditional GAN ​​is the encoder-decoder type, while pix2pix uses the U-Net network structure model, such as Figure 2 and Figure 3 shown.

[0065] U-Net is a fully convolutional architecture proposed by the Pattern Recognition and Image Processing Group at the University of Freiburg in Germany. Compared to the common encoder-decoder architecture, which downsamples to a lower dimension and then upsamples to the original resolution, U-Net incorporates a skip-connection mechanism. This mechanism concatenates corresponding feature maps and decoded feature maps of the same size, channel by channel, preserving pixel-level detail at different resolutions. U-Net is highly effective in enhancing detail.

[0066] Pix2PixHD is an upgraded version of Pix2Pix, which is essentially a conditional GAN ​​that effectively translates images from one domain into images from another under supervision. However, the generated images have a resolution of only 256x256, and using GANs to generate high-resolution images is challenging. Pix2PixHD can generate 2k high-definition images, converting semantic and instance segmentation information into realistic images. It can be considered the inverse of semantic and instance segmentation. The generator of Pix2pixHD has been improved upon the generator network of pix2pix.

[0067] The Pix2pixHD generator consists of two parts, G1 and G2, with G2 further divided into two parts. G1 is identical to the Pix2Pix generator. The left half of G2 extracts features, adds them to the previous layer's output layer of G1, and fuses them. This fused information is then fed into the second half of G2 to output a high-resolution image.

[0068] The Pix2pixHD generator used in this invention has made changes to the basic Block of the network from the traditional ResNetBlock to the Convnext Block. The specific network structures of ResNet and ConvNeXt are as follows: Figure 4 and Figure 5 The specific network structure parameters of ResNet and ConvNeXt are shown in the following table:

[0069]

[0070] The comparison results in the table above show that the ConvNeXt network has a higher computational load and larger parameters than the ResNet. Remote sensing images have a large field of view, and the color grading process must preserve the various features of the original image, such as the image's clarity. The image's information content is complex and diverse, so a large number of parameters facilitates learning more feature information and enables faster data processing.

[0071] In addition to replacing the network structure, we also optimized the details of the network block:

[0072] The activation function ReLU is replaced by Gelu

[0073] The formula of ReLU is ReLU(X)=MAX(0,X). If the input value is less than 0, the output is 0. If the input value is greater than 0, the output is equal to the input. The formula is as follows:

[0074]

[0075] From the differential equation of ReLU, we can see that if the input is greater than 0, the output is equal to 1, and if the input is less than 0, the output is 0. Therefore, when we use the ReLU activation function, we cannot get very small values. At the same time, because the content information of remote sensing images is complex and diverse, too many values ​​are less than 0 when calculating the gradient, so the weights cannot be updated, which leads to poor model performance.

[0076] The formula for GeLU is:

[0077]

[0078] Fewer activation functions

[0079] Only one Gelu activation function is added between two 1*1 convolutions. Excessive activation functions can convert linear functions into nonlinear functions, enhancing the ability of the network model to learn more complex information relationships. On the surface, this is beneficial for learning complex content in remote sensing images. However, as the complexity increases, the training requirements are also higher, which increases the difficulty of training and may even lead to problems such as overfitting of the model.

[0080] Fewer normalization layers

[0081] Only an LN is added before the first 1*1 convolution. The normalization strategy is to subtract the mean and divide it by the variance. After processing, the data will become a normal distribution with a mean of 0 and a variance of 1. For remote sensing images, the content is complex and diverse, which will effectively prevent the uneven distribution of some data.

[0082] BN layer is replaced by LN layer

[0083] We usually use BN in CNN, and LN is usually used to solve the problem that the BN normalized statistics will have too large deviations when the sample size is small. Although the performance of the residual network will decrease if BN is directly replaced with LN, it performs better than BN in ConvNeXt.

[0084] Discriminator network design

[0085] The discriminator network's task is to determine whether the output image is real or fake. Pix2Pix uses reconstruction to address low-frequency components and GANs to address high-frequency components. Traditional L1 loss is used to ensure that the generated image is as similar as possible to the training image, while GANs are used to construct high-frequency details. The discriminator uses a Markov discriminator (PatchGAN) to divide the generated image into multiple fixed-size patches for input for authenticity judgment, as different patches can be considered independent of each other. An image is cut into patches of different N x N sizes, and the discriminator judges each patch as real or fake. The results of all patches in an image are averaged as the final discriminator output. Extensive data testing has shown that when the patch size is 70*70, the visual result is similar to that of directly inputting the entire image into the discriminator.

[0086] The advantages of using PatchGAN are also obvious. Each block has a smaller amount of data and fewer parameters, which brings higher training efficiency and can be applied to the entire large image. In addition, because the generator G itself is fully convolutional, there is no limit on the image scale, and if the discriminator D processes the image according to the patch, there is no limit on the image size. This will make the entire C-GAN framework have no limit on the image size, increasing the scalability of the framework.

[0087] The Pix2pixHD discriminator differs from pix2pix in that it uses a multi-scale discriminator, performing discrimination at three levels of the feature map pyramid and averaging the results. The three scales used for discrimination are: the original feature map, a downsampled version of the feature map to half, and a downsampled version of the feature map to quarter. Clearly, higher pyramid levels have a larger receptive field and place greater emphasis on global color consistency in the image.

[0088] In an embodiment of the present invention, the conditional adversarial neural network model includes: a first generator, a second generator, a first discriminator, and a second discriminator, and step S106 includes the following steps:

[0089] The first generator of the initial remote sensing image block to be color-homogenized is used to obtain an initial remote sensing image block, wherein the initial remote sensing image block to be color-homogenized is any one of the remote sensing image blocks to be color-homogenized, the initial remote sensing image block is an initial remote sensing image block to be color-homogenized that has completed color-homogenization, and the resolution of the initial remote sensing image block is smaller than the resolution of the initial remote sensing image block to be color-homogenized;

[0090] Inputting the initial remote sensing image block to be color-homogenized and the initial remote sensing image block into the first discriminator to obtain a first prediction probability value, wherein the first prediction probability value is used to indicate whether the initial remote sensing image block to be color-homogenized and the initial remote sensing image block are a pair of real images;

[0091] If the first predicted probability value is greater than a first preset threshold, inputting the initial remote sensing image block to be uniformly colored and the initial remote sensing image block into the second generator to obtain an intermediate remote sensing image block, wherein the resolution of the intermediate remote sensing image block is equal to the resolution of the initial remote sensing image block to be uniformly colored;

[0092] Inputting the intermediate remote sensing image block and the initial color template image block into the second discriminator to obtain a second prediction probability value, wherein the initial color template image block is a color template image block having the same geographic coordinates as the initial remote sensing image block to be uniformly colored, and the second prediction probability value is used to indicate whether the intermediate remote sensing image block and the initial color template image block are a pair of real images;

[0093] If the second prediction probability value is greater than a second preset threshold, the intermediate remote sensing image block is determined as the target remote sensing image block.

[0094] In the embodiment of the present invention, first, the initial remote sensing image block to be uniformly colored is represented by x, and the initial color template image block is represented by y. Paired images (x and y) are required during training.

[0095] x is used as the input of the first generator G1 (removing the random noise z will not have much impact on the generation effect, but if x and z are combined as the first generator G1, more diverse outputs can be obtained. The diversity of results is not required in the color uniformity process) to generate the initial remote sensing image block G(x1);

[0096] Then G(x1) and x are merged together based on the channel dimension, and finally used as the input of the first discriminator to obtain the first predicted probability value, which indicates whether the input is a pair of real images. The closer the probability value is to 1, the more certain the first discriminator D is that the input is a pair of real images.

[0097] If the first predicted probability value is greater than the first preset threshold, the initial remote sensing image block to be uniformed and the initial remote sensing image block are input into the second generator to obtain the intermediate remote sensing image block G(x2).

[0098] The initial color template image block y and the intermediate remote sensing image block G(x2) are also merged together based on the channel dimension and used as the input of the second discriminator to obtain the second probability prediction value. Therefore, the training goal of the second discriminator is to output a small probability value (for example, the minimum is 0) when the input is not a pair of real images (x and G(x2)), and to output a large probability value (for example, the maximum is 1) when the input is a pair of real images (x and G(x2)).

[0099] When the second probability prediction value is greater than the second preset threshold, the intermediate remote sensing image block is the target remote sensing image block.

[0100] In an embodiment of the present invention, before step S108, the method further includes:

[0101] Calculating target parameters between the target remote sensing image block and the color template image block corresponding to the target remote sensing image block, wherein the target parameters include: mean, variance and standard deviation of each band, and peak signal-to-noise ratio;

[0102] Based on the target parameters, it is determined whether to perform image color correction on the target remote sensing image block.

[0103] In an embodiment of the present invention, after the network model training of the Pix2PixHD generator and discriminator is completed, the generator can be directly used to generate a target remote sensing image block with a color that is basically consistent with the color template image. After simple image stitching, the final remote sensing uniform color result image can be generated.

[0104] The mean, variance, standard deviation, etc. of each band of the color-homogenized image and the color template image within the same geographic coordinate range are statistically calculated. The smaller the mean difference, the better the automatic color balancing effect. The closer the histogram distribution, the better the automatic color balancing effect.

[0105] In addition, the embodiment of the present invention introduces a commonly used evaluation index in the field of image reconstruction - Peak Signal-to-Noise Ratio (PSNR) to measure the texture quality of the image after color uniformity. This value is calculated from the target remote sensing image block and the color template image block corresponding to the target remote sensing image block. The larger this value is, the better the image quality is.

[0106] The specific calculation formula is as follows:

[0107]

[0108] Where MSE is the root mean square error, m and n represent the size of the image, MAX is the maximum pixel value of the image, and PSNR represents the peak signal-to-noise ratio of the final calculation.

[0109] To address the problems of uneven radiation accuracy or oversaturation of brighter areas caused by existing automatic color grading algorithms for remote sensing images, the present invention introduces a Conditional Generative Adversarial Network (C-GAN), a recently popular model in the field of deep learning. This network uses iterative adversarial training of image generator and discriminator models between the image to be graded and the template image to form an optimal conversion relationship model from the source image to the target image. This allows for pixel-to-pixel translation without explicitly calculating information such as the variance and mean of the image to be graded and the reference image. This approach leverages the hyperparameters of the deep learning training model to address the problems of uneven radiation accuracy and oversaturation of brighter areas caused by traditional automatic color grading algorithms for remote sensing images.

[0110] Example 2:

[0111] An embodiment of the present invention also provides an automatic color equalization device for remote sensing images based on a conditional adversarial neural network. The automatic color equalization device for remote sensing images based on a conditional adversarial neural network is used to execute the automatic color equalization method for remote sensing images based on a conditional adversarial neural network provided in the above content of the embodiment of the present invention. The following is a specific introduction to the automatic color equalization device for remote sensing images based on a conditional adversarial neural network provided in an embodiment of the present invention.

[0112] like Figure 6 As shown, Figure 6 Schematic diagram of the automatic color equalization device for remote sensing images based on conditional adversarial neural network. The automatic color equalization device for remote sensing images based on conditional adversarial neural network includes: an acquisition unit 10, a cropping unit 20, a color equalization unit 30 and a splicing unit 40.

[0113] The acquisition unit is configured to acquire a remote sensing image to be color-homogenized and a color template image, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color-homogenized;

[0114] The cropping unit is configured to crop the remote sensing image to be color-homogenized and the color template image according to preset sizes based on geographic coordinates to obtain a remote sensing image block to be color-homogenized and a color template image block;

[0115] The color grading unit is configured to input the remote sensing image block to be color graded and the color template image block into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color graded, wherein the target remote sensing image block is the remote sensing image block to be color graded after color graded;

[0116] The stitching unit is used to stitch the target remote sensing image blocks to obtain a remote sensing image with uniform color.

[0117] In an embodiment of the present invention, a remote sensing image to be color-leveled and a color template image are obtained, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color-leveled; based on geographic coordinates, the remote sensing image to be color-leveled and the color template image are cropped according to preset sizes to obtain a remote sensing image block to be color-leveled and a color template image block; the remote sensing image block to be color-leveled and the color template image block are input into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color-leveled, wherein the target remote sensing image block is a remote sensing image block to be color-leveled that has completed color-leveling; the target remote sensing image blocks are spliced ​​to obtain a remote sensing image that has completed color-leveling, thereby achieving the purpose of automatically color-leveling the remote sensing image. Furthermore, the existing automatic color-leveling technology for remote sensing images has the technical problems of uneven radiation accuracy within the image or oversaturation of brighter areas, thereby achieving the technical effect of improving the quality of automatic color-leveling of remote sensing images.

[0118] Example 3:

[0119] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the method described in the above embodiment 1, and the processor is configured to execute the program stored in the memory.

[0120] See also Figure 7 An embodiment of the present invention further provides an electronic device 100, comprising: a processor 50, a memory 51, a bus 52 and a communication interface 53, wherein the processor 50, the communication interface 53 and the memory 51 are connected via the bus 52; the processor 50 is used to execute an executable module stored in the memory 51, such as a computer program.

[0121] The memory 51 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 53 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0122] The bus 52 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0123] Among them, the memory 51 is used to store programs, and the processor 50 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 50 or implemented by the processor 50.

[0124] The processor 50 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in the processor 50. The processor 50 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 51 , and the processor 50 reads the information in the memory 51 and completes the steps of the above method in combination with its hardware.

[0125] Example 4:

[0126] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the above embodiment 1 are executed.

[0127] In addition, in the description of the embodiments of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0128] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0132] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A remote sensing image automatic color grading method based on conditional adversarial neural network, characterized in that: include: Acquire a remote sensing image to be color-leveled and a color template image, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color-leveled; Based on the geographic coordinates, the remote sensing image to be color-uniformed and the color template image are cropped according to preset sizes to obtain a remote sensing image block to be color-uniformed and a color template image block; Inputting the remote sensing image block to be color-uniformed and the color template image block into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color-uniformed, wherein the target remote sensing image block is the remote sensing image block to be color-uniformed after color uniformity is achieved; Splicing the target remote sensing image blocks to obtain a remote sensing image with uniform color; The conditional adversarial neural network model includes: a first generator, a second generator, a first discriminator, and a second discriminator. The remote sensing image block to be color-homogenized and the color template image block are input into the conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color-homogenized, including: The first generator of the initial remote sensing image block to be color-homogenized is used to obtain an initial remote sensing image block, wherein the initial remote sensing image block to be color-homogenized is any one of the remote sensing image blocks to be color-homogenized, the initial remote sensing image block is an initial remote sensing image block to be color-homogenized that has completed color-homogenization, and the resolution of the initial remote sensing image block is smaller than the resolution of the initial remote sensing image block to be color-homogenized; Inputting the initial remote sensing image block to be color-homogenized and the initial remote sensing image block into the first discriminator to obtain a first prediction probability value, wherein the first prediction probability value is used to indicate whether the initial remote sensing image block to be color-homogenized and the initial remote sensing image block are a pair of real images; If the first predicted probability value is greater than a first preset threshold, inputting the initial remote sensing image block to be uniformly colored and the initial remote sensing image block into the second generator to obtain an intermediate remote sensing image block, wherein the resolution of the intermediate remote sensing image block is equal to the resolution of the initial remote sensing image block to be uniformly colored; Inputting the intermediate remote sensing image block and the initial color template image block into the second discriminator to obtain a second prediction probability value, wherein the initial color template image block is a color template image block having the same geographic coordinates as the initial remote sensing image block to be uniformly colored, and the second prediction probability value is used to indicate whether the intermediate remote sensing image block and the initial color template image block are a pair of real images; If the second prediction probability value is greater than a second preset threshold, the intermediate remote sensing image block is determined as the target remote sensing image block.

2. The method according to claim 1, characterized in that Before respectively cropping the remote sensing image to be color-homogenized and the color template image according to preset sizes based on the geographic coordinates to obtain the remote sensing image block to be color-homogenized and the color template image block, the method further includes: Determining whether there is a deviation between the geographic coordinates of the remote sensing image to be color-uniformed and the geographic coordinates of the color template image; If so, image set registration processing is performed on the remote sensing image to be uniformly colored and the color template image.

3. The method according to claim 1, characterized in that The conditional adversarial neural network model is a conditional adversarial neural network model built based on Pix2PixHD.

4. The method according to claim 1, wherein Before splicing the target remote sensing image blocks to obtain a remote sensing image with uniform color, the method further includes: Calculating target parameters between the target remote sensing image block and the color template image block corresponding to the target remote sensing image block, wherein the target parameters include: mean, variance and standard deviation of each band, and peak signal-to-noise ratio; Based on the target parameters, it is determined whether to perform image color correction on the target remote sensing image block.

5. A remote sensing image automatic color grading device based on conditional adversarial neural network, characterized in that: include: Acquisition unit, cropping unit, color uniformity unit and splicing unit, where: The acquisition unit is configured to acquire a remote sensing image to be color-homogenized and a color template image, wherein the area corresponding to the color template image includes the area corresponding to the remote sensing image to be color-homogenized; The cropping unit is configured to crop the remote sensing image to be color-homogenized and the color template image according to preset sizes based on geographic coordinates to obtain a remote sensing image block to be color-homogenized and a color template image block; The color grading unit is configured to input the remote sensing image block to be color graded and the color template image block into a conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color graded, wherein the target remote sensing image block is the remote sensing image block to be color graded after color graded; The stitching unit is used to stitch the target remote sensing image blocks to obtain a remote sensing image with uniform color; The conditional adversarial neural network model includes: a first generator, a second generator, a first discriminator, and a second discriminator. The remote sensing image block to be color-homogenized and the color template image block are input into the conditional adversarial neural network model to obtain a target remote sensing image block corresponding to the remote sensing image block to be color-homogenized, including: The first generator of the initial remote sensing image block to be color-homogenized is used to obtain an initial remote sensing image block, wherein the initial remote sensing image block to be color-homogenized is any one of the remote sensing image blocks to be color-homogenized, the initial remote sensing image block is an initial remote sensing image block to be color-homogenized that has completed color-homogenization, and the resolution of the initial remote sensing image block is smaller than the resolution of the initial remote sensing image block to be color-homogenized; Inputting the initial remote sensing image block to be color-homogenized and the initial remote sensing image block into the first discriminator to obtain a first prediction probability value, wherein the first prediction probability value is used to indicate whether the initial remote sensing image block to be color-homogenized and the initial remote sensing image block are a pair of real images; If the first predicted probability value is greater than a first preset threshold, inputting the initial remote sensing image block to be uniformly colored and the initial remote sensing image block into the second generator to obtain an intermediate remote sensing image block, wherein the resolution of the intermediate remote sensing image block is equal to the resolution of the initial remote sensing image block to be uniformly colored; Inputting the intermediate remote sensing image block and the initial color template image block into the second discriminator to obtain a second prediction probability value, wherein the initial color template image block is a color template image block having the same geographic coordinates as the initial remote sensing image block to be uniformly colored, and the second prediction probability value is used to indicate whether the intermediate remote sensing image block and the initial color template image block are a pair of real images; If the second prediction probability value is greater than a second preset threshold, the intermediate remote sensing image block is determined as the target remote sensing image block.

6. The device according to claim 5, characterized in that The device further comprises a registration unit, configured to: Determining whether there is a deviation between the geographic coordinates of the remote sensing image to be color-uniformed and the geographic coordinates of the color template image; If so, image set registration processing is performed on the remote sensing image to be uniformly colored and the color template image.

7. The device according to claim 5, characterized in that The device further includes a correction unit, configured to: Calculating target parameters between the target remote sensing image block and the color template image block corresponding to the target remote sensing image block, wherein the target parameters include: mean, variance and standard deviation of each band, and peak signal-to-noise ratio; Based on the target parameters, it is determined whether to perform image color correction on the target remote sensing image block.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store a program for supporting the processor to execute the method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are executed.

Images