Multi-source remote sensing image radiometric normalization method and system based on frequency domain decomposition

Abstract

The present invention discloses a multi-source remote sensing image radiation normalization method and system based on frequency domain decomposition. The method includes: obtaining a remote sensing image set, arbitrarily selecting two remote sensing images with different radiation domains to form an image pair; converting the radiation value of each channel into an input image matrix to obtain a new feature matrix; performing frequency domain decomposition to obtain low-frequency components and high-frequency components, performing multi-layer feature extraction on the low-frequency components and the reference image to obtain image features of each layer respectively; generating high-frequency image blocks from the high-frequency components and each layer of image features based on an image conversion fusion model; and generating a normalized remote sensing image with consistent color. The present invention generates source image domain data with the style of the target image domain, so that the features of the source image domain data after radiation migration are more similar to the feature distribution of the target image domain data. Therefore, the network can pre-learn the data distribution in the target image domain, thereby effectively solving the domain offset problem between remote sensing images and improving the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing data processing, and in particular to a multi-source remote sensing image radiation normalization method and system based on frequency domain decomposition. Background Art

[0002] The volume and quality of remote sensing data are experiencing rapid growth. However, the data's integrity and comparability across time and space remain insufficient, hindering further technological development and widespread implementation. At the core of this problem lies the poor interoperability and lack of consistency among multi-source data, which limits data integration and collaborative applications. Furthermore, this core issue negatively impacts the generalization capabilities of classification methods.

[0003] When performing remote sensing image feature extraction tasks, image noise has a significant impact, resulting in relatively low image quality due to differences in shooting conditions, sensor type, atmospheric radiation, and weather conditions. Domain shift often occurs, leading to significant distribution differences between the source and target image domains. This domain shift weakens the generalization capabilities of traditional models in cross-domain tasks, making them unable to meet the needs of complex cross-domain processing tasks between multi-source remote sensing. Currently, most radiometric normalization methods are based on paired image processing, requiring complex data pre-processing and requiring further improvement in normalization. Furthermore, there is a significant loss of texture detail.

[0004] In other words, there are still problems of domain offset between remote sensing images and recognition errors caused by domain offset. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a multi-source remote sensing image radiation normalization method and system based on frequency domain decomposition.

[0006] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0007] A multi-source remote sensing image radiometric normalization method based on frequency domain decomposition includes the following steps:

[0008] Obtain a remote sensing image set, and arbitrarily select two remote sensing images with different radiation fields to form an image pair, one of which is used as a reference image and the other as an image to be processed;

[0009] The image to be processed is converted into an input image matrix using the radiation values ​​of each channel and enhanced to obtain a new feature matrix, wherein the new feature matrix has the same dimension as the input image matrix;

[0010] Performing frequency domain decomposition on the new feature matrix based on each color channel to obtain low-frequency components and high-frequency components, performing multi-layer feature extraction on the low-frequency components and the reference image to obtain image features of each layer respectively;

[0011] Construct an image conversion fusion model, and generate high-frequency image blocks based on the high-frequency components and each layer of image features. Specifically, the image conversion fusion model includes a feature receiving module, a self-attention optimization module, an image conversion module and a splicing module. The feature receiving module receives the image features of each layer. The self-attention optimization module performs serial processing, attention processing and conversion processing on the image features of each layer to obtain an attention image and a transformation feature map. The image conversion module generates and discriminates the transformation feature map and the image features of each layer to obtain a low-frequency image block with the color of the reference image. The splicing module performs feature splicing on the low-frequency image block, the high-frequency component and the attention image to obtain a high-frequency image block.

[0012] The high-frequency image block is fused with a reference image to generate a normalized remote sensing image with consistent color.

[0013] As an implementation method, each new feature in the new feature matrix is ​​subjected to frequency domain decomposition processing in combination with each color channel through Haar wavelet transform, and each new feature is decomposed into a low-frequency component and a high-frequency component. The high-frequency component includes a horizontal component, a vertical component, and a diagonal component, which are expressed as follows:

[0014]

[0015] in, , Represent the low-frequency approximation coefficient and high-frequency detail coefficient of the channel respectively, where Respectively represent the row and column numbers of pixels in the image, and their value ranges are , , Represent the height and width of the remote sensing image respectively.

[0016] As an implementation method, the encoder performs multi-layer feature extraction on the low-frequency component and the reference image to obtain image features of each layer, and the image features of each layer are expressed as: ,in, Respectively represent the number of channels, the height of the remote sensing image and the width of the remote sensing image, Indicates a feature set with the same number of channels, height and width as the remote sensing image, Represents the image features of each layer, Indicates the number of layers.

[0017] As an implementation method, the self-attention optimization module performs concatenation processing, attention processing, and conversion processing on each layer of image features to obtain a transformed feature map, including the following steps:

[0018] Concatenate the image features of the current layer with the downsampled image features of the previous layers to obtain concatenated features;

[0019] pass The convolutional layer and normalization layer perform linear transformation on the concatenated features to obtain query embedding, key embedding, and value embedding respectively;

[0020] Calculating attention based on the query embedding and the key embedding, and obtaining statistics for each distribution based on the attention and the value embedding, the statistics including an attention-weighted mean and an attention-weighted standard deviation;

[0021] Based on each position and each color channel, the attention weighted standard deviation is used as a scaling factor and the attention weighted average is used as a shift factor to transform the image to obtain a transformed feature map;

[0022] The series characteristics are expressed as:

[0023] Query embedding, key embedding, and value embedding are represented as:

[0024]

[0025]

[0026]

[0027] The attention is expressed as:

[0028] The attention weighted average is expressed as:

[0029] The attention weighted standard deviation is expressed as:

[0030] The transformation feature map is expressed as:

[0031] in, Represents a bilinear interpolation layer that downsamples image features to match shape, represents the connection operation along the channel dimension, 、 and Respectively Convolutional layers, represents channel mean square error normalization, , , represents the element-wise vector product, represents query embedding, Indicates key embedding and represents the value embedding, S represents the attention-weighted standard deviation, M represents the attention-weighted mean, Indicates the number of layers, T indicates transposition, Represents the image features of each layer, Represents a collection, They represent the number of channels, the height of the remote sensing image, and the width of the remote sensing image respectively.

[0032] As an implementation method, the image conversion module includes a first generation and discrimination unit and a second generation and discrimination unit, the first generation and discrimination unit includes a first generator and a first discriminator, and the second generation and discrimination unit includes a second generator and a second discriminator;

[0033] The first generator converts the transformed feature map of the image to be processed into a feature image of the reference image, and the second generator translates the feature image of the reference image into the transformed feature map of the image to be processed;

[0034] The first discriminator discriminates the authenticity of the feature image of the reference image;

[0035] The second discriminator discriminates the authenticity of the transformed feature map of the processed image.

[0036] As an implementation method, the first generator and the second generator have the same structure, respectively including an encoder and a decoder, the encoder including a convolution layer, a first activation function, a first batch normalization and a pooling layer, and the decoder including a deconvolution layer, a second activation function, a second batch normalization, a skip connection layer and a final output layer;

[0037] The convolution layer performs convolution operations on the transformed feature map of the image to be processed or the feature map of the reference image based on a 3×3 convolution kernel to extract local features respectively;

[0038] The first activation function is the ReLU activation function, which learns complex features and obtains the second local features;

[0039] Based on the first batch normalization, the second local features are normalized. By normalizing the mean and variance of the second local features, the training process is stabilized and the convergence is accelerated.

[0040] Based on the pooling layer, the normalized second local feature is processed to reduce the spatial dimension to obtain a low-dimensional feature map;

[0041] The deconvolution layer of the decoder increases the dimension of the low-dimensional feature map to obtain a normal-resolution feature map;

[0042] The second activation function is the ReLU activation function, which learns complex features and obtains the third feature map;

[0043] Based on the second batch normalization, the third feature map is normalized. By normalizing the mean and variance of the third feature map, the training process is stabilized and the convergence is accelerated;

[0044] Through the skip connection layer, the feature map in the encoder is passed to the corresponding layer of the decoder to retain the detailed information;

[0045] Through the final output layer, the pixel values ​​are normalized to [-1, 1] or [0, 1] based on the Tanh activation function or the Sigmoid activation function, and the feature image of the reference image or the transformed feature map of the image to be processed with the same size as the transformed feature map of the image to be processed or the feature image of the reference image is output, that is, the low-frequency image block.

[0046] As an implementation method, the first discriminator and the second discriminator have the same structure, respectively including an input layer, a hidden layer and an output layer, and the hidden layer includes a convolution layer, a third activation function and a pooling layer;

[0047] The input layer receives the feature image of the reference image or the transformed feature map of the image to be processed;

[0048] The convolution layer uses a 3×3 convolution kernel to extract features;

[0049] The third activation function is the LeakyReLU activation function, which learns complex features;

[0050] The pooling layer gradually reduces the spatial dimension through pooling operations;

[0051] The output layer uses the Sigmoid activation function to output a probability value between 0 and 1 to judge the authenticity of the feature image of the reference image or the transformed feature map of the image to be processed.

[0052] As an implementation method, the performance of the image conversion fusion model is also evaluated, including:

[0053] The normalization results of the CycleGAN model are verified and evaluated based on the structural similarity index, peak signal-to-noise ratio, learned perceptual image patch similarity or spectral angle mapper, and the CycleGAN model is iteratively optimized based on the evaluation results.

[0054] As an implementation method, the structural similarity index is constructed by brightness similarity, contrast similarity and structural similarity, as follows:

[0055] The brightness similarity is expressed as follows:

[0056]

[0057] Contrast similarity, expressed as follows:

[0058]

[0059] Structural similarity, expressed as follows:

[0060]

[0061] Then the structural similarity index is expressed as follows:

[0062]

[0063] in, Represents images respectively The average value of Represents images respectively The standard deviation of Representing an image The covariance of Pick , the value range of the structural similarity index is , the closer the value is to 1, the more similar the two images are.

[0064] A multi-source remote sensing image radiometric normalization system based on frequency domain decomposition includes:

[0065] The data acquisition module acquires a remote sensing image set and arbitrarily selects two remote sensing images with different radiation fields to form an image pair, one of which is used as a reference image and the other as an image to be processed;

[0066] A feature decomposition module converts the image to be processed into an input image matrix using the radiance values ​​of each channel and performs enhancement processing to obtain a new feature matrix, wherein the new feature matrix has the same dimension as the input image matrix;

[0067] A feature extraction module performs frequency domain decomposition on the new feature matrix based on each color channel to obtain low-frequency components and high-frequency components, and performs multi-layer feature extraction on the low-frequency components and the reference image to obtain image features of each layer;

[0068] Construct a fusion module, construct an image conversion fusion model, and generate high-frequency image blocks based on the high-frequency components and each layer of image features based on the image conversion fusion model. Specifically, the image conversion fusion model includes a feature receiving module, a self-attention optimization module, an image conversion module and a splicing module. The feature receiving module receives the image features of each layer. The self-attention optimization module performs serial processing, attention processing and conversion processing on the image features of each layer to obtain an attention image and a transformation feature map. The image conversion module generates and discriminates the transformation feature map and the image features of each layer to obtain a low-frequency image block with the color of the reference image. The splicing module performs feature splicing on the low-frequency image block, the high-frequency component and the attention image to obtain a high-frequency image block.

[0069] The image fusion module fuses the high-frequency image block with the reference image to generate a normalized remote sensing image with consistent color.

[0070] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following method:

[0071] Obtain a remote sensing image set, and arbitrarily select two remote sensing images with different radiation fields to form an image pair, one of which is used as a reference image and the other as an image to be processed;

[0072] The image to be processed is converted into an input image matrix using the radiation values ​​of each channel and enhanced to obtain a new feature matrix, wherein the new feature matrix has the same dimension as the input image matrix;

[0073] Performing frequency domain decomposition on the new feature matrix based on each color channel to obtain low-frequency components and high-frequency components, performing multi-layer feature extraction on the low-frequency components and the reference image to obtain image features of each layer respectively;

[0074] Construct an image conversion fusion model, and generate high-frequency image blocks based on the high-frequency components and each layer of image features. Specifically, the image conversion fusion model includes a feature receiving module, a self-attention optimization module, an image conversion module and a splicing module. The feature receiving module receives the image features of each layer. The self-attention optimization module performs serial processing, attention processing and conversion processing on the image features of each layer to obtain an attention image and a transformation feature map. The image conversion module generates and discriminates the transformation feature map and the image features of each layer to obtain a low-frequency image block with the color of the reference image. The splicing module performs feature splicing on the low-frequency image block, the high-frequency component and the attention image to obtain a high-frequency image block.

[0075] The high-frequency image block is fused with a reference image to generate a normalized remote sensing image with consistent color.

[0076] A multi-source remote sensing image radiation normalization device based on frequency domain decomposition includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the following method is implemented:

[0077] Obtain a remote sensing image set, and arbitrarily select two remote sensing images with different radiation fields to form an image pair, one of which is used as a reference image and the other as an image to be processed;

[0078] The image to be processed is converted into an input image matrix using the radiation values ​​of each channel and enhanced to obtain a new feature matrix, wherein the new feature matrix has the same dimension as the input image matrix;

[0079] Performing frequency domain decomposition on the new feature matrix based on each color channel to obtain low-frequency components and high-frequency components, performing multi-layer feature extraction on the low-frequency components and the reference image to obtain image features of each layer respectively;

[0080] Construct an image conversion fusion model, and generate high-frequency image blocks based on the high-frequency components and each layer of image features. Specifically, the image conversion fusion model includes a feature receiving module, a self-attention optimization module, an image conversion module and a splicing module. The feature receiving module receives the image features of each layer. The self-attention optimization module performs serial processing, attention processing and conversion processing on the image features of each layer to obtain an attention image and a transformation feature map. The image conversion module generates and discriminates the transformation feature map and the image features of each layer to obtain a low-frequency image block with the color of the reference image. The splicing module performs feature splicing on the low-frequency image block, the high-frequency component and the attention image to obtain a high-frequency image block.

[0081] The high-frequency image block is fused with a reference image to generate a normalized remote sensing image with consistent color.

[0082] The present invention has significant technical effects due to the adoption of the above technical solutions:

[0083] This chapter proposes a novel pixel-level style transfer model based on generative adversarial networks for radiometric normalization of multi-source images. The core idea of ​​this method is to implement adversarial learning at the image level to transfer the data distribution from the source image domain to the target image domain. This can be used in conjunction with the subsequent semantic segmentation task of unlabeled target image domain images to enhance the key information of the image at the feature level.

[0084] The relative radiometric normalization method for remote sensing images of this invention aligns the data distributions of the source and target image domains. By generating source image domain data with the style of the target image domain, the features of the source image domain data after radiometric migration are more similar to the feature distribution of the target image domain data. This allows the network to pre-learn the data distribution in the target image domain, effectively solving the domain shift problem between remote sensing images and improving the model's generalization ability. In particular, in the task of extracting roads from remote sensing images, the model can more accurately extract the more obvious road features in the target image domain, reducing recognition errors caused by domain shift. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0086] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0087] Figure 2 It is a flow chart of a specific embodiment of the present invention;

[0088] Figure 3 It is a schematic diagram of the process of frequency domain decomposition of the present invention;

[0089] Figure 4 Schematic diagram of the specific process of the self-attention optimization module of the present invention;

[0090] Figure 5-Figure 8 It is a schematic diagram of experimental results of other methods and the method of the present invention;

[0091] Figure 9 It is an overall schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0092] The present invention will be further described in detail below with reference to the examples. The following examples are intended to explain the present invention but the present invention is not limited to the following examples.

[0093] Example 1:

[0094] A radiometric normalization method for multi-source remote sensing images based on frequency domain decomposition, such as Figure 1 As shown, the following steps are included:

[0095] S100, obtaining a remote sensing image set, arbitrarily selecting two remote sensing images with different radiation domains to form an image pair, one of which is used as a reference image and the other as an image to be processed;

[0096] S200, converting the image to be processed into an input image matrix using the radiance values ​​of each channel and performing enhancement processing to obtain a new feature matrix, wherein the new feature matrix has the same dimension as the input image matrix;

[0097] S300, performing frequency domain decomposition on the new feature matrix based on each color channel to obtain low-frequency components and high-frequency components, performing multi-layer feature extraction on the low-frequency components and the reference image to obtain image features of each layer respectively;

[0098] S400, constructing an image conversion fusion model, and generating high-frequency image blocks from high-frequency components and image features of each layer based on the image conversion fusion model, specifically: the image conversion fusion model includes a feature receiving module, a self-attention optimization module, an image conversion module and a splicing module, the feature receiving module receives the image features of each layer, the self-attention optimization module performs serial processing, attention processing and conversion processing on the image features of each layer to obtain an attention image and a transformation feature map, the image conversion module generates and discriminates the transformation feature map and the image features of each layer to obtain a low-frequency image block with the color of the reference image, and the splicing module performs feature splicing on the low-frequency image block, the high-frequency component and the attention image to obtain a high-frequency image block;

[0099] S500: Fusing the high-frequency image block with a reference image to generate a normalized remote sensing image with consistent colors.

[0100] Because the causes of radiometric inconsistencies between multi-source data are numerous and complex, achieving high-quality radiometric normalization requires considering different scenarios during model training and constructing a multi-causal radiometric normalization database. This multi-causal radiometric normalization database is a remote sensing image collection. In addition to collecting real observed image pairs as the database data source, it also increases the data volume by simulating different radiometric images through methods such as PS and data augmentation.

[0101] In addition, the present invention performs frequency domain decomposition and reconstruction to obtain more distinct components, thereby preserving high-frequency texture details and making the edges of image features clear. To explore the importance of spatial radiation information, the channel attention mechanism is improved based on the residual network to screen and enhance stable features.

[0102] In one embodiment, by improving the Haar wavelet transform, the spatial frequency decoupling of the remote sensing image can be achieved, and the mutation and edge information in the image can be effectively extracted to retain its texture features and detail structure. This not only maintains the integrity of the image details, but also enhances the network's ability to effectively learn and extract significant features. Before decomposition, the remote sensing image matrix First, perform point convolution to enhance the nonlinearity of the feature, thereby generating a new feature matrix with unchanged dimension. ; Then, the new feature matrix is ​​decomposed in the frequency domain based on each color channel to obtain low-frequency components and high-frequency components, such as Figure 3 As shown, the specific approach is: perform frequency domain decomposition on each new feature in the new feature matrix through the improved Haar wavelet transform, and decompose each new feature into a low-frequency component and a high-frequency component. The high-frequency component includes a horizontal component, a vertical component, and a diagonal component, which are expressed as follows:

[0103]

[0104] in, , Represent the low-frequency approximation coefficient and high-frequency detail coefficient of the channel respectively, where Respectively represent the row and column numbers of pixels in the image, and their value ranges are , , Represent the height and width of the remote sensing image respectively.

[0105] Next, the encoder performs multi-layer feature extraction on the low-frequency component and the reference image to obtain the image features of each layer. The image features of each layer are expressed as: ,in, Respectively represent the number of channels, the height of the remote sensing image and the width of the remote sensing image, Indicates a feature set with the same number of channels, height and width as the remote sensing image, Represents the image features of each layer, Indicates the number of layers. The high-frequency components, namely the horizontal, vertical and diagonal components, are not processed.

[0106] In one embodiment of the present invention, by combining the global connectivity characteristics of the self-attention mechanism with the ability of the residual network to prevent network degradation, the limitations of the original residual network in extracting features within a small window are improved, the global receptive field of feature extraction is expanded, and the multi-scale invariance is improved. The self-attention optimization module is used to process and transform the image features of each layer to obtain a transformed feature map, such as Figure 4 The specific steps are as follows:

[0107] Concatenate the features of the current layer with the downsampled features of the previous layers to obtain concatenated features;

[0108] pass The convolutional layer and normalization layer perform linear transformation on the concatenated features to obtain query embeddings , key embedding Sum value embedding ;

[0109] Query-based embedding and key embedding , calculate attention, based on the attention and value embedding , obtaining statistics of each distribution, wherein the statistics include an attention-weighted mean and an attention-weighted standard deviation;

[0110] Based on each position and each color channel of the image to be processed, the attention weighted standard deviation S is used as a scaling factor and the attention weighted mean M is used as a shift factor to transform the image to obtain a transformed feature map;

[0111] The series characteristics are expressed as:

[0112] 、 、 The matrices are represented as:

[0113]

[0114]

[0115]

[0116] Attention, expressed as:

[0117] The attention-weighted average is expressed as:

[0118] The attention-weighted standard deviation is expressed as:

[0119] The transformed feature map is expressed as:

[0120] in, represents a bilinear interpolation layer that downsamples the input features to match shape, represents the connection operation along the channel dimension, 、 and yes Convolutional layers, represents channel mean square error normalization, , , Represents element-wise vector product.

[0121] After obtaining the transformation feature map, the transformation feature map and each layer of image features are subjected to generation and discrimination processing by an image conversion module to obtain an image block, wherein the image conversion module includes a first generation and discrimination unit and a second generation and discrimination unit, the first generation and discrimination unit includes a first generator and a first discriminator, and the second generation and discrimination unit includes a second generator and a second discriminator;

[0122] The first generator converts the transformed feature map of the image to be processed into a feature image of the reference image, and the second generator translates the feature image of the reference image into the transformed feature map of the image to be processed;

[0123] The first discriminator discriminates the authenticity of the feature image of the reference image;

[0124] The second discriminator discriminates the authenticity of the transformed feature map of the processed image.

[0125] In one embodiment, the first generator and the second generator have the same structure, and respectively include an encoder and a decoder, the encoder including a convolution layer, a first activation function, a first batch normalization and a pooling layer, and the decoder including a deconvolution layer, a second activation function, a second batch normalization, a skip connection layer and a final output layer;

[0126] The convolution layer performs convolution operations on the transformed feature map of the image to be processed or the feature map of the reference image based on a 3×3 convolution kernel to extract local features respectively;

[0127] The first activation function is the ReLU activation function, which learns complex features and obtains the second local features;

[0128] Based on the first batch normalization, the second local features are normalized. By normalizing the mean and variance of the second local features, the training process is stabilized and the convergence is accelerated.

[0129] Based on the pooling layer, the normalized second local feature is processed to reduce the spatial dimension to obtain a low-dimensional feature map;

[0130] The deconvolution layer of the decoder increases the dimension of the low-dimensional feature map to obtain a normal-resolution feature map;

[0131] The second activation function is the ReLU activation function, which learns complex features and obtains the third feature map;

[0132] Based on the second batch normalization, the third feature map is normalized. By normalizing the mean and variance of the third feature map, the training process is stabilized and the convergence is accelerated;

[0133] Through the skip connection layer, the feature map in the encoder is passed to the corresponding layer of the decoder to retain the detailed information;

[0134] Through the final output layer, the pixel values ​​are normalized to [-1, 1]) or [0, 1] based on the Tanh activation function or the Sigmoid activation function, and the feature image of the reference image or the transformed feature map of the image to be processed with the same size as the transformed feature map of the image to be processed or the feature image of the reference image is output, that is, the low-frequency image block.

[0135] In addition, the first discriminator and the second discriminator have the same structure, respectively including an input layer, a hidden layer and an output layer, and the hidden layer includes a convolution layer, a third activation function and a pooling layer;

[0136] The input layer receives the feature image of the reference image or the transformed feature map of the image to be processed;

[0137] The convolution layer uses a 3×3 convolution kernel to extract features;

[0138] The third activation function is the LeakyReLU activation function, which learns complex features;

[0139] The pooling layer gradually reduces the spatial dimension through pooling operations;

[0140] The output layer uses the Sigmoid activation function to output a probability value between 0 and 1 to judge the authenticity of the feature image of the reference image or the transformed feature map of the image to be processed.

[0141] That is to say, there is currently a reference image. To convert another image to be processed into an image with the same color as the reference image but the same content, that is, a color-consistent normalized remote sensing image, the image to be processed needs to be frequency decomposed, that is, decoupled, and high-frequency texture information, that is, high-frequency components, and low-frequency color information, that is, low-frequency components, that is, radiation information, will be obtained. The purpose of the image conversion and fusion model is to process the low-frequency information. The image conversion and fusion model will enable the image to be processed to learn the color features of the reference image and obtain a low-frequency image block with the color of the reference image. The low-frequency image block, high-frequency component and attention image are then feature-spliced ​​to obtain a high-frequency image block. Finally, the high-frequency image block is fused with the reference image to generate a color-consistent normalized remote sensing image.

[0142] In addition, the performance of the image conversion fusion model is evaluated, including:

[0143] The normalized results of the CycleGAN model are verified and evaluated based on the structural similarity index, peak signal-to-noise ratio, learned perceptual image patch similarity, or spectral angle mapper, and the CycleGAN model is iteratively optimized based on the evaluation results. In this application, the structural similarity index is preferred for evaluation. The structural similarity index is constructed by brightness similarity, contrast similarity, and structural similarity, as follows:

[0144] The brightness similarity is expressed as follows:

[0145]

[0146] Contrast similarity, expressed as follows:

[0147]

[0148] Structural similarity, expressed as follows:

[0149]

[0150] Then the structural similarity index is expressed as follows:

[0151]

[0152] in, Represents images respectively The average value of Represents images respectively The standard deviation of Representing an image The covariance of Usually take , the value range of the structural similarity index is , the closer the value is to 1, the more similar the two images are.

[0153] In the specific experiment, the dataset used is the MtS-WH (Multi-temporal Scene WuHan) dataset, which is a multi-temporal scene change detection dataset released by the SIGMA Laboratory of Wuhan University. It is mainly used for theoretical research and verification of scene change detection methods in remote sensing images. The dataset contains two high-resolution (VHR) remote sensing images, acquired by the IKONOS sensor in February 2002 and June 2009, respectively. The image size is 7200×6000 pixels. After fusion using the Gram-Schmidt algorithm, the spatial resolution is 1 meter and it contains four spectral bands (blue, green, red and near-infrared).

[0154] The dataset is mainly used for scene change detection, that is, detecting and analyzing land use changes in a specific area at the scene semantic level. It is suitable for research in the fields of urban planning, environmental monitoring, disaster assessment, etc. Advantages of the dataset: It provides high-resolution multi-temporal images, which can support detailed analysis of land use changes, contains detailed label information, and is suitable for supervised learning and algorithm verification. The MtS-WH dataset has been used to study a variety of scene change detection algorithms, such as convolutional neural networks (CNN) and generative adversarial networks (GAN) based on deep learning. The multi-temporal characteristics of the dataset make it an important resource for studying change detection in remote sensing images. This dataset is applied to the solution of this application. The specific overall schematic diagram can be found in the attached. Figure 2 shown.

[0155] Table 1 shows the experimental results of this application and other methods

[0156]

[0157] As shown in Table 1, the method of the present application is compared with other methods. The experimental results are shown in Table 1. The method of the present application has the highest accuracy.

[0158] Also see the attached Figure 5 -Attached Figure 8 As shown in the figure, they are (a) original image; (b) reference image; (c) histogram matching; (d) Wallis filtering; (e) DEADiff; (f) CycleGAN; (g) AdaAttN; and (h) the method of the present invention. By comparison, it is found that the method of the present invention can significantly improve the domain offset problem.

[0159] The method of the present invention has achieved significant improvements in performance, reaching optimal results in all evaluation indicators and demonstrating strong robustness and adaptability. In contrast, other methods have varying degrees of deviation in the overall brightness and contrast of the image, affecting the final reconstruction quality. In particular, our method shows significant improvements in the SSIM indicator, which can effectively balance the brightness and contrast of images in different domains, making the generated images more natural and consistent. At the same time, the improvement in the SAM indicator is also particularly prominent, indicating that the method of the present invention can better maintain the similarity of pixel spectral vectors, thereby improving the consistency of spectral information and optimizing the spectral fidelity of the reconstructed image.

[0160] Example 2:

[0161] A multi-source remote sensing image radiometric normalization system based on frequency domain decomposition, such as Figure 9 As shown, including:

[0162] The data acquisition module 100 acquires a remote sensing image set and arbitrarily selects two remote sensing images with different radiation fields to form an image pair, one of which is used as a reference image and the other as an image to be processed;

[0163] The feature decomposition module 200 converts the image to be processed into an input image matrix using the radiance values ​​of each channel and performs enhancement processing to obtain a new feature matrix, wherein the new feature matrix has the same dimension as the input image matrix;

[0164] The feature extraction module 300 performs frequency domain decomposition on the new feature matrix based on each color channel to obtain low-frequency components and high-frequency components, and performs multi-layer feature extraction on the low-frequency components and the reference image to obtain image features for each layer.

[0165] Constructing a fusion module 400, constructing an image conversion fusion model, and generating a high-frequency image block from the high-frequency components and each layer of image features based on the image conversion fusion model. Specifically, the image conversion fusion model includes a feature receiving module, a self-attention optimization module, an image conversion module, and a splicing module. The feature receiving module receives the image features of each layer. The self-attention optimization module performs serial processing, attention processing, and conversion processing on the image features of each layer to obtain an attention image and a transformation feature map. The image conversion module generates and discriminates the transformation feature map and the image features of each layer to obtain a low-frequency image block with the color of the reference image. The splicing module performs feature splicing on the low-frequency image block, the high-frequency component, and the attention image to obtain a high-frequency image block.

[0166] The image fusion module 500 fuses the high-frequency image block with the reference image to generate a normalized remote sensing image with consistent colors.

[0167] Various changes and modifications can be made without departing from the spirit and scope of the present invention, and all equivalent technical solutions also fall within the scope of the present invention.

[0168] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referenced to each other.

[0169] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0173] It should be noted that:

[0174] References in this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, appearances of the phrases "one embodiment" or "an embodiment" in various places throughout this specification do not necessarily refer to the same embodiment.

[0175] Furthermore, it should be noted that the specific embodiments described in this specification may vary in the shapes and names of their components. Any equivalent or simple variations based on the structure, features, and principles described in the patented concept of this invention are included within the scope of protection of this patent. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments, and these modifications, as long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, shall fall within the scope of protection of this invention.

Claims

1. A multi-source remote sensing image radiometric normalization method based on frequency domain decomposition, characterized in that: The following steps are involved: Obtain a remote sensing image set, and arbitrarily select two remote sensing images with different radiation fields to form an image pair, one of which is used as a reference image and the other as an image to be processed; The image to be processed is converted into an input image matrix using the radiation values ​​of each channel and enhanced to obtain a new feature matrix, wherein the new feature matrix has the same dimension as the input image matrix; Performing frequency domain decomposition on the new feature matrix based on each color channel to obtain low-frequency components and high-frequency components, performing multi-layer feature extraction on the low-frequency components and the reference image to obtain image features of each layer respectively; Construct an image conversion fusion model, and generate high-frequency image blocks based on the high-frequency components and each layer of image features. Specifically, the image conversion fusion model includes a feature receiving module, a self-attention optimization module, an image conversion module and a splicing module. The feature receiving module receives the image features of each layer. The self-attention optimization module performs serial processing, attention processing and conversion processing on the image features of each layer to obtain an attention image and a transformation feature map. The image conversion module generates and discriminates the transformation feature map and the image features of each layer to obtain a low-frequency image block with the color of the reference image. The splicing module performs feature splicing on the low-frequency image block, the high-frequency component and the attention image to obtain a high-frequency image block. The high-frequency image block is fused with a reference image to generate a normalized remote sensing image with consistent color.

2. The multi-source remote sensing image radiation normalization method based on frequency domain decomposition according to claim 1 is characterized in that: Through Haar wavelet transform, each new feature in the new feature matrix is ​​decomposed in the frequency domain in combination with each color channel, and each new feature is decomposed into a low-frequency component and a high-frequency component. The high-frequency component includes a horizontal component, a vertical component, and a diagonal component, which are expressed as follows: in, , Represent the low-frequency approximation coefficient and high-frequency detail coefficient of the channel respectively, where Respectively represent the row and column numbers of pixels in the image, and their value ranges are , , Represent the height and width of the remote sensing image respectively.

3. The multi-source remote sensing image radiation normalization method based on frequency domain decomposition according to claim 2 is characterized in that: The encoder performs multi-layer feature extraction on the low-frequency component and the reference image to obtain the image features of each layer. The image features of each layer are expressed as: ,in, Respectively represent the number of channels, the height of the remote sensing image and the width of the remote sensing image, Indicates a feature set with the same number of channels, height and width as the remote sensing image, Represents the image features of each layer, Indicates the number of layers.

4. The multi-source remote sensing image radiation normalization method based on frequency domain decomposition according to claim 1 is characterized in that: The self-attention optimization module performs concatenation processing, attention processing, and conversion processing on each layer of image features to obtain a transformed feature map, including the following steps: Concatenate the image features of the current layer with the downsampled image features of the previous layers to obtain concatenated features; pass The convolutional layer and normalization layer perform linear transformation on the concatenated features to obtain query embedding, key embedding, and value embedding respectively; Calculating attention based on the query embedding and the key embedding, and obtaining statistics for each distribution based on the attention and the value embedding, the statistics including an attention-weighted mean and an attention-weighted standard deviation; Based on each position and each color channel, the attention weighted standard deviation is used as a scaling factor and the attention weighted average is used as a shift factor to transform the image to obtain a transformed feature map; The series characteristics are expressed as: Query embedding, key embedding, and value embedding are represented as: The attention is expressed as: The attention weighted average is expressed as: The attention weighted standard deviation is expressed as: The transformation feature map is expressed as: in, Represents a bilinear interpolation layer that downsamples image features to match shape, represents the connection operation along the channel dimension, 、 and Respectively Convolutional layers, represents channel mean square error normalization, , , represents the element-wise vector product, represents query embedding, Indicates key embedding and represents the value embedding, S represents the attention-weighted standard deviation, M represents the attention-weighted mean, Indicates the number of layers, T indicates transposition, Represents the image features of each layer, Represents a collection, They represent the number of channels, the height of the remote sensing image, and the width of the remote sensing image respectively.

5. The multi-source remote sensing image radiation normalization method based on frequency domain decomposition according to claim 4 is characterized in that: The image conversion module includes a first generation and discrimination unit and a second generation and discrimination unit, the first generation and discrimination unit includes a first generator and a first discriminator, and the second generation and discrimination unit includes a second generator and a second discriminator; The first generator converts the transformed feature map of the image to be processed into a feature image of the reference image, and the second generator translates the feature image of the reference image into the transformed feature map of the image to be processed; The first discriminator discriminates the authenticity of the feature image of the reference image; The second discriminator discriminates the authenticity of the transformed feature map of the processed image.

6. The multi-source remote sensing image radiation normalization method based on frequency domain decomposition according to claim 5 is characterized in that: The first generator and the second generator have the same structure, respectively including an encoder and a decoder, the encoder including a convolution layer, a first activation function, a first batch normalization and a pooling layer, and the decoder including a deconvolution layer, a second activation function, a second batch normalization, a skip connection layer and a final output layer; The convolution layer performs convolution operations on the transformed feature map of the image to be processed or the feature map of the reference image based on a 3×3 convolution kernel to extract local features respectively; The first activation function is the ReLU activation function, which learns complex features and obtains the second local features; Based on the first batch normalization, the second local features are normalized. By normalizing the mean and variance of the second local features, the training process is stabilized and the convergence is accelerated. Based on the pooling layer, the normalized second local feature is processed to reduce the spatial dimension to obtain a low-dimensional feature map; The deconvolution layer of the decoder increases the dimension of the low-dimensional feature map to obtain a normal-resolution feature map; The second activation function is the ReLU activation function, which learns complex features and obtains the third feature map; Based on the second batch normalization, the third feature map is normalized. By normalizing the mean and variance of the third feature map, the training process is stabilized and the convergence is accelerated; Through the skip connection layer, the feature map in the encoder is passed to the corresponding layer of the decoder to retain the detailed information; Through the final output layer, the pixel values ​​are normalized to [-1, 1] or [0, 1] based on the Tanh activation function or the Sigmoid activation function, and the feature image of the reference image or the transformed feature map of the image to be processed with the same size as the transformed feature map of the image to be processed or the feature image of the reference image is output, that is, the low-frequency image block.

7. The multi-source remote sensing image radiometric normalization method based on frequency domain decomposition according to claim 6, characterized in that: The first discriminator and the second discriminator have the same structure, respectively including an input layer, a hidden layer and an output layer, and the hidden layer includes a convolution layer, a third activation function and a pooling layer; The input layer receives the feature image of the reference image or the transformed feature map of the image to be processed; The convolution layer uses a 3×3 convolution kernel to extract features; The third activation function is the LeakyReLU activation function, which learns complex features; The pooling layer gradually reduces the spatial dimension through pooling operations; The output layer uses the Sigmoid activation function to output a probability value between 0 and 1 to judge the authenticity of the feature image of the reference image or the transformed feature map of the image to be processed.

8. The multi-source remote sensing image radiometric normalization method based on frequency domain decomposition according to claim 1, characterized in that: It also includes an evaluation of the performance of the image translation fusion model, including: The normalization results of the CycleGAN model are verified and evaluated based on the structural similarity index, peak signal-to-noise ratio, learned perceptual image patch similarity or spectral angle mapper, and the CycleGAN model is iteratively optimized based on the evaluation results.

9. The multi-source remote sensing image radiation normalization method based on frequency domain decomposition according to claim 8, characterized in that: The structural similarity index is constructed by brightness similarity, contrast similarity and structural similarity, as follows: The brightness similarity is expressed as follows: Contrast similarity, expressed as follows: Structural similarity, expressed as follows: Then the structural similarity index is expressed as follows: in, Represents images respectively The average value of Represents images respectively The standard deviation of Representing an image The covariance of Pick , the value range of the structural similarity index is , the closer the value is to 1, the more similar the two images are.

10. A multi-source remote sensing image radiometric normalization system based on frequency domain decomposition, characterized in that: include: The data acquisition module acquires a remote sensing image set and arbitrarily selects two remote sensing images with different radiation fields to form an image pair, one of which is used as a reference image and the other as an image to be processed; A feature decomposition module converts the image to be processed into an input image matrix using the radiance values ​​of each channel and performs enhancement processing to obtain a new feature matrix, wherein the new feature matrix has the same dimension as the input image matrix; A feature extraction module performs frequency domain decomposition on the new feature matrix based on each color channel to obtain low-frequency components and high-frequency components, and performs multi-layer feature extraction on the low-frequency components and the reference image to obtain image features of each layer; Construct a fusion module, construct an image conversion fusion model, and generate high-frequency image blocks based on the high-frequency components and each layer of image features based on the image conversion fusion model. Specifically, the image conversion fusion model includes a feature receiving module, a self-attention optimization module, an image conversion module and a splicing module. The feature receiving module receives the image features of each layer. The self-attention optimization module performs serial processing, attention processing and conversion processing on the image features of each layer to obtain an attention image and a transformation feature map. The image conversion module generates and discriminates the transformation feature map and the image features of each layer to obtain a low-frequency image block with the color of the reference image. The splicing module performs feature splicing on the low-frequency image block, the high-frequency component and the attention image to obtain a high-frequency image block. The image fusion module fuses the high-frequency image block with the reference image to generate a normalized remote sensing image with consistent color.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A device for normalizing multi-source remote sensing image radiation based on frequency domain decomposition, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images