Diffusion model-based remote sensing image feature optimization and alignment method and device

Abstract

The invention discloses a remote sensing image feature optimization and alignment method and device based on a diffusion model. The method comprises the following steps: acquiring a plurality of remote sensing images with different sources, different time phases and different scales in the same scene and corresponding image information; enabling the plurality of remote sensing images to pass through an image information layer and an image conversion layer to obtain a plurality of converted remote sensing images; constructing a U-Net structure based on a conditional diffusion model; performing feature optimization on the plurality of remote sensing images through the constructed U-Net structure by using the plurality of converted remote sensing images and the corresponding image information, generating new remote sensing images based on the optimized image features, and performing preliminary alignment on the remote sensing images; and carrying out iterative optimization through the constructed U-Net structure based on the new remote sensing image so as to realize final alignment of the remote sensing image in the iterative optimization process. According to the method, optimization and alignment of remote sensing image features in a complex scene are realized, the feature difference between different-source, different-time and different-scale images is solved, and the model performance can be improved under the condition of limited training data.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image feature processing, and in particular to a remote sensing image feature optimization and alignment method and device based on a diffusion model. Background Art

[0002] Remote sensing technology has been widely used in many fields such as military defense, disaster management, and resource exploration due to its large detection range, rich information, short cycle, and high timeliness. However, in applications such as monitoring of abnormal water and soil disturbance areas, remote sensing images are usually affected by a variety of complex factors such as terrain, vegetation, and land use, making image features extremely complex and diverse. This complexity and diversity poses a challenge to the accuracy of remote sensing image feature extraction, and the accuracy of feature extraction has a decisive influence on the efficient processing of massive data and its wide practical application in many fields.

[0003] At present, in order to improve the accuracy of remote sensing image feature extraction, feature optimization and alignment are usually performed. Under current technical conditions, remote sensing image feature optimization and alignment technology mainly faces the following challenges: 1) Due to the limitations of remote sensing image acquisition conditions, there are obvious feature differences between remote sensing images of different sources, different phases, and different scales but in the same scene, which makes it difficult for traditional feature extraction methods to obtain similar features, affecting the alignment accuracy; 2) Remote sensing images in complex scenes contain a large number of detailed features and local differences. Existing methods are difficult to take both into account, especially when processing high-resolution images; 3) There is a lack of specialized large-scale training data sets in the current methods for remote sensing image feature optimization and alignment, and there is a lack of relevant data when training models in urgently needed application fields. Therefore, how to provide model performance and efficiency under limited data conditions, while effectively completing multi-source, multi-phase, and multi-scale remote sensing image feature optimization and alignment in complex scenes, is the primary problem to be solved in this field. Summary of the invention

[0004] In order to solve the problems existing in the feature processing technology of multi-source, multi-phase and multi-scale remote sensing images in complex scenes, the present invention proposes a remote sensing image feature optimization and alignment method and device based on a diffusion model, which can efficiently and accurately realize the feature fusion of multi-source, multi-phase and multi-scale images, and improve the accuracy of remote sensing image feature extraction in complex scenes.

[0005] According to a first aspect of the present invention, a remote sensing image feature optimization and alignment method based on a diffusion model is provided. The method comprises the following steps:

[0006] Obtain multiple remote sensing images and corresponding image information of different sources, times and scales in the same scene;

[0007] Passing a plurality of remote sensing images through an image information layer and an image conversion layer, thereby obtaining a plurality of converted remote sensing images;

[0008] Construct a U-Net structure based on the conditional diffusion model;

[0009] Using multiple converted remote sensing images and corresponding image information, the constructed U-Net structure is used to optimize the features of multiple remote sensing images, generate new remote sensing images based on the optimized image features, and perform preliminary alignment of remote sensing images;

[0010] Based on the new remote sensing image, iterative feature optimization is performed through the constructed U-Net structure to achieve the final alignment of the remote sensing image in the iterative feature optimization process.

[0011] In one embodiment, multiple remote sensing images of different sources, different times and different scales and corresponding image information are obtained for the same scene, including: when the number of remote sensing images is insufficient, new remote sensing images of different sources, different times and different scales are produced by rotating, grayscale adjusting and / or partially occluding existing remote sensing images.

[0012] In one embodiment, the image information includes sensor type, imaging time, and resolution size.

[0013] In one embodiment, the U-net structure includes an encoder, a conditional input unit, an attention mechanism introduction unit, a multi-scale fusion unit and a decoder;

[0014] The encoder is used to perform step-by-step downsampling through convolution and pooling layers of different scales to extract global basic features and local detail features of the converted remote sensing image;

[0015] The condition input unit is used to encode the image information into a condition vector;

[0016] The attention mechanism introduction unit is used to first optimize the global basic features and local detail features through the self-attention mechanism to highlight important information, and then use the cross-attention mechanism to achieve feature fusion between different input sources;

[0017] The multi-scale fusion unit is used to introduce the attention mechanism into the output of the unit and the conditional vector for splicing and fusion to obtain optimized image features;

[0018] The decoder is used to perform step-by-step upsampling through convolution and pooling layers of different scales based on the optimized image features, generate new remote sensing images by optimizing the loss function, and perform image alignment.

[0019] In one embodiment, the loss function in the decoder adopts a composite loss function, including reconstruction loss, feature matching loss and adversarial loss, to ensure the quality of restored features.

[0020] In one embodiment, multiple converted remote sensing images and corresponding image information are used to optimize the features of the multiple remote sensing images through a constructed U-Net structure, and new remote sensing images are generated based on the optimized image features and preliminary alignment of the remote sensing images is performed, which also includes:

[0021] After optimizing the features of multiple remote sensing images through the constructed U-Net structure, the local outlier factor LOF algorithm is used to identify outliers in the optimized feature space that affect the features. Feature points with LOF values ​​significantly higher than the threshold are marked as outliers and corrected by interpolation or local reconstruction. The calculation formula of the LOF value is as follows:

[0022]

[0023] Among them, lrd k is the k-local reachability density, N k (P) is the set of neighborhood points of point P.

[0024] In one embodiment, the remote sensing image feature optimization and alignment method based on the diffusion model further includes:

[0025] For the alignment results, fine configuration correction is performed, including using the local weighted regression method to perform nonlinear transformation on the local area to eliminate unnecessary geometric deviations; the weight w in the local weighted regression method is ij The calculation formula is as follows:

[0026]

[0027] Among them, x i and x j They respectively represent the coordinates of two different pixel points in the local area to be corrected, and h represents the size of the local area.

[0028] In one embodiment, the remote sensing image feature optimization and alignment method based on the diffusion model further includes:

[0029] Evaluate the feature reconstruction quality and alignment accuracy of the new remote sensing images generated after iterative feature optimization;

[0030] Among them, the feature reconstruction quality is evaluated using the peak signal-to-noise ratio PSNR and the structural similarity index SSIM;

[0031] The calculation formula of PSNR is as follows:

[0032]

[0033] Among them, MAX IIt represents the maximum value of image pixel, MSE is the mean square error;

[0034] The calculation formula of SSIM is as follows:

[0035]

[0036] Among them, μ x and μ y are the average brightness values ​​of remote sensing images x and y, respectively, x and σ y are the variances of remote sensing images x and y, σ xy is the covariance of remote sensing image x and remote sensing image y, c1 and c2 are constants;

[0037] The alignment accuracy is evaluated using the root mean square error (RMSE). The calculation formula for RMSE is:

[0038]

[0039] Among them, y i is the actual value, is the predicted value, and n is the number of samples.

[0040] According to a second aspect of the present invention, a remote sensing image feature optimization and alignment device based on a diffusion model using the method described in the first aspect of the present invention is provided. The device comprises:

[0041] The acquisition module is used to obtain multiple remote sensing images of different sources, different times and different scales in the same scene and the corresponding image information;

[0042] A preprocessing module is used to process multiple remote sensing images through an image information layer and an image conversion layer to obtain multiple converted remote sensing images;

[0043] Construction module, used to build the U-Net structure based on the conditional diffusion model;

[0044] The preliminary alignment module is used to optimize the features of multiple remote sensing images using the converted remote sensing images and the corresponding image information through the constructed U-Net structure, generate new remote sensing images based on the optimized image features, and perform preliminary alignment of the remote sensing images;

[0045] The iterative optimization module is used to perform iterative feature optimization based on the new remote sensing image through the constructed U-Net structure to achieve the final alignment of the remote sensing image during the iterative feature optimization process.

[0046] According to a third aspect of the present invention, a terminal is provided, which includes a processor and a storage medium, wherein the storage medium is used to store instructions, and the processor is used to operate according to the instructions to execute the steps of the method according to the first aspect of the present invention.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. Input the preprocessed remote sensing images of different sources, different phases and different scales and their corresponding image information into the constructed U-Net structure based on the conditional diffusion model. The model gradually adds noise through the forward process, and then gradually removes the noise through the reverse process to extract the fusion features of the remote sensing images. In this process, the model can learn the common basic features between heterogeneous images, while retaining their respective detail features to achieve feature fusion. In addition, the model can learn the features of the fused images, and randomly generate simulated remote sensing images of different sources, different phases and different scales during training. This operation allows the model to automatically adapt to images in random and complex scenes.

[0049] 2. By detecting and removing outliers on the optimized features, the local outlier factor (LOF) algorithm is used to identify outliers in the feature space, and feature points with LOF values ​​significantly higher than the threshold are marked as outliers. They are corrected by interpolation or local reconstruction, thereby further improving the quality of feature optimization and alignment, eliminating possible errors and anomalies, and obtaining more accurate and reliable results.

[0050] In summary, the method of the present invention realizes the accurate extraction of heterogeneous, heterogeneous phase and heterogeneous scale image features by introducing conditional information, and at the same time utilizes the generation ability and learning ability of the diffusion model to fuse local detail features and global basic features while randomly generating simulated heterogeneous, heterogeneous phase and heterogeneous scale remote sensing images, thereby improving the adaptability of the model. At the same time, the conditional diffusion model can handle multiple tasks at the same time, and can achieve more comprehensive and efficient remote sensing image feature optimization and alignment. In addition, the present invention utilizes the powerful generation ability and learning ability of the diffusion model for complex scenes, and learns the characteristics and change laws of remote sensing images in complex scenes through the process of gradually adding and removing noise. Through a large amount of training, the model is allowed to automatically generate features randomly, simulate remote sensing images in complex scenes, solve the difficulties encountered by traditional methods in processing multi-source, multi-phase and multi-scale images in complex scenes, and improve the generalization ability of the model. The generation process of the diffusion model can also be regarded as a feature optimization process, which is conducive to extracting more significant and more representative features. In terms of application, the present invention has achieved multi-source, multi-phase, and multi-scale remote sensing image feature optimization and alignment technology in complex scenes with practical value, which significantly improves the efficiency of feature optimization and the accuracy of feature alignment, and provides a basis for the subsequent processing and application of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more intuitively illustrate the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0052] Figure 1 The following is a general flow chart of an embodiment of the present invention.

[0053] Figure 2 This is a U-Net structure framework diagram of an embodiment of the present invention.

[0054] Figure 3 The figure shows the results of one embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The present invention mainly aims at the problems existing in the current remote sensing image feature optimization and alignment methods, such as large differences in multi-source, multi-phase, and multi-scale images in complex scenes, difficulty in extracting detailed features, and insufficient data volume. The present invention combines the conditional diffusion model with remote sensing image analysis to construct a new feature optimization and alignment method and device.

[0057] Example 1

[0058] See also Figure 1-3 This embodiment provides a remote sensing image feature optimization and alignment method based on a diffusion model. The method includes the following steps:

[0059] Step 1: Obtain remote sensing images and corresponding image information of different sources, times and scales in the same scene.

[0060] Among them, when obtaining remote sensing images of different sources, different times and different scales in the same scene, if there are too few images, the same remote sensing image can be rotated, grayscale adjusted, partially blocked and other operations can be performed to artificially produce remote sensing images of different sources, different times and different scales. Preferably, according to the characteristics of different source images, corresponding image information, such as sensor type, imaging time, resolution size, etc., is obtained as input conditions for the conditional diffusion model in subsequent steps.

[0061] Step 2: Pass multiple remote sensing images through the image information layer and the image conversion layer to obtain multiple converted remote sensing images.

[0062] Among them, the image information layer is used to extract various information contained in the image, which can be basic image features such as pixel value, color, texture, shape, etc., or more advanced semantic information such as objects, scenes, emotions, etc. The image conversion layer is used to enhance, transform, and encode the original remote sensing image to facilitate subsequent processing.

[0063] Step 3: Construct a U-Net structure based on the conditional diffusion model.

[0064] Specifically, this step includes constructing a U-Net structure to include: an encoder, a conditional input unit, an attention mechanism introduction unit, a multi-scale fusion unit, and a decoder.

[0065] The encoder is used to extract the global basic features and local detail features of the converted remote sensing image by stepwise downsampling. The encoder may include multiple convolutional layers (used in conjunction with the activation function ReLU) and pooling layers of different scales to gradually reduce the size and number of channels of the feature map, thereby gradually extracting higher-level feature information. Among them, the global basic features usually include features such as color, texture, shape, and spatial layout, and the local detail features usually include features such as edges, corners, key points, and local textures.

[0066] The condition input unit is used to encode the image information into a condition vector.

[0067] The attention mechanism introduction unit is used to first optimize the global basic features and local detail features through the self-attention mechanism to highlight important information, and then use the cross-attention mechanism to achieve feature fusion between different input sources, thereby enhancing the model's ability to process multi-source data.

[0068] The multi-scale fusion unit is used to introduce the attention mechanism into the output of the unit and concatenate and fuse it with the conditional vector to obtain optimized image features.

[0069] The decoder is used to reconstruct the image by gradually upsampling based on the optimized image features, generate a new remote sensing image by optimizing the loss function, and perform image alignment. The decoder may also include multiple convolutional layers and pooling layers. These layers work together to gradually restore the feature image extracted by the encoder part to an image of the same size as the input image.

[0070] In addition, the decoder in this embodiment also constructs a loss function. The loss function adopts a composite form, including reconstruction loss, feature matching loss and adversarial loss, to ensure the quality of restored features.

[0071] The reconstruction loss is used to measure the difference between the reconstructed image and the original image. In this embodiment, the reconstruction loss can be obtained by using mean square error (MSE) or mean absolute error (MAE).

[0072] The feature matching loss is used to measure the difference between the reconstructed image and the original image in the feature space. In this embodiment, the feature matching loss is achieved by extracting a feature map on a convolution layer and calculating the mean square error MSE or distance between the two.

[0073] Adversarial loss is used to introduce an adversarial training mechanism for Generative Adversarial Networks (GANs) to improve the realism and quality of reconstructed images. The specific form of the adversarial loss depends on the GAN variant used.

[0074] The reconstruction loss, feature matching loss, and adversarial loss may have their own corresponding weight coefficients to balance the contributions between different loss terms. These weight coefficients can be determined experimentally to ensure that the composite loss function can guide the generation of high-quality reconstructed images.

[0075] In addition, in the decoder, the remote sensing images can be aligned by performing feature matching and geometric transformation on the reconstructed new remote sensing image and the original remote sensing image.

[0076] Further preferably, the U-Net structure may also include a skip connection unit for directly connecting the feature map of the encoder to the corresponding layer of the decoder so that the decoder can utilize the low-level, high-resolution feature information in the encoder. Such a skip connection unit helps the decoder to better restore the detail information of the image and improve the accuracy of image reconstruction.

[0077] Step 4: Use multiple converted remote sensing images and corresponding image information to optimize the features of multiple remote sensing images through the constructed U-Net structure, generate new remote sensing images based on the optimized image features, and perform preliminary alignment of the remote sensing images.

[0078] Based on the U-Net structure constructed in step three, after the remote sensing image and the corresponding impact information are input into the U-Net structure, the optimized image features can be obtained in the multi-scale fusion module in the U-Net structure, and then a new remote sensing image can be generated in the decoder and image alignment can be performed.

[0079] Preferably, in this step, in order to further improve the quality of feature optimization and alignment, eliminate possible errors and anomalies, and obtain more accurate and reliable results, after optimizing the features of multiple remote sensing images through the constructed U-Net structure, the local outlier factor LOF algorithm can be used to identify outliers in the optimized feature space that affect the features, and mark the feature points whose LOF values ​​are significantly higher than the threshold as abnormalities, and correct them by interpolation or local reconstruction methods; wherein the calculation formula of the LOF value is as follows:

[0080]

[0081] Among them, lrd k is the k-local reachability density, N k (P) is the set of neighborhood points of point P.

[0082] Further preferably, for the alignment result, a fine configuration correction can be implemented, including using a local weighted regression method to perform a nonlinear transformation on the local area to eliminate redundant geometric deviations; wherein the weight w in the local weighted regression method is ij The calculation formula is as follows:

[0083]

[0084] Among them, x i and x j They respectively represent the coordinates of two different pixel points in the local area to be corrected, and h represents the size of the local area.

[0085] Step 5: Perform iterative feature optimization based on the new remote sensing image through the constructed U-Net structure to achieve the final alignment of the remote sensing image during the iterative optimization process.

[0086] Specifically, this step can be achieved by repeatedly executing step 4. The difference from step 4 is that in step 5, the new remote sensing image is input into the constructed U-Net structure for feature optimization, and an updated remote sensing image is generated based on the optimized image features to further align the remote sensing images. In addition, referring to step 4, after feature optimization, the local outlier factor LOF algorithm is still used to identify outliers in the optimized feature space that affect the features, and feature points with LOF values ​​significantly higher than the threshold are marked as outliers, and corrected by interpolation or local reconstruction methods.

[0087] In step 5, the optimization and alignment process of image features are carried out interactively, and the optimized features are continuously improved through multiple iterations for alignment operations. This integrated method fully utilizes the advantages of the conditional diffusion model and can simultaneously improve feature optimization efficiency and alignment accuracy.

[0088] Step 6: Evaluate the U-Net structure based on the conditional diffusion model after iterative optimization.

[0089] This evaluation focuses on indicators such as feature reconstruction quality, alignment accuracy, and computational efficiency. For feature reconstruction quality, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used for evaluation. The calculation formula of PSNR is as follows:

[0090]

[0091] Among them, MAX I It represents the maximum value of the image pixel, and MSE is the mean square error. The formula of SSIM is as follows:

[0092]

[0093] Among them, μ x and μ y are the average values ​​of images x and y, respectively, x and σ y are their variances, σ xy is their covariance, c1 and c2 are constants. When evaluating the alignment accuracy, the root mean square error (RMSE) is used, and its calculation formula is:

[0094]

[0095] Among them, y i is the actual value, is the predicted value, and n is the number of samples. In addition, computational efficiency is also considered, including model inference time and memory usage. In terms of qualitative analysis, the original image, optimized feature map, and alignment results are compared through visualization methods to intuitively display the effect of the model. At the same time, the stability and robustness of the model are tested, and experiments are conducted on remote sensing images of different types and resolutions to evaluate the generalization ability of the model. By conducting comparative experiments with existing image feature optimization and alignment methods, the advantages and potential room for improvement of this method can be fully analyzed.

[0096] In this embodiment, multiple remote sensing images of different sources, different phases and different scales after pretreatment and their corresponding image information are input into the constructed U-Net structure based on the conditional diffusion model. The model gradually adds noise through the forward process, and then gradually removes the noise through the reverse process to extract the fusion features of the remote sensing image. In this process, the model can learn the common basic features between heterogeneous images, while retaining their respective detail features to achieve feature fusion. In addition, the model can learn the image features after fusion, and will randomly generate simulated heterogeneous, heterogeneous, and heterogeneous remote sensing images during training. This operation allows the model to automatically adapt to images in random complex scenes. In order to further improve the quality of feature optimization and alignment, eliminate possible errors and anomalies, and obtain more accurate and reliable results, the present invention performs outlier detection and removal on the optimized features, uses the local anomaly factor (LOF) algorithm to identify outliers in the feature space, marks the feature points whose LOF values ​​are significantly higher than the threshold as abnormal, and corrects them by interpolation or local reconstruction methods.

[0097] By using the optimized remote sensing image features and alignment results in the present invention, various remote sensing applications can be carried out, such as change detection, target recognition, and object classification. Through comparative experiments with traditional methods, the effectiveness and superiority of this method in different application scenarios are verified. At the same time, according to different application requirements, corresponding feature extraction and analysis strategies are designed to give full play to the advantages of the optimized features of the diffusion model, greatly improving the accuracy and efficiency of remote sensing applications.

[0098] Example 2.

[0099] The present invention provides a remote sensing image feature optimization and alignment device based on a diffusion model using the method described in Example 1. The device comprises:

[0100] The acquisition module is used to obtain multiple remote sensing images of different sources, different times and different scales in the same scene and the corresponding image information;

[0101] A preprocessing module is used to process multiple remote sensing images through an image information layer and an image conversion layer to obtain multiple converted remote sensing images;

[0102] Construction module, used to build the U-Net structure based on the conditional diffusion model;

[0103] The preliminary alignment module is used to optimize the features of multiple remote sensing images using the converted remote sensing images and the corresponding image information through the constructed U-Net structure, generate new remote sensing images based on the optimized image features, and perform preliminary alignment of the remote sensing images;

[0104] The iterative optimization module is used to perform iterative optimization based on the new remote sensing image through the constructed U-Net structure to achieve the final alignment of the remote sensing image during the iterative optimization process.

[0105] Preferably, the acquisition module is also used to: when the number of remote sensing images is insufficient, produce new remote sensing images of different sources, different times and different scales by rotating, adjusting grayscale and / or partially blocking existing remote sensing images.

[0106] Preferably, the image information includes sensor type, imaging time, and resolution size.

[0107] Preferably, the building module is also used to construct a U-net structure including an encoder, a conditional input unit, an attention mechanism introduction unit, a multi-scale fusion unit and a decoder;

[0108] The encoder is used to extract global basic features and local detail features of the converted remote sensing image by stepwise downsampling;

[0109] The condition input unit is used to encode the image information into a condition vector;

[0110] The attention mechanism introduction unit is used to first optimize the global basic features and local detail features through the self-attention mechanism to highlight important information, and then use the cross-attention mechanism to achieve feature fusion between different input sources;

[0111] The multi-scale fusion unit is used to introduce the attention mechanism into the output of the unit and the conditional vector for splicing and fusion to obtain optimized image features;

[0112] The decoder is used to perform image reconstruction by stepwise upsampling based on the optimized image features, generate new remote sensing images by optimizing the loss function, and perform image alignment.

[0113] Preferably, the loss function in the decoder adopts a composite loss function, including reconstruction loss, feature matching loss and adversarial loss, to ensure the quality of restored features.

[0114] Preferably, the preliminary alignment module is further used for:

[0115] After optimizing the features of multiple remote sensing images through the constructed U-Net structure, the local outlier factor LOF algorithm is used to identify outliers in the optimized feature space that affect the features. Feature points with LOF values ​​significantly higher than the threshold are marked as outliers and corrected by interpolation or local reconstruction. The calculation formula of the LOF value is as follows:

[0116]

[0117] Among them, lrd k是 k-local reachability density, N k (P) is the set of neighborhood points of point P.

[0118] Preferably, the device further comprises an alignment correction module. The alignment correction module is used for:

[0119] For the alignment results, fine configuration correction is performed, including using the local weighted regression method to perform nonlinear transformation on the local area to eliminate unnecessary geometric deviations; the weight w in the local weighted regression method is ij The calculation formula is as follows:

[0120]

[0121] Among them, x i and x j They respectively represent the coordinates of two different pixel points in the local area to be corrected, and h represents the size of the local area.

[0122] Preferably, the device further comprises an evaluation module. The evaluation module is used for:

[0123] Evaluate the iteratively optimized U-Net structure based on the conditional diffusion model; the evaluation includes evaluating the feature reconstruction quality and alignment accuracy of the new remote sensing image generated after the iterative optimization;

[0124] Among them, the feature reconstruction quality is evaluated using the peak signal-to-noise ratio PSNR and the structural similarity index SSIM;

[0125] The calculation formula of PSNR is as follows:

[0126]

[0127] Among them, MAX I It represents the maximum value of image pixel, MSE is the mean square error;

[0128] The calculation formula of SSIM is as follows:

[0129]

[0130] Among them, μ x and μ y are the average brightness values ​​of remote sensing images x and y, respectively, x and σ y are the variances of remote sensing images x and y, σ xy is the covariance of remote sensing image x and remote sensing image y, c1 and c2 are constants;

[0131] The alignment accuracy is evaluated using the root mean square error (RMSE). The calculation formula for RMSE is:

[0132]

[0133] Among them, y i is the actual value, is the predicted value, and n is the number of samples.

[0134] Example 3

[0135] This embodiment provides a terminal, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the remote sensing image feature optimization and alignment method based on the diffusion model as described in Example 1 of the present invention are implemented.

[0136] The detailed steps are the same as those of the remote sensing image feature optimization and alignment method based on the diffusion model provided in Example 1, and will not be repeated here.

[0137] The beneficial effect of the present invention is that, compared with the prior art,

[0138] 1. Input the preprocessed remote sensing images of different sources, different phases and different scales and their corresponding image information into the constructed U-Net structure based on the conditional diffusion model. The model gradually adds noise through the forward process, and then gradually removes the noise through the reverse process to extract the fusion features of the remote sensing images. In this process, the model can learn the common basic features between heterogeneous images, while retaining their respective detail features to achieve feature fusion. In addition, the model can learn the features of the fused images, and randomly generate simulated remote sensing images of different sources, different phases and different scales during training. This operation allows the model to automatically adapt to images in random and complex scenes.

[0139] 2. By detecting and removing outliers on the optimized features, the local outlier factor (LOF) algorithm is used to identify outliers in the feature space, and feature points with LOF values ​​significantly higher than the threshold are marked as outliers. They are corrected by interpolation or local reconstruction, thereby further improving the quality of feature optimization and alignment, eliminating possible errors and anomalies, and obtaining more accurate and reliable results.

[0140] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0141] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0142] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0143] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A remote sensing image feature optimization and alignment method based on a diffusion model, characterized in that: The following steps are involved: Obtain multiple remote sensing images and corresponding image information of different sources, times and scales in the same scene; Passing a plurality of remote sensing images through an image information layer and an image conversion layer, thereby obtaining a plurality of converted remote sensing images; Construct a U-Net structure based on the conditional diffusion model; Using multiple converted remote sensing images and corresponding image information, the constructed U-Net structure is used to optimize the features of multiple remote sensing images, generate new remote sensing images based on the optimized image features, and perform preliminary alignment of remote sensing images; Based on the new remote sensing image, iterative feature optimization is performed through the constructed U-Net structure to achieve the final alignment of the remote sensing image in the iterative feature optimization process.

2. The remote sensing image feature optimization and alignment method based on diffusion model according to claim 1, characterized in that: The method of obtaining multiple remote sensing images of different sources, different times and different scales in the same scene and corresponding image information includes: when the number of remote sensing images is insufficient, rotating, grayscale adjusting and / or partially blocking the existing remote sensing images to produce new remote sensing images of different sources, different times and different scales.

3. The remote sensing image feature optimization and alignment method based on diffusion model according to claim 1, characterized in that: The image information includes sensor type, imaging time, and resolution size.

4. The remote sensing image feature optimization and alignment method based on diffusion model according to claim 1, characterized in that: The construction of the U-Net structure based on the conditional diffusion model includes: The U-net structure is constructed to include an encoder, a conditional input unit, an attention mechanism introduction unit, a multi-scale fusion unit, and a decoder; The encoder is used to extract global basic features and local detail features of the converted remote sensing image by stepwise downsampling; The condition input unit is used to encode the image information into a condition vector; The attention mechanism introduction unit is used to first optimize the global basic features and local detail features through the self-attention mechanism to highlight important information, and then use the cross-attention mechanism to achieve feature fusion between different input sources; The multi-scale fusion unit is used to introduce the attention mechanism into the output of the unit and the conditional vector for splicing and fusion to obtain optimized image features; The decoder is used to perform image reconstruction by stepwise upsampling based on the optimized image features, generate new remote sensing images by optimizing the loss function, and perform image alignment.

5. The remote sensing image feature optimization and alignment method based on diffusion model according to claim 4, characterized in that: The loss function in the decoder adopts a composite loss function, including reconstruction loss, feature matching loss and adversarial loss, to ensure the quality of restored features.

6. The remote sensing image feature optimization and alignment method based on diffusion model according to claim 1, characterized in that: The method uses a plurality of converted remote sensing images and corresponding image information to optimize the features of the plurality of remote sensing images through a constructed U-Net structure, generates a new remote sensing image based on the optimized image features, and performs preliminary alignment of the remote sensing images, and also includes: After optimizing the features of multiple remote sensing images through the constructed U-Net structure, the local outlier factor LOF algorithm is used to identify outliers in the optimized feature space that affect the features. Feature points with LOF values ​​significantly higher than the threshold are marked as outliers and corrected by interpolation or local reconstruction. The calculation formula of the LOF value is as follows: Among them, lrd k is the k-local reachability density, N k (P) is the set of neighborhood points of point P.

7. The remote sensing image feature optimization and alignment method based on diffusion model according to claim 1, characterized in that: Also includes: For the alignment results, fine configuration correction is performed, including using the local weighted regression method to perform nonlinear transformation on the local area to eliminate unnecessary geometric deviations; the weight w in the local weighted regression method is ij The calculation formula is as follows: Among them, x i and x j They respectively represent the coordinates of two different pixel points in the local area to be corrected, and h represents the size of the local area.

8. The remote sensing image feature optimization and alignment method based on diffusion model according to claim 1, characterized in that: Also includes: Evaluating the U-Net structure based on the conditional diffusion model after iterative feature optimization; the evaluation includes evaluating the feature reconstruction quality and alignment accuracy of the new remote sensing image generated after iterative feature optimization; Among them, the feature reconstruction quality is evaluated using the peak signal-to-noise ratio PSNR and the structural similarity index SSIM; The calculation formula of PSNR is as follows: Among them, MAX I It represents the maximum value of image pixel, MSE is the mean square error; The calculation formula of SSIM is as follows: Among them, μ x and μ y are the average brightness values ​​of remote sensing images x and y, respectively, x and σ y are the variances of remote sensing images x and y, σ xy is the covariance of remote sensing image x and remote sensing image y, c1 and c2 are constants; The alignment accuracy is evaluated using the root mean square error (RMSE). The calculation formula for RMSE is: Among them, y i is the actual value, is the predicted value, and n is the number of samples.

9. A remote sensing image feature optimization and alignment device based on a diffusion model using the method according to any one of claims 1 to 8, characterized in that: include: The acquisition module is used to obtain multiple remote sensing images of different sources, different times and different scales in the same scene and the corresponding image information; A preprocessing module is used to process multiple remote sensing images through an image information layer and an image conversion layer to obtain multiple converted remote sensing images; Construction module, used to build the U-Net structure based on the conditional diffusion model; The preliminary alignment module is used to optimize the features of multiple remote sensing images using the converted remote sensing images and the corresponding image information through the constructed U-Net structure, generate new remote sensing images based on the optimized image features, and perform preliminary alignment of the remote sensing images; The iterative optimization module is used to perform iterative feature optimization based on the new remote sensing image through the constructed U-Net structure to achieve the final alignment of the remote sensing image during the iterative feature optimization process.

10. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-8.

Images