Image fusion method, device, storage medium and program product

Abstract

The present application discloses an image fusion method, device, storage medium and program product, and relates to the field of computer vision technology. The present application can distinguish between unchanged areas and changed areas for differential processing by screening the changed areas between a first rough image at a reference moment and a second rough image at a time to be predicted, so that the spatial information of the second rough image can be subsequently supplemented according to the unchanged areas with smaller feature differences, and the changed areas with larger feature differences can be prevented from interfering with the information supplement; the first fine image is processed by a changing binary mask, so that the feature response capability of the unchanged areas can be enhanced and the feature response capability of the changed areas can be weakened, so that the unchanged areas can be focused on during subsequent feature fusion; the fusion result is image repaired based on the changing binary mask, so that the changed areas can be image repaired, and the accuracy of image fusion is improved by combining the spatial information supplement of the unchanged areas in the previous step.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to image fusion methods, devices, storage media, and program products. Background Art

[0002] Remote sensing imagery is capable of acquiring large-scale, multi-dimensional observational information on the Earth's surface, and therefore plays an increasingly important role in various observation tasks. Currently, due to technological limitations, it is difficult to obtain remote sensing imagery with both high temporal and spatial resolutions. Specifically, high-temporal-resolution imagery has a high observation frequency, providing continuous, intensive surface observation data in a short period of time, but has a lower spatial resolution. High-spatial-resolution imagery can provide more detailed information on the structure of land objects, but is acquired less frequently, making it difficult to meet the needs of scenarios requiring high temporal continuity.

[0003] Related technologies use spatiotemporal fusion methods to fuse high-temporal-resolution images with high-spatial-resolution images to generate remote sensing images with both high temporal and high spatial resolutions. However, due to the uncertainty of the time interval between the high-temporal-resolution and high-spatial-resolution images, as well as the difference in spatial resolution between the two, spatiotemporal fusion results in low accuracy. Summary of the Invention

[0004] The present application provides an image fusion method, device, storage medium and program product to at least solve the problem of low accuracy when performing spatiotemporal fusion of remote sensing images.

[0005] The present application provides an image fusion method, which includes:

[0006] Acquire a first coarse image at a reference moment, a first fine image at the reference moment, and a second coarse image at a time to be predicted; wherein the spatial resolution of the first fine image is higher than the spatial resolution of the first coarse image;

[0007] screening a change region between the first coarse image and the second coarse image to generate a change binary mask;

[0008] Strengthening features of unchanged regions and suppressing features of changed regions of the first fine image using the changed binary mask to obtain an enhanced first fine image;

[0009] performing feature fusion on the second coarse image and the enhanced first fine image to obtain a fusion result;

[0010] Image restoration is performed on the fusion result based on the change binary mask to obtain a fine image prediction result for the time to be predicted.

[0011] The present application also provides an image fusion device, comprising:

[0012] An image acquisition module, configured to acquire a first coarse image at a reference moment, a first fine image at a reference moment, and a second coarse image at a time to be predicted; wherein the spatial resolution of the first fine image is higher than the spatial resolution of the first coarse image;

[0013] a screening module, configured to screen a change region between the first coarse image and the second coarse image to generate a change binary mask;

[0014] an enhancement module, configured to enhance features of unchanged regions and suppress features of changed regions of the first fine image using the changed binary mask to obtain an enhanced first fine image;

[0015] a fusion module, configured to perform feature fusion on the second coarse image and the enhanced first fine image to obtain a fusion result;

[0016] The restoration module is used to perform image restoration on the fusion result based on the change binary mask to obtain a fine image prediction result at the time to be predicted.

[0017] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned image fusion methods when executing the computer program.

[0018] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned image fusion methods are implemented.

[0019] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned image fusion methods when executed by a processor.

[0020] Through the present application, by screening the change area between the first coarse image at the reference moment and the second coarse image at the time to be predicted, a change binary mask is generated, which can distinguish the unchanged area and the change area for differential processing, so that the spatial information of the second coarse image can be supplemented according to the unchanged area with smaller feature differences, and the change area with larger feature differences can be prevented from interfering with the information supplement, thereby improving the accuracy of subsequent image fusion; by using the change binary mask to strengthen the features of the unchanged area and suppress the features of the change area of ​​the first fine image, the feature response ability of the unchanged area can be enhanced and the feature response ability of the change area can be weakened, so that the focus can be placed on the unchanged area during subsequent feature fusion, further improving the accuracy of subsequent image fusion; by performing image restoration on the fusion result based on the change binary mask, the image restoration can be performed on the change area, and combined with the spatial information supplement of the unchanged area in the previous step, a fine image prediction result of the time to be predicted with both spatial details and temporal consistency is obtained, further improving the accuracy of image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A schematic diagram of a flow chart of an image fusion method provided by an embodiment of the present invention;

[0023] Figure 2 A schematic flow chart of another image fusion method provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of a remote sensing image spatiotemporal fusion model provided by an embodiment of the present invention;

[0025] Figure 4 A schematic structural diagram of a feature mask modulator provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the structure of a feature fusion encoder provided in an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of the structure of an image fine restoration module provided by an embodiment of the present invention;

[0028] Figure 7 An example diagram of prediction results of image fusion using different reference images provided by an embodiment of the present invention;

[0029] Figure 8An example diagram comparing prediction results of different image fusion methods provided by an embodiment of the present invention;

[0030] Figure 9 This is a structural block diagram of an image fusion device according to an embodiment of the present invention;

[0031] Figure 10 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0033] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0034] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0035] Remote sensing imagery has the ability to capture large-scale, multi-dimensional observations of the Earth's surface and construct dense time series data. Consequently, it plays an increasingly important role in various observation tasks, such as agricultural monitoring, forestry management, and ecological and environmental assessments. With the development of intelligent technology, the quality requirements for remote sensing images are gradually increasing. In particular, high-quality remote sensing imagery has become a key factor in improving model performance in tasks such as change detection, object classification, and target segmentation. However, due to the limitations of current remote sensing systems in sensor design, satellite payload capacity, launch and operating costs, data storage, and transmission efficiency, a single remote sensing platform struggles to acquire remote sensing imagery with both high temporal and high spatial resolution. Therefore, related technologies use multi-source remote sensing imagery for information complementation to overcome the conflict between different resolutions. Specifically, images acquired by satellites with a short revisit period but low spatial resolution are generally considered to be high temporal resolution images, which have a high observation frequency and can provide continuous and intensive surface observation data in a short period of time; while images acquired by satellites with a long revisit period but high spatial resolution are considered to be high spatial resolution images, which can provide more detailed information on the structure of land objects, but the acquisition frequency within an observation period is low, which makes it difficult to meet the requirements of scenarios with high temporal continuity.

[0036] In related technologies, spatiotemporal fusion methods of remote sensing images are used. The dynamic change information contained in high temporal resolution images is used to compensate for the sparsity of high spatial resolution images in the temporal dimension. High temporal resolution images are fused with high spatial resolution images to reconstruct a high spatial resolution remote sensing image sequence with a complete time series. Remote sensing images with both high temporal resolution and high spatial resolution are generated, effectively improving the comprehensive quality of remote sensing data in both spatial and temporal dimensions, and providing clearer, continuous, and complete information support for surface observation tasks. In addition, in scenes involving historical remote sensing images, there is also the problem of insufficient temporal resolution or spatial resolution. By jointly modeling and reconstructing remote sensing images from different times or different sensors, the missing information can be effectively supplemented.

[0037] Image super-resolution technology can directly improve the spatial resolution of images and has been widely used in fields such as computer vision, remote sensing, and medical imaging. However, traditional super-resolution methods typically target scenes with resolution magnification factors of 2 to 8 times, making them inadequate for the larger scale differences found in remote sensing spatiotemporal fusion tasks. For example, the spatial resolution between different satellites can vary by a factor of 16, far exceeding the processing capabilities of traditional super-resolution methods. In remote sensing image spatiotemporal fusion tasks, the goal is to generate a high-spatial-resolution image at the target moment. Since only low-spatial-resolution images are typically available at the target moment, and these images differ significantly from the available high-spatial-resolution images, direct super-resolution reconstruction is difficult. Therefore, high-spatial-resolution images acquired at other times are used as reference images. These images are then combined with the high-temporal-resolution images corresponding to the target moment, and reconstruction is performed through fusion. This completes the information in both spatial and temporal dimensions, ultimately generating a high-spatial-resolution remote sensing image at the target moment.

[0038] The development of deep learning technologies, such as fully connected neural networks, has significantly improved the ability of models to extract and reconstruct remote sensing image features. Deep learning-based spatiotemporal fusion methods for remote sensing images jointly model a high-temporal-resolution image at the target moment and a high-spatial-resolution image at the reference moment. Leveraging the nonlinear mapping capabilities of deep networks, they directly generate a high-spatial-resolution image at the target moment.

[0039] However, due to the uncertainty of the time interval between the high temporal resolution image and the high spatial resolution image used as the reference image, significant phenological changes and landform evolution may occur within a long time interval, and even extreme natural events such as floods and droughts may occur, causing the spatial structure information contained in the reference image to deviate from the target time, making it difficult for the model to extract accurate features from the reference image to restore the target image, resulting in accumulated prediction errors, seriously affecting the spatiotemporal consistency and true expression ability of the fused image; moreover, there are also differences in spatial resolution between the two, making it difficult for the model to directly and accurately model this mapping relationship, resulting in low accuracy when performing spatiotemporal fusion.

[0040] The embodiment of the present application provides an image fusion method, Figure 1 FIG. 1 is a flow chart of an image fusion method provided by an embodiment of the present invention. Figure 1 As shown, the process includes the following steps.

[0041] The specific steps are as follows:

[0042] Step S101 : acquiring a first coarse image at a reference moment, a first fine image at a reference moment, and a second coarse image at a time to be predicted.

[0043] The time to be predicted is the target time of the remote sensing image spatiotemporal fusion task, and the second coarse image is the target image of the remote sensing image spatiotemporal fusion task. The second coarse image is a high-temporal-resolution image at the time to be predicted, but lacks high-spatial resolution. The first coarse image at the reference time and the first fine image at the reference time serve as reference images for the remote sensing image spatiotemporal fusion task. The first coarse image at the reference time can be a pre-collected high-spatial-resolution image at the reference time, and the first fine image at the reference time can be a pre-collected high-spatial-resolution image at the reference time, where the spatial resolution of the first fine image is higher than that of the first coarse image. This solution treats the second coarse image at the time to be predicted as a fine image with missing information, compares the difference with the first coarse image, extracts change information, and then extracts corresponding detailed spatial information from the first fine image corresponding to the first coarse image and fuses them together to supplement the missing spatial information of the second coarse image and restore details, obtaining a prediction result for the fine image corresponding to the second coarse image, thereby achieving spatiotemporal fusion of remote sensing images.

[0044] Step S102 : screening the changed region between the first coarse image and the second coarse image to generate a changed binary mask.

[0045] The first and second coarse images are compared, and change information is extracted from the high-temporal-resolution image at the reference time and the high-temporal-resolution image at the predicted time. The changing regions between the first and second coarse images are screened out, while the unchanged regions between the first and second coarse images are also screened out. Unchanged regions have smaller feature differences and can serve as a reference for subsequent spatial information completion. However, changed regions have larger feature differences and can interfere with subsequent spatial information completion. By distinguishing between unchanged and changed regions, a differentiated strategy can be adopted in subsequent steps to improve the accuracy and stability of image fusion.

[0046] The change binary mask is used to mark the change area and the unchanged area between the first rough image and the second rough image, for example, marking the change area with a first value and marking the unchanged area with a second value, so as to facilitate the distinction between the change area and the unchanged area in subsequent steps.

[0047] Step S103 , enhancing features of unchanged regions and suppressing features of changed regions of the first fine image using the changed binary mask to obtain an enhanced first fine image.

[0048] The changed binary mask is used to distinguish the changed areas and unchanged areas of the image, and then the features of the unchanged areas corresponding to the first fine image are enhanced to improve the feature response ability of the unchanged areas. The features of the changed areas corresponding to the first fine image are suppressed to weaken the feature response ability of the changed areas. This allows subsequent feature fusion to focus on texture recovery in the unchanged areas, thereby improving the accuracy of texture reconstruction.

[0049] Step S104: performing feature fusion on the second coarse image and the enhanced first fine image to obtain a fusion result.

[0050] The second coarse image at the prediction moment and the enhanced first fine image at the reference moment are subjected to feature fusion. The missing spatial information in the invariant area corresponding to the second coarse image is supplemented and the texture is restored by using the spatial features in the first fine image to obtain a fusion result.

[0051] Step S105 , performing image restoration on the fusion result based on the changed binary mask to obtain a fine image prediction result at the time to be predicted.

[0052] By varying the binary mask, the changed regions of the second coarse image are distinguished and image restoration is performed on these changed regions to fill in the content of the changed regions. Combined with the spatial information completion of the unchanged regions in the previous step, a fine image prediction result with both spatial detail and temporal consistency is obtained for the predicted moment, completing the spatiotemporal fusion task of remote sensing imagery. This can be achieved by employing image restoration techniques from related technologies. Image restoration techniques can automatically infer and generate image content with reasonable structure and continuous texture based on the contextual environment to achieve content filling.

[0053] The image fusion method provided in this embodiment generates a change binary mask by screening the change area between the first coarse image at the reference moment and the second coarse image at the time to be predicted. This can distinguish between the unchanged area and the changed area for differential processing, so that the spatial information of the second coarse image can be subsequently supplemented based on the unchanged area with smaller feature differences, and the changed area with larger feature differences can be prevented from interfering with the information supplement, thereby improving the accuracy of subsequent image fusion; the unchanged area features are enhanced and the changed area features are suppressed in the first fine image through the change binary mask, which can enhance the feature response capability of the unchanged area and weaken the feature response capability of the changed area, so that the feature fusion can be focused on the unchanged area during subsequent feature fusion, further improving the accuracy of subsequent image fusion; by performing image restoration on the fusion result based on the change binary mask, the changed area can be restored. Combined with the spatial information restoration of the unchanged area in the previous step, a fine image prediction result for the time to be predicted with both spatial details and temporal consistency is obtained, further improving the accuracy of image fusion.

[0054] In this embodiment, an image fusion method is provided. Figure 2 FIG. 1 is a flow chart of another image fusion method provided by an embodiment of the present invention. Figure 2 As shown, the process includes the following steps.

[0055] The specific steps are as follows:

[0056] Step S201 : obtaining a first coarse image at a reference moment, a first fine image at a reference moment, and a second coarse image at a time to be predicted.

[0057] The spatial resolution of the first fine image is higher than that of the first coarse image.

[0058] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0059] For example, obtain the reference time The first rough image , reference time The first fine image and the time to be predicted The second rough image The goal of this scheme is to treat the prediction time Fine image of Make predictions and get fine image prediction results .

[0060] Step S202 : screening the changed region between the first coarse image and the second coarse image to generate a changed binary mask.

[0061] First, the first rough image and the second rough image are subtracted to obtain an intermediate image. Since remote sensing images usually contain multiple channels, the intermediate image also includes multiple channels. Since different channels correspond to different spectral information, the corresponding change areas are also different. Therefore, a change binary mask is generated for the intermediate image of each channel, and the binary image of the change area is identified. The change area and the unchanged area are screened out to guide the differential processing of the change area and the unchanged area in the subsequent fusion process. For example, the first rough image and the second coarse image Subtract and get the intermediate image .

[0062] Next, specifically, an averaging operation and an interquartile range calculation are performed on the intermediate image of each channel to generate an upper threshold value and a lower threshold value for change determination corresponding to each channel.

[0063] Optionally, when performing averaging operations and interquartile range calculations on the intermediate images of each channel respectively, first perform an averaging operation on the intermediate images of each channel to obtain the average result of each channel, then perform interquartile range calculation on the intermediate images of each channel to obtain the interquartile range calculation result of each channel, and then obtain the upper threshold value for change determination of each channel based on the sum of the average result and the interquartile range calculation result of each channel, and obtain the lower threshold value for change determination of each channel based on the difference between the average result and the interquartile range calculation result of each channel.

[0064] For example, Taking the channel as an example, find the upper threshold value for change judgment and change judgment threshold The formula is as follows:

[0065]

[0066]

[0067] in, Refers to the intermediate image No. Channels of image data; Indicates averaging operation; Represents the interquartile range calculation, specifically the difference between the third quartile (the 25th percentile of the value) and the first quartile (the bottom 25th percentile of the value).

[0068] Finally, the intermediate image of each channel is compared with the change judgment upper threshold and change judgment lower threshold corresponding to each channel to generate the change binary mask of each channel.

[0069] Optionally, when generating a binary change mask for each channel, the pixel values ​​of each spatial coordinate in the intermediate image of each channel are compared with the upper and lower change determination thresholds corresponding to each channel. If the pixel value of the target spatial coordinate is greater than the upper change determination threshold or the pixel value of the target spatial coordinate is less than the lower change determination threshold, the binary change mask corresponding to the target spatial coordinate is determined to be 1; otherwise, the binary change mask corresponding to the target spatial coordinate is 0; the target spatial coordinate is any one of the spatial coordinates. Screening out significantly changed regions by adaptive thresholding can guide differentiated processing of changed and unchanged regions in the subsequent fusion process.

[0070] For example, the change binary mask is calculated The formula is:

[0071]

[0072] in, represents the spatial coordinates in the image, Indicates the aisle The change binary mask corresponding to the coordinates.

[0073] For example, a remote sensing image spatiotemporal fusion model is constructed. Figure 3 This is a schematic diagram of the structure of the remote sensing image spatiotemporal fusion model provided by the embodiment of the present invention. Figure 3 As shown, the remote sensing image spatiotemporal fusion model includes a change detection module (for executing step S202), a feature mask modulator (for executing step S203), an invariant region fusion module (for executing step S204), a coarse image restoration module (for executing step S205), and a fine image restoration module (for executing step S206). After fine image restoration, cross entropy loss and structural loss supervision can also be performed to obtain a fine image prediction result at the predicted time.

[0074] By analyzing the differences between the high-temporal-resolution image at the time to be predicted and the high-temporal-resolution image corresponding to the reference time, extracting regional change information and generating a binary mask of changes, it is possible to accurately identify and locate the changed areas, allowing the model to adopt differentiated modeling strategies for unchanged areas and changed areas with significant changes, effectively alleviating the prediction error caused by the inconsistency between the spatial structure of the reference image and the true state of the image to be predicted.

[0075] Step S203 : enhancing features of unchanged regions and suppressing features of changed regions of the first fine image using the changed binary mask to obtain an enhanced first fine image.

[0076] The first fine image is affine transformed by the change binary mask to perform channel-by-channel scaling and offset to obtain an enhanced first fine image, which is convenient for subsequent adjustment of the feature responses of the change area and the unchanged area respectively. The affine transformation includes multiplicative modulation and additive modulation. By setting the modulation factor corresponding to the unchanged area to a larger value for multiplicative modulation, the feature expression of the unchanged area can be enhanced; by setting the modulation factor corresponding to the changed area to a smaller value or even 0 for multiplicative modulation, the influence of the changed area can be weakened. Multiplicative modulation through the modulation factor can make the model focus on the unchanged area with higher credibility, thereby improving the subsequent fusion effect. Additive modulation through the bias factor can adjust the feature response positively or negatively according to demand, perform fine-grained adjustment of the feature distribution, and improve the model's flexible correction of the feature position.

[0077] Specifically, the changing binary mask is first encoded by a modulation factor encoder to obtain a first encoding result; the first encoding result is nonlinearly transformed by a first activation function to obtain a modulation factor; the changing binary mask is encoded by a modulation bias encoder to obtain a second encoding result; the second encoding result is nonlinearly transformed by a second activation function to obtain a bias factor; the first fine image is multiplicatively modulated by the modulation factor, and the first fine image is additively modulated by the bias factor to obtain an enhanced first fine image.

[0078] For example, first pass the modulation factor encoder Encode the changing binary mask M to obtain the first encoding result , and then the first encoding result is nonlinearly transformed by the first activation function Sigmoid to obtain the modulation factor Calculate the modulation factor The formula is as follows:

[0079]

[0080] in, is the modulation factor encoder, and the first activation function is the Sigmoid function.

[0081] Next, by modulating the bias encoder Encode the changed binary mask M to obtain the second encoding result ; The second encoding result is processed by the second activation function Tanh Perform nonlinear transformation to obtain the bias factor . Calculate the bias factor The formula is as follows:

[0082]

[0083] in, is a modulated bias encoder, and the second activation function is the Tanh function.

[0084] Finally, by modulating the factor For the first fine image Multiplicative modulation is performed by bias factor For the first fine image Additive modulation is performed to obtain an enhanced first fine image Specifically, by modulating the factor For the first fine image When performing multiplicative modulation, find the modulation factor With the first fine image The Hadamard product of ; By using the bias factor For the first fine image When additive modulation is performed, the intermediate enhancement result Add the bias factor , get the enhanced first fine image .

[0085] Exemplarily, the enhanced first fine image is obtained The formula is as follows:

[0086]

[0087] in, represents the Hadamard product.

[0088] Exemplarily, a feature mask modulator is constructed, which is based on a binary mask Generated prior conditions To reference time The first fine image Modulation is performed to obtain an enhanced first fine image . Figure 4 This is a schematic diagram of the structure of the feature mask modulator provided by an embodiment of the present invention. The modulation factor encoder sequentially includes a first convolution operation (Conv3×3, convolution kernel size 3×3), a normalization and activation operation (BN / ReLU), and a second convolution operation (Conv1×1, convolution kernel size 1×1). The modulation bias encoder sequentially includes a first convolution operation, a normalization and activation operation, and a second convolution operation. Figure 4 The input features in correspond to the first refined image, Figure 4 The modulation features in correspond to the changing binary mask.

[0089] Step S204: performing feature fusion on the second coarse image and the enhanced first fine image to obtain a fusion result.

[0090] Specifically, feature extraction is performed on the second coarse image to obtain a first intermediate feature; feature extraction is performed on the enhanced first fine image to obtain a second intermediate feature; the first intermediate feature and the second intermediate feature are fused to obtain a first fused feature; feature extraction is performed on the first intermediate feature through a first encoder to obtain a first encoded feature; feature extraction is performed on the first fused feature through a second encoder to obtain a second encoded feature; channel attention fusion processing is performed on the first encoded feature and the second encoded feature to obtain a fusion result.

[0091] Optionally, when performing channel attention fusion processing on the first coding feature and the second coding feature, the first coding feature and the second coding feature are channel-spliced ​​to obtain a first splicing result; global average pooling is performed on the first splicing result to obtain a global average pooling result; global maximum pooling is performed on the first splicing result to obtain a global maximum pooling result; the global average pooling result and the global maximum pooling result are spliced ​​to obtain a second splicing result; the second splicing result is nonlinearly transformed by a first activation function to obtain a splicing activation result; the splicing activation result is multiplied by the first splicing result to obtain a first intermediate adjustment result; the first intermediate adjustment result is added to the first splicing result to obtain a second intermediate adjustment result; and feature fusion is performed on the second intermediate adjustment result by a residual block to obtain a fusion result.

[0092] Optionally, when feature fusion is performed on the second intermediate adjustment result through the residual block, the second intermediate adjustment result is subjected to a first convolution process, normalization, and activation to obtain a first residual result; the first residual result is subjected to a second convolution process, normalization, and activation to obtain a second residual result; the second intermediate adjustment result is subjected to a second convolution process, normalization, and activation to obtain a third residual result; the second residual result and the third residual result are added to obtain a fusion result.

[0093] For example, an invariant region fusion module is constructed. The invariant region fusion module is composed of three feature fusion encoders and three feature fusion decoders in sequence, and adopts a layer-by-layer fusion strategy. The encoder is responsible for Extract the first intermediate feature , the encoder also separately from the second rough image and enhanced first fine image Extract information and fuse it to get the first fusion feature , and then sent to the next encoder as input. Finally, after being encoded in sequence by three layers of encoders, the final fusion feature is obtained and coarse image coding features . Combine the coarse image encoding features of each layer with the fusion features Send them to the decoder together to get the final unchanged region fusion result .

[0094] For example, take the j-th layer encoder as an example, j∈{1,2,3}, the input of the j-th layer encoder is the rough feature and fusion features Among them, the second rough image As the rough features, the enhanced first fine image As the fusion feature, it is input into the first layer encoder. Each layer encoder has two branches, which process the rough features and fusion features separately and then fuse them. , using the first encoder Perform simple feature extraction on the rough features to obtain the first coding feature , the formula is as follows:

[0095]

[0096] For fusion features , using the second encoder Perform feature extraction to obtain the second coding feature , the formula is as follows:

[0097]

[0098] Next, the first intermediate feature and the second intermediate feature Splice by channel to get the first splicing result , the formula is as follows:

[0099]

[0100] Next, the first splicing result Perform global average pooling and global maximum pooling respectively to convert the features into channel-level representations, then process them through the fully connected layer to capture the dependencies between channels, and finally use the activation function to generate the weights of each channel, and multiply these weights by the original feature map to obtain the final adjusted feature representation. Perform global average pooling to obtain the global average pooling result ; For the first splicing result Perform global maximum pooling to obtain the global maximum pooling result ; The global average pooling result And the global maximum pooling result Perform splicing to obtain the second splicing result ; The second splicing result is obtained by the first activation function Perform nonlinear transformation to obtain splicing activation results ; The splicing activation result and the first splicing result Multiply to get the first intermediate adjustment result ; The first intermediate adjustment result and the first splicing result Add together to get the second intermediate adjustment result , the formula is as follows:

[0101]

[0102] Finally, the second intermediate adjustment result is adjusted by the residual block Further feature fusion is performed. Specifically, the second intermediate adjustment result Perform the first convolution, normalization and activation to obtain the first residual result ; Perform the second convolution, normalization and activation on the first residual result to obtain the second residual result ; Perform a second convolution, normalization, and activation on the second intermediate adjustment result to obtain a third residual result ; Add the second residual result and the third residual result to obtain the fusion result , the formula is as follows:

[0103]

[0104] in, Represents the process of convolution, normalization and activation in sequence, Indicates that the convolution kernel size used for the first convolution process is , Indicates that the convolution kernel size used for the first convolution process is The activation function used is the Sigmoid function.

[0105] Repeat the above steps three times to get the final fusion result. , the final fusion result Send it to the decoder to get the fusion result .

[0106] Figure 5 The structure diagram of the feature fusion encoder provided by the embodiment of the present invention. The function of the feature fusion encoder is to convert the rough features into and fusion features The rough features are the information that needs to be retained, so the corresponding first encoder It consists of only one residual block, which is responsible for matching the number of channels and obtaining rough features. , which is also the input of the next layer of rough features ; Fusion features The corresponding second encoder It consists of two branches, ASPP (Atrous Spatial Pyramid Pooling) spatial pyramid (corresponding to Figure 5 The pyramid branch in the image is responsible for multi-scale feature extraction, while the branch composed of residual blocks is responsible for local detail feature extraction. The features extracted by the two branches are stacked by channel and modulated by a layer of residual blocks to obtain high-level features extracted from the fusion features. Next, the extracted high-level features With the adjusted roughness characteristics After being sent to the three channel attention modules for feature screening and enhancement, they are fused through a residual block to obtain the fused features of the next layer. By setting up the ASPP spatial pyramid and channel attention module, it is possible to take into account the complex change relationship in both time and space dimensions, effectively model the model, improve the adaptability of the model in long-term and multi-type change situations, improve the robustness of the model, and thus improve the accuracy of image fusion. Among them, Figure 5 The fine encoder shown in refers to the second encoder.

[0107] Among them, the ASPP spatial pyramid consists of convolutions of multiple scales. Assuming that the input feature is , the output features are , then the processing process of ASPP spatial pyramid is as follows:

[0108]

[0109] Where r represents the expansion rate, Represents an upsampling operation.

[0110] Step S205 , performing a rough repair on the changed area corresponding to the fusion result based on the changed binary mask to obtain a rough repair result.

[0111] Specifically, the pixels in the changed area of ​​the fusion result are set to 0 using the change binary mask to obtain a first intermediate coarse restoration result; image restoration processing is performed on the first intermediate coarse restoration result to obtain a second intermediate coarse restoration result; the pixels that have undergone image restoration in the second intermediate coarse restoration result are extracted using the change binary mask and replaced with the pixels at the corresponding position in the fusion result to obtain a coarse restoration result. By performing coarse restoration on the changed area, the advantages of image restoration technology in region completion tasks can be leveraged to use a restoration network to perform contextual feature modeling and detail generation on the changed area, thereby improving the structural coherence and texture rationality of the mutation area. Compared with the blind fusion of reference image content in related technologies, this solution can dynamically generate image content that is closer to the actual state at the time of prediction based on the actual changes, thereby improving overall prediction accuracy and the accuracy of image fusion.

[0112] For example, by changing the binary mask The fusion result The pixels in the changed area are set to 0 to obtain a first intermediate rough restoration result; the first intermediate rough restoration result is subjected to image restoration processing to obtain a second intermediate rough restoration result , the specific formula is as follows:

[0113]

[0114] Then, through the change binary mask Extract the pixels that have been restored in the second intermediate rough restoration result and replace the fusion result The pixels at the corresponding position in the image are used to obtain the rough restoration result. The specific formula is as follows:

[0115]

[0116] For example, an image restoration module CIM (Image Restoration Module) is constructed. The rough image restoration module is mainly composed of multiple gated encoders, each of which is composed of gated convolution, normalization layer and activation function. For an input feature with a channel number of C and a length and width of H×W , firstly, the input features are transformed into The number of channels is increased to 2C, and then the first half of the channels are used as features, and the second half of the channels are used as learnable gates. The formula for the overall process is as follows:

[0117]

[0118]

[0119] in, represents the Sigmoid activation function, Only the number of channels is doubled without changing the feature size.

[0120] Step S206 , performing global style unification processing and local edge restoration on the coarse restoration result to obtain a fine image prediction result at the time to be predicted.

[0121] Specifically, the coarse restoration result is subjected to global style uniform processing through a global style encoder to obtain a global style encoding result; the area composed of replaced pixels in the coarse restoration result is subjected to local edge restoration through a local edge encoder to obtain a local edge encoding result; the global style encoding result and the local edge encoding result are decoded to obtain a fine image prediction result at the time to be predicted.

[0122] For example, a fine image restoration module is constructed to further refine and unify the rough restoration results. The fine image restoration module consists of two encoders and a decoder. The two encoders respectively refine and unify the rough restoration results. Perform global style unification and local edge repair, and the output results are spliced ​​along the channel dimension and sent to the decoder to obtain the fine image prediction result at the time to be predicted .

[0123] For example, first pass the global style encoder The rough restoration result is processed with global style uniformity to obtain the global style encoding result. The formula is as follows:

[0124]

[0125] Then, through the local edge encoder Perform local edge repair on the area composed of replaced pixels in the rough repair result, and refine the regional edge to achieve a more natural edge transition, and obtain the local edge encoding result. The formula is as follows:

[0126]

[0127] Finally, the global style coding result and the local edge coding result are decoded to obtain the fine image prediction result at the time to be predicted .

[0128] For example, Figure 6 A schematic diagram of the structure of the image fine restoration module provided by an embodiment of the present invention. Global style encoder Mainly composed of ASPP spatial pyramid and self-attention module, local edge encoder It consists of two gated encoders, which are the same as the gated encoders in the image roughening module.

[0129] The ASPP spatial pyramid is first used to extract multi-scale features of the rough restoration results, and then the self-attention mechanism of the self-attention module is used to model the global information, enhance the long-distance dependencies between features, and improve the overall structural consistency and style unity. The corresponding formula of the self-attention module is as follows:

[0130]

[0131] in, is the scaling factor, Q is the query, K is the key, and V is the value. represents a normalization layer.

[0132] As one or more specific application examples of the embodiments of the present invention, the optimal implementation scheme or the solution that the inventor most wants to embody is described below in combination with specific application scenarios.

[0133] Figure 7An example diagram of the prediction results of image fusion using different reference images provided in an embodiment of the present invention. Due to the sudden occurrence of flood disasters at the time to be predicted, it is difficult to accurately predict the fine image of the area at the time to be predicted from images at other times. Among them, the first column is the target image that needs to be achieved in the image fusion task, and the subsequent three columns are the fine image prediction results of the time to be predicted obtained by selecting different reference images and using this scheme to perform image fusion. It can be seen that this scheme can still restore the texture of the flood area more accurately at different reference times, showing strong robustness to sudden changes in scenes, and effectively breaking through the dependence of the image fusion method in related technologies on the temporal proximity of the reference images. This scheme can also maintain stable reconstruction performance under long time span conditions.

[0134] Figure 8 The example diagram for comparing prediction results of different image fusion methods provided in the embodiment of the present invention shows the corresponding prediction results obtained by using different image fusion methods for image fusion. Among them, column (a) is the real observed image, columns (b)-(f) are the prediction results of the methods FSDAF (Flexible Spatiotemporal Data Fusion, a flexible spatiotemporal fusion method), Fit-FC (a three-step remote sensing spatiotemporal fusion method consisting of regression model fitting (RM fitting), spatial filtering (SF) and residual compensation (RC)), GAN-STFM (GAN-based SpatioTemporal Fusion Model, generative adversarial network-spatiotemporal fusion model), MLFF-GAN (multilevel feature fusion with generative adversarial network), and ECPW-STFN (the enhanced cross-pairedwavelet based spatiotemporal fusion networks) on dataset A, respectively. Dataset A can be a dataset containing remote sensing images of different spatiotemporal resolutions that has been disclosed in the field of this technology. Column (g) is the prediction result of this scheme. It can be seen from the locally enlarged images in the second row that this scheme can still maintain a good restoration effect when dealing with sudden changes in landform types caused by floods, without losing the texture details of the flood.

[0135] Table 1 shows the comparative results of metrics on Dataset B (a dataset used for spatiotemporal fusion research of remote sensing images) and Dataset A. Dataset B can be a publicly available dataset used for spatiotemporal fusion research of remote sensing images in this technical field. It can be seen that this scheme has certain advantages over other methods in terms of structural similarity (corresponding to SSIM, Structural Similarity Index), spectral restoration effect (corresponding to SAM, Spectral Angle Mapper), and pixel value accuracy (corresponding to MAE and RMSE, MAE is Mean Absolute Error, RMSE is Root Mean Squared Error).

[0136] Table 1: Comparison results of indicators of this scheme and other methods.

[0137]

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0139] The embodiment of the present invention further provides an image fusion device, such as Figure 9 As shown, the image fusion device includes an image acquisition module 901 , a screening module 902 , an enhancement module 903 , a fusion module 904 and a restoration module 905 .

[0140] An image acquisition module 901 is configured to acquire a first coarse image at a reference moment, a first fine image at a reference moment, and a second coarse image at a time to be predicted; wherein the spatial resolution of the first fine image is higher than the spatial resolution of the first coarse image;

[0141] A screening module 902 is configured to screen a change region between the first rough image and the second rough image to generate a change binary mask;

[0142] An enhancement module 903 is configured to enhance features of unchanged regions and suppress features of changed regions of the first fine image using the changed binary mask to obtain an enhanced first fine image;

[0143] A fusion module 904 is configured to perform feature fusion on the second coarse image and the enhanced first fine image to obtain a fusion result;

[0144] The restoration module 905 is configured to perform image restoration on the fusion result based on the changed binary mask to obtain a refined image prediction result at the time to be predicted.

[0145] For the description of the features in the embodiment corresponding to the image fusion device, reference can be made to the relevant description of the embodiment corresponding to the image fusion method, which will not be repeated here.

[0146] An embodiment of the present invention further provides an electronic device, such as Figure 10 As shown, it includes a memory 10 and a processor 20. The memory 10 stores a computer program, and the processor 20 is configured to run the computer program to execute the steps in any of the above-mentioned image fusion method embodiments.

[0147] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned image fusion method embodiments when running.

[0148] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0149] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned image fusion method embodiments are implemented.

[0150] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned image fusion method embodiments are implemented.

[0151] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] The above is a detailed introduction to the image fusion method, device, storage medium and program product provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core ideas. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of this application.

Claims

1. An image fusion method, characterized in that: The image fusion method comprises: Acquire a first coarse image at a reference moment, a first fine image at the reference moment, and a second coarse image at a time to be predicted; wherein the spatial resolution of the first fine image is higher than the spatial resolution of the first coarse image; screening a change region between the first coarse image and the second coarse image to generate a change binary mask; The first fine image is subjected to the changing binary mask to enhance features of invariant regions and suppress features of changing regions, thereby obtaining an enhanced first fine image; wherein a Hadamard product of a modulation factor and the first fine image is obtained to perform multiplicative modulation on the first fine image; the Hadamard product of the modulation factor and the first fine image and a bias factor are added to perform additive modulation on the first fine image; the modulation factor is obtained by encoding the changing binary mask using a modulation factor encoder; and the bias factor is obtained by encoding the changing binary mask using a modulation bias encoder; performing feature fusion on the second coarse image and the enhanced first fine image to obtain a fusion result; Image restoration is performed on the fusion result based on the change binary mask to obtain a fine image prediction result for the time to be predicted.

2. The image fusion method according to claim 1, characterized in that: The step of screening the change region between the first coarse image and the second coarse image to generate a change binary mask includes: Subtracting the first rough image from the second rough image to obtain an intermediate image; the intermediate image includes multiple channels; Performing an averaging operation and an interquartile range calculation on the intermediate image of each channel to generate an upper threshold value and a lower threshold value for change determination corresponding to each channel; The intermediate image of each channel is compared with the change judgment upper threshold and change judgment lower threshold corresponding to each channel to generate a change binary mask for each channel.

3. The image fusion method according to claim 2, characterized in that: The step of performing an averaging operation and an interquartile range calculation on the intermediate image of each channel to generate an upper threshold value and a lower threshold value for change determination corresponding to each channel includes: Performing an averaging operation on the intermediate image of each channel to obtain an average result of each channel; Performing interquartile range calculation on the intermediate image of each channel to obtain an interquartile range calculation result of each channel; The upper threshold for change determination of each channel is obtained based on the sum of the average result and the interquartile range calculation result of each channel; The lower threshold for change determination of each channel is obtained based on the difference between the average result of each channel and the interquartile range calculation result.

4. The image fusion method according to claim 3, characterized in that: The step of comparing the intermediate image of each channel with the change determination upper threshold and the change determination lower threshold corresponding to each channel to generate a change binary mask for each channel includes: Comparing the pixel values ​​of each spatial coordinate in the intermediate image of each channel with the change determination upper threshold and the change determination lower threshold corresponding to each channel respectively; If the pixel value of the target space coordinate is greater than the upper threshold for change determination or the pixel value of the target space coordinate is less than the lower threshold for change determination, the change binary mask corresponding to the target space coordinate is determined to be 1; otherwise, the change binary mask corresponding to the target space coordinate is 0; the target space coordinate is any one of the spatial coordinates.

5. The image fusion method according to any one of claims 1 to 4, characterized in that: The step of enhancing features of unchanged regions and suppressing features of changed regions of the first fine image using the changed binary mask to obtain an enhanced first fine image includes: Encoding the changed binary mask by a modulation factor encoder to obtain a first encoding result; Performing a nonlinear transformation on the first coding result by using a first activation function to obtain a modulation factor; encoding the changing binary mask by a modulation bias encoder to obtain a second encoding result; Performing a nonlinear transformation on the second encoding result by using a second activation function to obtain a bias factor; The first fine image is multiplicatively modulated by the modulation factor, and the first fine image is additively modulated by the bias factor to obtain an enhanced first fine image.

6. The image fusion method according to claim 5, characterized in that: The multiplicative modulation of the first fine image by the modulation factor and the additive modulation of the first fine image by the bias factor to obtain an enhanced first fine image includes: Calculating a Hadamard product between the modulation factor and the first refined image to obtain an intermediate enhancement result; The intermediate enhancement result is added to the bias factor to obtain an enhanced first fine image.

7. The image fusion method according to any one of claims 1 to 4, characterized in that: Performing feature fusion on the second coarse image and the enhanced first fine image to obtain a fusion result includes: performing feature extraction on the second rough image to obtain a first intermediate feature; performing feature extraction on the enhanced first fine image to obtain a second intermediate feature; fusing the first intermediate feature and the second intermediate feature to obtain a first fused feature; Performing feature extraction on the first intermediate feature by a first encoder to obtain a first encoded feature; Performing feature extraction on the first fusion feature by a second encoder to obtain a second encoding feature; Channel attention fusion processing is performed on the first coding feature and the second coding feature to obtain a fusion result.

8. The image fusion method according to claim 7, characterized in that: The performing channel attention fusion processing on the first coding feature and the second coding feature to obtain a fusion result includes: Channel-joining the first coding feature and the second coding feature to obtain a first joining result; Performing global average pooling on the first splicing result to obtain a global average pooling result; Performing global maximum pooling on the first splicing result to obtain a global maximum pooling result; Concatenate the global average pooling result and the global maximum pooling result to obtain a second concatenation result; Performing a nonlinear transformation on the second splicing result by using a first activation function to obtain a splicing activation result; multiplying the splicing activation result by the first splicing result to obtain a first intermediate adjustment result; Adding the first intermediate adjustment result to the first splicing result to obtain a second intermediate adjustment result; The second intermediate adjustment result is subjected to feature fusion through the residual block to obtain a fusion result.

9. The image fusion method according to claim 8, characterized in that: Performing feature fusion on the second intermediate adjustment result through the residual block to obtain a fusion result, including: Performing a first convolution process, normalization, and activation on the second intermediate adjustment result to obtain a first residual result; Performing a second convolution process, normalization, and activation on the first residual result to obtain a second residual result; Performing a second convolution process, normalization, and activation on the second intermediate adjustment result to obtain a third residual result; The second residual result and the third residual result are added together to obtain a fusion result.

10. The image fusion method according to any one of claims 1 to 4, characterized in that: Performing image restoration on the fusion result based on the change binary mask to obtain a fine image prediction result for the time to be predicted, including: Performing a rough repair on the changed area corresponding to the fusion result based on the changed binary mask to obtain a rough repair result; The rough restoration result is subjected to global style unification processing and local edge restoration to obtain a fine image prediction result for the prediction moment.

11. The image fusion method according to claim 10, characterized in that: Performing a rough repair on the changed area corresponding to the fusion result based on the changed binary mask to obtain a rough repair result, including: The pixels in the changed area of ​​the fusion result are set to 0 using the changed binary mask to obtain a first intermediate rough restoration result; performing image restoration processing on the first intermediate coarse restoration result to obtain a second intermediate coarse restoration result; The pixels that have undergone image restoration in the second intermediate coarse restoration result are extracted through the change binary mask, and the pixels at corresponding positions in the fusion result are replaced to obtain a coarse restoration result.

12. The image fusion method according to claim 11, characterized in that: The performing of global style unification processing and local edge restoration on the coarse restoration result to obtain a fine image prediction result for the prediction moment includes: Performing global style uniform processing on the rough restoration result through a global style encoder to obtain a global style encoding result; Performing local edge restoration on the area consisting of replaced pixels in the rough restoration result by using a local edge encoder to obtain a local edge encoding result; The global style coding result and the local edge coding result are decoded to obtain a fine image prediction result at the time to be predicted.

13. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the image fusion method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the image fusion method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image fusion method according to any one of claims 1 to 12 are implemented.

Images