Image fusion method and device based on joint loss, equipment, medium and product

Abstract

This invention provides an image fusion method, apparatus, device, medium, and product based on joint loss, relating to the field of remote sensing satellite technology. The method includes: inputting panchromatic and multispectral images into a remote sensing image fusion network to obtain a target fused image; wherein the remote sensing image fusion network is trained based on a joint loss function, which is determined based on a spatial loss function and a spectral loss function; the spatial loss function is used to minimize the multidirectional consistency between the sample panchromatic image and the image fusion result; the spectral loss function is used to minimize the spectral consistency between the sample multispectral image and the image fusion result. Through this method, the target fused image can maintain both spectral consistency with the multispectral image and spatial consistency with the panchromatic image, achieving a dynamic balance between spectral fidelity and spatial consistency, which is beneficial for improving the image fusion effect.

Description

Technical Field

[0001] This invention relates to the field of remote sensing satellite technology, and in particular to an image fusion method, apparatus, device, medium and product based on joint loss. Background Technology

[0002] Spatial-spectral fusion is a remote sensing image fusion technique that combines a panchromatic image (PAN) with high spatial resolution and a multispectral image (MS) with high spectral resolution to generate a high-resolution multispectral (HRMS) image that has both high spatial resolution and high spectral fidelity.

[0003] In some related technologies, panchromatic and multispectral images can be fused using pre-trained deep learning networks to generate high-resolution multispectral images. However, existing deep learning networks suffer from poorly designed loss functions during model training: Regarding spectral constraints, most existing loss functions can only be used for band-by-band measurements, easily neglecting the consistency of pixel-level multiband spectral vectors. This can easily induce hue shift and cross-band imbalance, resulting in low spectral fidelity of the final high-resolution multispectral image. Regarding spatial constraints, existing loss functions typically only achieve isotropic or unidirectional gradient constraints, lacking consideration for the consistency of multi-directional edge responses in the image. This can easily lead to misalignment in the directional structure between the panchromatic image and the reconstructed high-resolution multispectral image, thus affecting the spatial resolution of the final high-resolution multispectral image.

[0004] In summary, the loss functions used in existing remote sensing image fusion methods have limitations in terms of spectral and spatial constraints, which can easily lead to poor image fusion performance of the trained deep learning networks, thus limiting the application of spatial-spectral fusion technology in remote sensing image processing. Summary of the Invention

[0005] This invention provides an image fusion method, apparatus, device, medium, and product based on joint loss, which addresses the limitations of the loss functions used in the prior art in terms of spectral and spatial constraints, which easily leads to poor image fusion performance of the trained deep learning network and limits the application of spatial-spectral fusion technology in remote sensing image processing.

[0006] This invention provides an image fusion method based on joint loss, comprising: acquiring panchromatic images and multispectral images; inputting the panchromatic images and multispectral images into a remote sensing image fusion network to obtain a target fused image output by the remote sensing image fusion network; wherein, the remote sensing image fusion network is trained based on sample panchromatic images and sample multispectral images; the remote sensing image fusion network is trained based on a joint loss function, which is determined based on a spatial loss function and a spectral loss function; the spatial loss function is used to minimize the multidirectional consistency between the sample panchromatic images and the image fusion result; the spectral loss function is used to minimize the spectral consistency between the sample multispectral images and the image fusion result.

[0007] According to the image fusion method based on joint loss provided by the present invention, the spatial loss function is a multi-directional consistency loss function; wherein, the spatial loss function is determined based on the Pearson correlation coefficient and multiple directional gradients, and one directional gradient is obtained by edge extraction of the sample panchromatic image and the image fusion result in a preset direction.

[0008] According to the image fusion method based on joint loss provided by the present invention, the spectral loss function is the spectral hyperspheric aberration loss function; wherein, the spectral loss function is determined based on the similarity between the spectral vector of the sample multispectral image and the spectral vector of the image fusion result on a unit hypersphere.

[0009] According to the image fusion method based on joint loss provided by the present invention, the joint loss function is determined by weighted aggregation operation based on the spatial loss function, the first weighting factor corresponding to the spatial loss function, the spectral loss function, and the second weighting factor corresponding to the spectral loss function; wherein, the first weighting factor and the second weighting factor are determined experimentally based on a short-cycle training mode.

[0010] According to the image fusion method based on joint loss provided by the present invention, the remote sensing image fusion network is trained based on the following steps: obtaining a training dataset and a test dataset; both the training dataset and the test dataset include multiple sample panchromatic images and multiple sample multispectral images; training an initial model based on the training dataset until the initial model passes the validation of the test dataset, thereby obtaining the remote sensing image fusion network.

[0011] According to the present invention, an image fusion method based on joint loss is provided to obtain panchromatic images and multispectral images, including: obtaining panchromatic images and original multispectral images; the panchromatic images and original multispectral images are obtained by satellites capturing images of a target area; and upsampling the original multispectral images to obtain the multispectral images.

[0012] This invention also provides an image fusion device based on joint loss, comprising: an acquisition module for acquiring panchromatic images and multispectral images; and a remote sensing image fusion module for inputting the panchromatic images and multispectral images into a remote sensing image fusion network to obtain a target fused image output by the remote sensing image fusion network; wherein the remote sensing image fusion network is trained based on sample panchromatic images and sample multispectral images; the remote sensing image fusion network is trained based on a joint loss function, which is determined based on a spatial loss function and a spectral loss function; the spatial loss function is used to minimize the multidirectional consistency between the sample panchromatic images and the image fusion result; and the spectral loss function is used to minimize the spectral consistency between the sample multispectral images and the image fusion result.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the image fusion methods based on joint loss as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the image fusion methods based on joint loss as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the image fusion methods based on joint loss as described above.

[0016] The image fusion method, apparatus, device, medium, and product based on joint loss provided by this invention, in the model training stage, obtains a remote sensing image fusion network through training with a joint loss function. The joint loss function is constructed based on a spatial loss function and a spectral loss function. The spatial loss function is used to minimize the multi-directional consistency between the sample panchromatic image and the image fusion result, and the spectral loss function is used to minimize the spectral consistency between the sample multispectral image and the image fusion result. By training the model through the above-mentioned joint loss function, the trained remote sensing image fusion network can simultaneously constrain the image fusion process in both spectral and spatial dimensions. Therefore, in the model application stage, it is beneficial to ensure that the target fused image generated by the remote sensing image fusion network can maintain both spectral consistency with the multispectral image and spatial consistency with the panchromatic image, achieving a dynamic balance between spectral fidelity and spatial consistency, which is beneficial to improving the image fusion effect and thus facilitating the widespread application of spatial-spectral fusion technology in the field of remote sensing image processing. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the image fusion method based on joint loss provided by the present invention.

[0019] Figure 2 This is one of the construction diagrams of the joint loss function provided by the present invention.

[0020] Figure 3 This is the second construction diagram of the joint loss function provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the image fusion device based on joint loss provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating the image fusion method based on joint loss provided by the present invention. Figure 1 As shown, in this embodiment, the image fusion method based on joint loss includes steps S110 to S120, and the specific steps are as follows: S110: Acquire panchromatic and multispectral images.

[0025] Specifically, in the model application phase, panchromatic images (i.e., PAN images) and raw multispectral images (i.e., MS images) are first acquired; the panchromatic images and raw multispectral images are obtained by satellites taking pictures of the target area.

[0026] Furthermore, the original multispectral image is upsampled to obtain a multispectral image (i.e., a UPMS image). "Indicates upsampling processing".

[0027] S120: Input the panchromatic image and multispectral image into the remote sensing image fusion network to obtain the target fused image output by the remote sensing image fusion network.

[0028] The target fusion image is a high-resolution multispectral image (i.e., HRMS image).

[0029] The remote sensing image fusion network is trained based on sample panchromatic images and sample multispectral images.

[0030] The remote sensing image fusion network is trained based on a joint loss function, which is determined based on the spatial loss function and the spectral loss function.

[0031] The spatial loss function is used to minimize the multidirectional consistency between the sample panchromatic image and the image fusion result.

[0032] The spectral loss function is used to minimize the spectral consistency of sample multispectral images and image fusion results.

[0033] Specifically, this embodiment proposes a joint loss function constructed by a spatial loss function and a spectral loss function. During the model training phase, self-supervised learning training or unsupervised learning training is introduced, and the network is constrained and optimized by the joint loss function proposed in this embodiment. This enables the network to achieve joint enhancement of image features in both spatial and spectral dimensions, thereby ensuring that the fusion result of the network has both high spatial resolution and high spectral fidelity during the model application phase.

[0034] Optionally, if unsupervised learning training is introduced during the model training phase, the image fusion result is a sample fusion image (the sample fusion image is a sample high-resolution multispectral image). That is, the sample fusion image is used as the result label during the model training phase, and the sample panchromatic image, sample multispectral image and sample fusion image are input into the initial model for training. After training, a remote sensing image fusion network is obtained.

[0035] Optionally, if self-supervised learning training is introduced during the model training phase, there is no need to introduce result labels. That is, the sample panchromatic image and sample multispectral image are directly input into the initial model for training. During the training process, the initial model can learn to generate high-resolution multispectral images based on the sample panchromatic image and sample multispectral image as the image fusion result, and obtain the remote sensing image fusion network after training is completed.

[0036] Optionally, the expression for the remote sensing image fusion network is: ; in, This represents a remote sensing image fusion network; This represents the numerical values ​​related to different parameter settings in the remote sensing image fusion network; Represents a panchromatic image. This represents a multispectral image that has undergone upsampling (symbol " "). "Indicates upsampling processing" and This serves as the input data for a remote sensing image fusion network; the core objective of this network is to fuse complementary information from panchromatic images and upsampled multispectral images to obtain high-resolution multispectral images. .

[0037] The image fusion method based on joint loss provided in this embodiment obtains a remote sensing image fusion network through training a joint loss function during the model training phase. The joint loss function is constructed based on spatial loss function and spectral loss function. The spatial loss function is used to minimize the multi-directional consistency between the sample panchromatic image and the image fusion result, while the spectral loss function is used to minimize the spectral consistency between the sample multispectral image and the image fusion result. By training the model through the above-mentioned joint loss function, the trained remote sensing image fusion network can simultaneously constrain the image fusion process in both spectral and spatial dimensions. Therefore, in the model application phase, it is beneficial to ensure that the target fused image generated by the remote sensing image fusion network can maintain both spectral consistency with the multispectral image and spatial consistency with the panchromatic image, achieving a dynamic balance between spectral fidelity and spatial consistency. This is beneficial to improving the image fusion effect and thus facilitating the widespread application of spatial-spectral fusion technology in the field of remote sensing image processing.

[0038] In some embodiments, the spatial loss function is a multi-directional consistency loss function; wherein, the spatial loss function is determined based on the Pearson correlation coefficient and multiple directional gradients, and one directional gradient is obtained by edge extraction in a preset direction from the sample panchromatic image and the image fusion result.

[0039] Please see Figure 2 and Figure 3 , Figure 2 This is one of the construction diagrams of the joint loss function provided by the present invention. Figure 3 This is the second construction diagram of the joint loss function provided by the present invention.

[0040] Specifically, such as Figure 2 and Figure 3As shown, to address the problem that remote sensing image fusion networks struggle to balance spectral consistency and spatial structure consistency during model training, this embodiment proposes a joint loss function of spectral hypersphere difference and multi-directional consistency (SHDI-MDC Loss), composed of spectral hypersphere difference loss and multi-directional consistency loss. The core design idea of ​​this joint loss function is: high-resolution multispectral images (i.e.,...) Figure 2 and Figure 3 The spatial information of the HRMS image should be consistent with that of the panchromatic image (i.e., Figure 2 and Figure 3 The PAN image in the image should be aligned with the upsampled multispectral image (i.e., the spectral characteristics of which should be consistent with the upsampled multispectral image). Figure 2 and Figure 3 In (To maintain consistency)

[0041] Based on this, the spatial loss function in this embodiment adopts the multidirectional consistency (MDC) loss function, while the spectral loss function adopts the spectral hyperspheric aberration (SHDI) loss function. Through the joint constraint of the multidirectional consistency (MDC) loss function and the spectral hyperspheric aberration (SHDI) loss function, the remote sensing image fusion network can improve the quality of the fused image from both spatial and spectral dimensions.

[0042] Specifically, the space loss function The consistency of spatial information between the sample panchromatic image and the image fusion result (i.e., the sample high-resolution multispectral image or the high-resolution multispectral image generated by the model) can be ensured by calculating the directional gradient magnitude response in different preset directions.

[0043] Optionally, the number and angle of preset directions can be set according to actual needs.

[0044] Preferably, the multiple preset directions are 0°, 45°, 90° and 135° respectively.

[0045] Preferably, the space loss function The Sobel operator, consisting of 3×3 convolution kernels, can be used in each preset direction to extract edges from each band of the sample panchromatic image and the image fusion result, and to calculate its gradient map.

[0046] Among them, the space loss function By minimizing the differences in gradient maps in different preset directions, multi-directional structural alignment between the sample panchromatic image and the image fusion result is achieved. This constrains the spatial detail injection process of the image fusion result, ensuring that the image fusion result can inherit the edge features of the sample panchromatic image, while suppressing false edges and noise artifacts in the image fusion result.

[0047] Optionally, in order to conform to the optimization rule of minimizing the loss function in deep learning, the spatial loss function... The expression is as follows: ; ; in, The Pearson correlation coefficient is used to measure... and Linear correlation between them, for example Indicates calculation and Linear correlation between them; Indicates expansion along the channel dimension; Indicates application orientation The Sobel operator is used to calculate the directional gradient; Indicates the angle of the preset direction.

[0048] Preferably, The corresponding expression for the Sobel operator, which consists of 3×3 convolution kernels, is as follows: ; ; ; .

[0049] In some embodiments, the spectral loss function is the spectral hyperspheric aberration loss function; wherein, the spectral loss function is determined based on the similarity between the spectral vector of the sample multispectral image and the spectral vector of the image fusion result on a unit hypersphere.

[0050] Specifically, the spectral loss function Spectral consistency can be maintained by measuring the directional difference of the multi-band spectral vectors at the pixel level between the image fusion result and the upsampled sample multispectral image: normalize the spectral vectors of the image fusion result and the sample multispectral image to a unit hypersphere, and use the angular similarity (i.e., cosine similarity) between the spectral vectors of the two images to measure the consistency of the spectral shape and amplitude of the two images.

[0051] Among them, the spectral loss function The lower the value, the more consistent the image fusion result is with the upsampled multispectral image of the sample in terms of spectral features, thereby improving the spectral fidelity of the image fusion result.

[0052] Optionally, the spectral loss function The expression is as follows: ; ; in, Indicates the width of the image fusion result; Indicates the height of the image fusion result; This represents the image fusion result and the upsampled multispectral image of the sample at the pixel level. Cosine similarity at the location; This indicates the image fusion result at the pixel level. Spectral vector at that location; This indicates the multispectral image of the sample after upsampling at the pixel level. The spectral vector at that location.

[0053] In some embodiments, the joint loss function is determined by weighted aggregation operations based on the spatial loss function, the first weighting factor corresponding to the spatial loss function, the spectral loss function, and the second weighting factor corresponding to the spectral loss function; wherein the first weighting factor and the second weighting factor are determined experimentally based on short-cycle training mode.

[0054] Specifically, the joint loss function It is determined by the space loss function and spectral loss function It is obtained by combining the corresponding weighting factors.

[0055] Optionally, joint loss function The expression is as follows: ; ; ; in, Space loss function The corresponding first weighting factor; Spectral loss function The corresponding second weighting factor; and The value of was determined through experiments using a short-cycle training mode.

[0056] In some embodiments, the remote sensing image fusion network is trained based on the following steps: obtaining a training dataset and a test dataset; both the training dataset and the test dataset include multiple sample panchromatic images and multiple sample multispectral images; and training an initial model based on the training dataset until the initial model passes the validation of the test dataset to obtain the remote sensing image fusion network.

[0057] It should be noted that, in this embodiment, the High Resolution Series (GF) satellite dataset is selected as both the training and testing datasets.

[0058] Optionally, the training dataset includes satellite datasets such as GF-1, GF-1B, GF-1C, GF-1D, GF-2, and GF-6 for model training.

[0059] Optionally, the test dataset includes the WorldView-2 (WV-2) satellite dataset to verify the adaptability and generalization ability of the method proposed in the above embodiments under different satellite data.

[0060] Optionally, given that the WorldView-2 (WV-2) satellite dataset provides more spectral bands than the GF series satellite datasets, to ensure the consistency of spectral range, four bands (2, 3, 4, and 7) corresponding to the spectral coverage of the GF series satellite datasets can be selected from the WV-2 satellite dataset for testing. This allows the weights of the remote sensing image fusion network trained on the GF series satellite datasets to be effectively transferred and applied to the WV-2 satellite dataset, achieving stable fusion under cross-satellite conditions.

[0061] Specifically, both the training and testing datasets include multiple sample panchromatic images and multiple sample multispectral images. The initial model is trained using the training dataset until it passes the validation of the testing dataset, thus obtaining the remote sensing image fusion network.

[0062] In some embodiments, acquiring panchromatic and multispectral images includes: acquiring panchromatic and raw multispectral images; the panchromatic and raw multispectral images are obtained by satellite capturing images of the target area; and upsampling the raw multispectral image to obtain the multispectral image.

[0063] Compared with existing technologies, the image fusion method based on joint loss provided in this embodiment has at least the following technical advantages: (1) A spectral hyperspheric aberration (SHDI) loss function is proposed. Compared with the commonly used spectral angle mapping (SAM) loss function, this method normalizes the spectral vectors of high-resolution multispectral images (HRMS) and upsampled multispectral images (UPMS) at the pixel level, and measures the consistency between the two bands by the similarity of the vector angles. This can more accurately reflect the directional similarity between the two in different spectral bands. The spectral hyperspheric aberration loss can avoid measuring the channel-by-channel amplitude difference between high-resolution multispectral images and upsampled multispectral images only at the band level. It focuses more on the directional consistency of the multi-band spectral vectors at the pixel level, which can effectively reduce the deviation caused by the brightness scale change, reduce cross-band imbalance and distortion, and thus maintain higher spectral fidelity and robustness in cross-sensor scenarios.

[0064] (2) By introducing the Multidirectional Consistency (MDC) loss function, this method overcomes the shortcomings of existing methods that rely solely on single-directional loss or gradient loss in terms of spatial detail preservation. This method calculates the directional edge responses of the panchromatic image and the high-resolution multispectral image in four principal directions: 0°, 45°, 90°, and 135°, respectively, and ensures the consistency of their multidirectional structures through correlation constraints. This approach not only more comprehensively preserves the texture and edge details in the panchromatic image but also effectively suppresses false edges and oversharpening during the fusion process, resulting in a significant improvement in both spatial resolution and structural preservation of the final high-resolution multispectral image.

[0065] (3) In terms of loss mechanism design, this invention combines spectral hyperspheric aberration (SHDI) constraints with multi-directional consistency (MDC) constraints for the first time, forming a joint optimization mechanism. This mechanism synergistically constrains the fusion process from both spectral and spatial dimensions, achieving a dynamic balance between spectral consistency and spatial detail preservation. Compared with existing methods that rely solely on a single loss or empirical combinations, the joint loss function proposed in this invention is more in line with the essence of fusion. At the same time, experimental results on different satellite sensor data show that while improving image fusion accuracy, this mechanism has higher generalization ability and robustness, and can meet the needs of high-quality remote sensing image fusion in complex application scenarios.

[0066] The present invention also provides an image fusion apparatus based on joint loss. See also... Figure 4 , Figure 4 This is a schematic diagram of the image fusion device based on joint loss provided by the present invention. In this embodiment, the image fusion device based on joint loss includes an acquisition module 410 and a remote sensing image fusion module 420.

[0067] The acquisition module 410 is used to acquire panchromatic and multispectral images.

[0068] The remote sensing image fusion module 420 is used to input panchromatic images and multispectral images into the remote sensing image fusion network to obtain the target fused image output by the remote sensing image fusion network.

[0069] The remote sensing image fusion network is trained based on sample panchromatic images and sample multispectral images.

[0070] The remote sensing image fusion network is trained based on a joint loss function, which is determined based on the spatial loss function and the spectral loss function.

[0071] The spatial loss function is used to minimize the multidirectional consistency between the sample panchromatic image and the image fusion result.

[0072] The spectral loss function is used to minimize the spectral consistency of sample multispectral images and image fusion results.

[0073] In some embodiments, the spatial loss function is a multi-directional consistency loss function; wherein, the spatial loss function is determined based on the Pearson correlation coefficient and multiple directional gradients, and one directional gradient is obtained by edge extraction in a preset direction from the sample panchromatic image and the image fusion result.

[0074] In some embodiments, the spectral loss function is the spectral hyperspheric aberration loss function; wherein, the spectral loss function is determined based on the similarity between the spectral vector of the sample multispectral image and the spectral vector of the image fusion result on a unit hypersphere.

[0075] In some embodiments, the joint loss function is determined by weighted aggregation operations based on the spatial loss function, the first weighting factor corresponding to the spatial loss function, the spectral loss function, and the second weighting factor corresponding to the spectral loss function; wherein the first weighting factor and the second weighting factor are determined experimentally based on short-cycle training mode.

[0076] In some embodiments, the remote sensing image fusion network is trained based on the following steps: obtaining a training dataset and a test dataset; both the training dataset and the test dataset include multiple sample panchromatic images and multiple sample multispectral images; and training an initial model based on the training dataset until the initial model passes the validation of the test dataset to obtain the remote sensing image fusion network.

[0077] In some embodiments, acquiring panchromatic and multispectral images includes: acquiring panchromatic and raw multispectral images; the panchromatic and raw multispectral images are obtained by satellite capturing images of the target area; and upsampling the raw multispectral image to obtain the multispectral image.

[0078] The present invention also provides an electronic device. Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. The processor 510, communication interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an image fusion method based on joint loss. The image fusion method based on joint loss includes: acquiring panchromatic and multispectral images; inputting the panchromatic and multispectral images into a remote sensing image fusion network to obtain a target fused image output by the remote sensing image fusion network; wherein the remote sensing image fusion network is trained based on sample panchromatic and sample multispectral images; the remote sensing image fusion network is trained based on a joint loss function, which is determined based on a spatial loss function and a spectral loss function; the spatial loss function is used to minimize the multidirectional consistency of the sample panchromatic image and the image fusion result; the spectral loss function is used to minimize the spectral consistency of the sample multispectral image and the image fusion result.

[0079] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the image fusion method based on joint loss provided by the above methods. The image fusion method based on joint loss includes: acquiring a panchromatic image and a multispectral image; inputting the panchromatic image and the multispectral image into a remote sensing image fusion network to obtain a target fused image output by the remote sensing image fusion network; wherein, the remote sensing image fusion network is trained based on sample panchromatic images and sample multispectral images; the remote sensing image fusion network is trained based on a joint loss function, which is determined based on a spatial loss function and a spectral loss function; the spatial loss function is used to minimize the multidirectional consistency of the sample panchromatic image and the image fusion result; the spectral loss function is used to minimize the spectral consistency of the sample multispectral image and the image fusion result.

[0081] This invention also provides a computer program product, comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image fusion method based on joint loss provided by the above methods. The image fusion method based on joint loss includes: acquiring a panchromatic image and a multispectral image; inputting the panchromatic image and the multispectral image into a remote sensing image fusion network to obtain a target fused image output by the remote sensing image fusion network; wherein, the remote sensing image fusion network is trained based on sample panchromatic images and sample multispectral images; the remote sensing image fusion network is trained based on a joint loss function, which is determined based on a spatial loss function and a spectral loss function; the spatial loss function is used to minimize the multidirectional consistency of the sample panchromatic image and the image fusion result; the spectral loss function is used to minimize the spectral consistency of the sample multispectral image and the image fusion result.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A joint loss based image fusion method, characterized in that, include: Acquire panchromatic and multispectral images; The panchromatic image and the multispectral image are input into a remote sensing image fusion network to obtain the target fused image output by the remote sensing image fusion network; The remote sensing image fusion network is trained based on sample panchromatic images and sample multispectral images; The remote sensing image fusion network is trained based on a joint loss function, which is determined based on a spatial loss function and a spectral loss function. The spatial loss function is used to minimize the multidirectional consistency between the sample panchromatic image and the image fusion result; The spectral loss function is used to minimize the spectral consistency between the sample multispectral image and the image fusion result; The spatial loss function is a multi-directional consistency loss function; The spatial loss function is determined based on the Pearson correlation coefficient and multiple directional gradients. One of the directional gradients is obtained by edge extraction in a preset direction from the sample panchromatic image and the image fusion result. The expression for the space loss function is as follows: ； ； in, This represents the space loss function; The Pearson correlation coefficient is used to measure... and Linear correlation between them Indicates calculation and Linear correlation between them; Indicates expansion along the channel dimension; Indicates application orientation The Sober operator is used to calculate the directional gradient; An angle representing the preset direction; The spectral loss function is the spectral hyperspheric aberration loss function; The spectral loss function is determined based on the similarity between the spectral vector of the sample multispectral image and the spectral vector of the image fusion result on a unit hypersphere. The expression for the spectral loss function is as follows: ； ； in, Represents the spectral loss function; This represents the width of the image fusion result; Indicates the height of the image fusion result; This indicates that the image fusion result and the upsampled multispectral image of the sample are represented at the pixel level. Cosine similarity at the location; This indicates that the image fusion result is at the pixel level. Spectral vector at that location; This indicates that the multispectral image of the sample after upsampling processing is at the pixel level. The spectral vector at that location.

2. The image fusion method based on joint loss according to claim 1, characterized in that, The joint loss function is determined by weighted aggregation operation based on the spatial loss function, the first weighting factor corresponding to the spatial loss function, the spectral loss function, and the second weighting factor corresponding to the spectral loss function; The first weighting factor and the second weighting factor were determined based on experiments using a short-cycle training mode.

3. The image fusion method based on joint loss according to claim 1, characterized in that, The remote sensing image fusion network is trained based on the following steps: Obtain a training dataset and a test dataset; both the training dataset and the test dataset include multiple panchromatic images of the samples and multiple multispectral images of the samples. Based on the training dataset, the initial model is trained until the initial model passes the validation of the test dataset, thus obtaining the remote sensing image fusion network.

4. The image fusion method based on joint loss according to claim 1, characterized in that, The acquisition of panchromatic and multispectral images includes: Acquire panchromatic and raw multispectral images; the panchromatic and raw multispectral images are obtained by satellite images of the target area; The original multispectral image is upsampled to obtain the multispectral image.

5. An image fusion apparatus based on joint loss, characterized in that, include: The acquisition module is used to acquire panchromatic and multispectral images; The remote sensing image fusion module is used to input the panchromatic image and the multispectral image into the remote sensing image fusion network to obtain the target fused image output by the remote sensing image fusion network; The remote sensing image fusion network is trained based on sample panchromatic images and sample multispectral images; The remote sensing image fusion network is trained based on a joint loss function, which is determined based on a spatial loss function and a spectral loss function. The spatial loss function is used to minimize the multidirectional consistency between the sample panchromatic image and the image fusion result; The spectral loss function is used to minimize the spectral consistency between the sample multispectral image and the image fusion result; The spatial loss function is a multi-directional consistency loss function; The spatial loss function is determined based on the Pearson correlation coefficient and multiple directional gradients. One of the directional gradients is obtained by edge extraction in a preset direction from the sample panchromatic image and the image fusion result. The expression for the space loss function is as follows: ； ； in, This represents the space loss function; The Pearson correlation coefficient is used to measure... and Linear correlation between them Indicates calculation and Linear correlation between them; Indicates expansion along the channel dimension; Indicates application orientation The Sober operator is used to calculate the directional gradient; An angle representing the preset direction; The spectral loss function is the spectral hyperspheric aberration loss function; The spectral loss function is determined based on the similarity between the spectral vector of the sample multispectral image and the spectral vector of the image fusion result on a unit hypersphere. The expression for the spectral loss function is as follows: ； ； in, Represents the spectral loss function; This represents the width of the image fusion result; Indicates the height of the image fusion result; This indicates that the image fusion result and the upsampled multispectral image of the sample are represented at the pixel level. Cosine similarity at the location; This indicates that the image fusion result is at the pixel level. Spectral vector at that location; This indicates that the multispectral image of the sample after upsampling processing is at the pixel level. The spectral vector at that location.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image fusion method based on joint loss as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image fusion method based on joint loss as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image fusion method based on joint loss as described in any one of claims 1 to 4.

Images