A Self-Supervised Learning-Based SAR and Visible Light Image Fusion Method and System

Abstract

This invention discloses a method and system for fusion of SAR and visible light images based on self-supervised learning. The method includes the following steps: inputting paired SAR and visible light images into a deep image fusion network model based on self-supervised learning to obtain a fused image. This model includes two feature encoding sub-networks, one feature fusion sub-network, and one feature decoding sub-network. The two feature encoding sub-networks are used to extract SAR feature maps and visible light feature maps. The feature fusion sub-network is used to extract gradient features from the two feature maps and concatenate them with the original features along the channel dimension to obtain fused features. Finally, the fused image is obtained by decoding through the feature decoding sub-network. This invention addresses the common problems of poor utilization of source images, low contrast, unclear texture details, and poor visual performance in current algorithms for SAR and visible light image fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a SAR and visible light image fusion method and system based on self-supervised learning. BACKGROUND

[0002] Synthetic Aperture Radar (SAR) and visible light are two common data acquisition methods in the field of remote sensing. Visible light images are visually intuitive, high-resolution, and can capture rich detail features and texture information, making them suitable for human visual interpretation and widely used optical remote sensing data. However, they are affected by weather and lighting conditions and cannot obtain effective images at night or in bad weather conditions. SAR uses microwave imaging and has all-weather, all-day imaging capabilities, unaffected by weather and light conditions, and has significant advantages in penetrating clouds, rain, and vegetation. However, due to the acquisition principle of SAR images, there is speckle noise in the images, making the visual quality poor and the detail information unclear. SAR and visible light image fusion can complement the characteristics of the two types of images, break through the limitations of bad weather and lighting conditions, and make the fused image have important target feature information of SAR images and detail and spectral information of visible light images, improving the classification and target recognition ability of the image.

[0003] Image fusion algorithms can be roughly divided into two categories: traditional methods and deep learning-based methods. Traditional methods include multi-scale transformation, saliency detection, subspace, and sparse representation-based methods. In recent years, deep learning has made many outstanding achievements in the field of computer vision, and relevant researchers have also applied it to the field of image fusion, developing a variety of excellent methods.

[0004] Currently, many algorithms have emerged in the field of multi-source image fusion, especially in the field of infrared and visible light image fusion, which can obtain fused images with good contrast and clear details. However, there is relatively less research in the field of SAR and visible light image fusion, and the current algorithms for SAR and visible light image fusion generally have poor utilization of source images, low contrast of fusion results, unclear texture detail information, and poor visual performance. SUMMARY

[0005] The technical problem to be solved by the present application: In view of the above problems of the prior art, the present application provides a SAR and visible light image fusion method and system based on self-supervised learning, which aims to solve the problem of poor utilization of source images, low contrast of fusion results, unclear texture detail information, and poor visual performance of the current algorithms for SAR and visible light image fusion.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is:

[0007] A SAR and visible light image fusion method based on self-supervised learning, comprising the following steps: inputting a pair of SAR images and visible light images into a deep image fusion network model based on self-supervised learning to obtain a fusion image , the deep image fusion network model comprising two feature encoding sub-networks, a feature fusion sub-network and a feature decoding sub-network, the two feature encoding sub-networks being respectively used for extracting features from the SAR images and visible light images to obtain SAR feature maps and visible light feature maps , the feature fusion sub-network being used for gradient feature extraction on the SAR feature maps and visible light feature maps and channel dimension splicing with the original SAR feature maps and visible light feature maps to obtain fusion features , and the feature decoding sub-network being used for decoding the fusion features to obtain the fusion image .

[0008] Optionally, the feature encoding sub-network is composed of a 3x3 convolution, a nonlinear activation function ReLU and three residual modules ResConv connected in sequence, and the input and output between the first two residual modules ResConv are connected in a way of jump splicing, the residual module ResConv comprising two 3x3 convolutions, a 1x1 convolution layer, a jump connection and two nonlinear activation functions ReLU, the features input into the residual module ResConv being divided into two branches, one of which is sequentially processed by a 3x3 convolution, a first nonlinear activation function ReLU and a 3x3 convolution, and the other of which is sequentially processed by a 1x1 convolution layer, and the outputs of the two branches are connected by jump connection and then activated by a second nonlinear activation function ReLU to obtain the output features of the residual module ResConv.

[0009] Optionally, the feature fusion sub-network comprises two fusion branches, a gradient feature construction module F_cons, a splicing module, a 3x3 convolution and a nonlinear activation function ReLU, the two fusion branches each comprising a 3x3 convolution and a gradient feature extraction module F_grad for extracting gradient features by using a Laplacian operator, the SAR feature maps and visible light feature maps input into the feature fusion sub-network entering one fusion branch respectively to obtain corresponding gradient information and , the gradient feature construction module F_cons combines the SAR feature map and the visible light feature map , gradient information and constructs the gradient information feature map of the two images , the gradient information feature map of the two images and the original SAR feature map and the visible light feature map After being spliced in the channel dimension by the splicing module, the fusion feature is obtained through 3x3 convolution and nonlinear activation function ReLU .

[0010] Optionally, the gradient feature extraction module F_grad extracts gradient features using a Laplacian operator, which means that a Laplacian operator with a size of 3x3, a center weight of -1, and a peripheral weight of 1 / 8 is used to filter the feature map to extract gradient features in the image.

[0011] Optionally, the gradient feature construction module F_cons combines the SAR feature map and the visible light feature map , gradient information and constructs the gradient information feature map of the two images The functional expression of the gradient feature construction module F_cons is:

[0012] ,

[0013] wherein, is a nonlinear activation function sigmoid, which is used to limit the output range to [0, 1], represents the part of the visible light feature map with large visible light gradient and small SAR gradient at the corresponding position, represents the part of the SAR feature map with large SAR gradient and small visible light gradient at the corresponding position.

[0014] Optionally, the feature decoding subnetwork includes two 3x3 convolutions and one nonlinear activation function ReLU, and the fusion feature input into the feature decoding subnetwork is first processed by the first 3x3 convolution, then activated by the nonlinear activation function ReLU, and then processed by the second 3x3 convolution to generate the fusion image .

[0015] Optionally, when the deep image fusion network model based on self-supervised learning is trained by self-supervised learning, the functional expression of the total loss function used is:

[0016] ,

[0017] in, For the total loss function, , , The hyperparameters used to balance the three loss functions, This is a key information loss technique used to preserve relatively more important information in the input image; Gradient loss is used to preserve the edge and texture information of SAR and visible light images in the fusion result; Pixel loss is used to make it easier to retain the relatively higher pixel values ​​in the SAR and visible light images in the fused image;

[0018] ,

[0019] ,

[0020] ,

[0021] ,

[0022] ,

[0023] ,

[0024] ,

[0025] in, For similarity function, The fused image output by the deep image fusion network model. To calculate the mean square error, , and As an intermediate feature, For normalization operations, To increase contrast; To find the locations where the pixel values ​​differ significantly between the two images, if the difference is large... The pixel value at the corresponding position is 1; otherwise, it is 0. To extract gradients, To calculate the mean absolute error, To find the maximum value.

[0026] Furthermore, the present invention also provides a SAR and visible light image fusion system based on self-supervised learning, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the SAR and visible light image fusion method based on self-supervised learning.

[0027] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the self-supervised learning-based SAR and visible light image fusion method by a processor.

[0028] Furthermore, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the self-supervised learning-based SAR and visible light image fusion method via a processor.

[0029] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The method of the present invention includes inputting paired SAR images and visible light images into a deep image fusion network model based on self-supervised learning to obtain a fused image. This model includes two feature encoding sub-networks, one feature fusion sub-network, and one feature decoding sub-network. The two feature encoding sub-networks are used to extract SAR feature maps and visible light feature maps. The feature fusion sub-network is used to extract gradient features from the two feature maps and concatenate them with the original features along the channel dimension to obtain fused features. The fused image is then decoded by the feature decoding sub-network to obtain the fused image. The present invention uses the feature fusion sub-network to extract SAR feature maps... and visible light feature map Gradient feature extraction is performed and compared with the original SAR feature map. and visible light feature map Concatenation is performed along the channel dimension to obtain fusion features. This results in high contrast, rich texture details, and excellent visual performance in the fusion results. It can solve the problems that current algorithms used for SAR and visible light image fusion generally have, such as poor utilization of source images, low contrast, unclear texture details, and poor visual performance in the fusion results. It has the advantages of high fusion efficiency, high contrast of fusion results, rich texture details, and good visual effects.

[0030] 1. The efficient fusion method of SAR and visible light images based on self-supervised learning of the present invention uses an encoding-fusion-decoding structure as the backbone network. The input image is first extracted by the feature encoding sub-network to extract detailed information and deep features, and then the gradient feature map is generated by the feature fusion sub-network and concatenated with the original feature map in the channel dimension to form the fused feature. Finally, the fused image is reconstructed by the feature decoding sub-network. The fusion result has high contrast, rich texture detail information and excellent visual performance.

[0031] 2. The method of the present invention consists of two feature encoding networks, one feature fusion network and one feature decoding network. The network structure is simple and has a low number of parameters. It is a lightweight network that uses a redundant structure design, resulting in low computational training cost and high efficiency.

[0032] 3. This invention can further utilize a hybrid loss function consisting of three types of losses during the network training phase, wherein the key information loss... Gradient loss is used to preserve relatively more important information in the input image. Used to preserve the edge and texture information of SAR and visible light images in the fusion result; Pixel loss This method is used to make it easier to retain the relatively higher pixel values ​​in SAR and visible light images in the fused image, thereby effectively preserving the gradient and intensity information of the source image, and further resulting in high contrast, rich texture details, and excellent visual performance in the fused result. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the network structure of the deep image fusion network model in an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the network structure of the feature coding subnetwork in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the network structure of the feature fusion subnetwork in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the network structure of the feature decoding subnetwork in an embodiment of the present invention.

[0037] Figure 5 The following is a quantitative result image of the WHU-OPT-SAR dataset fusion in an embodiment of the present invention, wherein: (a) is a SAR image; (b) is a visible light image; (c) is a DenseFuse fused image; (d) is a DIF-Net fused image; (e) is a DWT fused image; (f) is a DDCT fused image; and (g) is a fused image obtained by the method of this embodiment. Detailed Implementation

[0038] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0039] like Figure 1 As shown, the SAR and visible light image fusion method based on self-supervised learning in this embodiment includes the following steps: fusing paired SAR images... and visible light images Input a self-supervised learning-based deep image fusion network model to obtain a fused image. The deep image fusion network model includes two feature encoding subnetworks, one feature fusion subnetwork, and one feature decoding subnetwork. The two feature encoding subnetworks are used to process SAR images... and visible light images SAR feature map is obtained by extracting features. and visible light feature map The feature fusion subnetwork is used for SAR feature maps and visible light feature map Gradient feature extraction is performed and compared with the original SAR feature map. and visible light feature map Concatenation is performed along the channel dimension to obtain fusion features. The feature decoding subnetwork is used for fused features. Decode to obtain the fused image .

[0040] The acquired SAR and visible light images are paired, meaning they were acquired by different sensors at the same or similar shooting distance, angle, and time for the same area. The acquired multi-source images need to be registered and scaled to the same resolution. This embodiment uses paired SAR images... and visible light images Input a self-supervised learning-based deep image fusion network model to obtain a fused image. Previously, it also included registering and scaling pairs of raw SAR images and visible light images to the same resolution size. The pairs of raw SAR images and visible light images refer to images acquired by synthetic aperture radar and visible light sensors at the same or similar shooting distance, angle, and time for the same area.

[0041] like Figure 2 As shown, the feature encoding subnetwork in this embodiment consists of a 3×3 convolution (Conv), a non-linear activation function ReLU, and three residual modules ResConv connected in series. The input and output of the first two residual modules ResConv are connected by a skip connection. The residual module ResConv includes two 3×3 convolutions, a 1×1 convolutional layer, skip connections, and two non-linear activation functions ReLU. The features input to the residual module ResConv are divided into two branches. One branch is processed by a 3×3 convolution, the first non-linear activation function ReLU, and another 3×3 convolution in sequence. The other branch is processed by a 1×1 convolutional layer in sequence. The outputs of the two branches are connected by a skip connection and then activated by the second non-linear activation function ReLU to obtain the output features of the residual module ResConv. The above structure can improve the feature extraction performance of the encoding network.

[0042] likeFigure 3 As shown, the feature fusion subnetwork in this embodiment includes two fusion branches, a gradient feature construction module F_cons, a concatenation module, a 3×3 convolution, and a non-linear activation function ReLU. Each of the two fusion branches includes a 3×3 convolution and a gradient feature extraction module F_grad that uses the Laplacian operator to extract gradient features. The SAR feature map input to the feature fusion subnetwork... and visible light feature map Each path enters a fusion branch to obtain the corresponding gradient information. and The gradient feature construction module F_cons combines SAR feature maps and visible light feature map gradient information and Construct gradient information feature maps for two types of images Gradient information feature maps of the two images and the original SAR feature map and visible light feature map After being stitched together in the channel dimension by the stitching module, the fused features are obtained by 3×3 convolution and the non-linear activation function ReLU. .

[0043] In this embodiment, the gradient feature extraction module F_grad uses the Laplacian operator to extract gradient features, which means using a Laplacian operator with a size of 3×3, a center weight of -1, and a perimeter weight of 1 / 8 to filter the feature map in order to extract gradient features from the image.

[0044] In this embodiment, the gradient feature construction module F_cons combines the SAR feature map. and visible light feature map gradient information and Construct gradient information feature maps for two types of images The function expression is:

[0045] ,

[0046] in, The sigmoid function is a non-linear activation function used to restrict the output range to [0,1]. This indicates the extraction of visible light feature maps. The region with a large visible light gradient and a corresponding small SAR gradient. Indicates the extraction of SAR feature maps The part with a large SAR gradient and a small visible light gradient at the corresponding location.

[0047] likeFigure 4 As shown, the feature decoding subnetwork in this embodiment includes two 3×3 convolutions and a non-linear activation function ReLU. The fused features input to the feature decoding subnetwork... The image is first processed by a 3×3 convolution followed by ReLU activation, and then by a second 3×3 convolution to generate the fused image. .

[0048] like Figure 1 As shown, the overall loss function of the self-supervised learning-based deep image fusion network model in this embodiment during self-supervised learning training is expressed as follows:

[0049] ,

[0050] in, For the total loss function, , , These are hyperparameters used to balance the three loss functions (the values ​​can be chosen according to actual needs, for example, in this embodiment, they are 1, 1, and 4 respectively). This is a key information loss technique used to preserve relatively more important information in the input image; Gradient loss is used to preserve the edge and texture information of SAR and visible light images in the fusion result; The pixel loss is used to make it easier to retain the parts with relatively higher pixel values ​​in the SAR and visible light images in the fused image; in this embodiment, a hybrid loss function consisting of three losses is used, which can further make the fused result have high contrast, rich texture detail information, and excellent visual performance.

[0051] The calculation function expressions for the above three loss functions are as follows:

[0052] ,

[0053] ,

[0054] ,

[0055] ,

[0056] ,

[0057] ,

[0058] ,

[0059] in, For similarity function, The fused image output by the deep image fusion network model. To calculate the mean square error, , and As an intermediate feature, For normalization operations, To increase contrast; To find the locations where the pixel values ​​differ significantly between the two images, if the difference is large... The pixel value at the corresponding position is 1; otherwise, it is 0. To extract gradients, To calculate the mean absolute error, To find the maximum value.

[0060] To verify the efficient SAR and visible light image fusion method based on self-supervised learning proposed in this embodiment, experiments were conducted on a public dataset. The experimental platform hardware was: 12th Gen Intel(R) Core i7-12700F + NVIDIA GeForce RTX 3060. Five metrics were used as evaluation indicators for the image fusion results: information entropy (EN), standard deviation (SD), average gradient (AG), spatial frequency (SF), and visual fidelity (VIF).

[0061] In this experiment, the WHU-OPT-SAR dataset was used. Collected in Hubei Province, it covers 51,448.56 square kilometers with a resolution of 5 meters and contains 100 pairs of 5556x3704 (pixel) images. The optical images in the dataset are from the GF-1 satellite (2-meter resolution), and the SAR images are from the GF-3 satellite (5-meter resolution). The resolution was uniformly set to 5 meters using bilinear interpolation. In the experiment, the RGB images in the dataset were converted to grayscale images. These grayscale images and the SAR images in the dataset were then cropped into 256x256 non-overlapping pixel blocks, resulting in 29,400 pairs of grayscale and SAR images. Half of these were randomly selected as the training set, and 100 pairs were randomly selected from the remaining half as the test set.

[0062] In the experiment, an ablation experiment was first conducted on the fusion method proposed in this embodiment to verify the effectiveness of the proposed loss function and feature fusion sub-network. The ablation experiment results of the loss function and feature fusion sub-network are shown in Table 1 and Table 2, respectively. The benchmark fusion network was used for comparison with the feature fusion sub-network. This network has two fusion branches, each consisting of a 3x3 convolution. The features extracted by the fusion branches are concatenated along the channel dimension, and then passed through a 3x3 convolution and a ReLU activation function to obtain the fused features.

[0063] Table 1: Ablation Experiment Results of the Loss Function of the Method in this Embodiment

[0064]

[0065] As can be seen from Table 1, each of the three loss functions L1, L2, and L3 proposed in this embodiment has its own advantages and disadvantages in the five indicators when used alone. The overall performance of the hybrid loss function, which is a combination of the three loss functions, is the best.

[0066] Table 2: Experimental Results of Feature Fusion Subnetwork Ablation in this Embodiment

[0067]

[0068] As can be seen from Table 2, the feature fusion subnetwork in the SAR and visible light image fusion method based on self-supervised learning proposed in this embodiment outperforms the benchmark fusion network in terms of EN, SD, AG, SF, and VIF indices.

[0069] Subsequently, the fusion method proposed in this embodiment is compared with four existing image fusion methods: DenseFuse, DIF-Net, DWT, and DDCT. The experimental results are shown in Table 3.

[0070] Table 3: Comparative experimental results of the method in this embodiment and four existing methods

[0071]

[0072] As can be seen from Table 3, the SAR and visible light image fusion method based on self-supervised learning proposed in this embodiment demonstrates excellent performance on the WHU-OPT-SAR dataset. It outperforms other methods in terms of EN, SD, SF, and VIF metrics, and is second only to the DIF-Net method in terms of AG metrics.

[0073] Figure 5 The images shown are quantitative results of WHU-OPT-SAR dataset fusion in this embodiment of the invention, where: (a) is a SAR image; (b) is a visible light image; (c) is a DenseFuse fused image; (d) is a DIF-Net fused image; (e) is a DWT fused image; (f) is a DDCT fused image; and (g) is a fused image obtained using the method of this embodiment. Figure 5 As can be seen, the fused image obtained by the method proposed in this embodiment has the best edge preservation effect, and can complementarily utilize the gradient advantage information of different source images, resulting in better visual effects. Furthermore, the model parameter size in this embodiment is 0.22M, and the FLOPs for inference using the test set are 14187.23M, with FPS of 80.4.

[0074] In summary, this embodiment's SAR and visible light image fusion method based on self-supervised learning uses an encoding-fusion-decoding structure as its backbone network. The feature encoding subnetwork consists of convolutional layers, nonlinear activation functions, and residual modules with skip connections, effectively preserving rich feature information while avoiding network parameter redundancy. The feature fusion subnetwork extracts gradient features from the input feature map and concatenates them with the original feature map along the channel dimension to generate fused features, effectively preserving edge information in the image. Finally, the feature decoding subnetwork reconstructs the fused features into a fused image. Furthermore, a hybrid loss function consisting of key information loss, gradient loss, and pixel loss is designed to guide backpropagation, where the key information loss... Gradient loss is used to preserve relatively more important information in the input image. Used to preserve the edge and texture information of SAR and visible light images in the fusion result; Pixel loss This method aims to better preserve the relatively higher pixel values ​​in SAR and visible light images in the fused image, thereby effectively retaining the gradient and intensity information of the source images. This results in a fused image with high contrast, rich texture detail, and excellent visual performance. The SAR and visible light image fusion method based on self-supervised learning in this embodiment has the advantages of high fusion efficiency, high contrast, rich texture detail, and good visual effects.

[0075] Furthermore, this embodiment also provides a SAR and visible light image fusion system based on self-supervised learning, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the SAR and visible light image fusion method based on self-supervised learning.

[0076] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the self-supervised learning-based SAR and visible light image fusion method by a processor.

[0077] Furthermore, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the self-supervised learning-based SAR and visible light image fusion method via a processor.

[0078] Those skilled in the art will understand that the technical solutions provided by embodiments of the present invention may take the form of methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A SAR and optical image fusion method based on self-supervised learning, characterized in that, comprising the following steps: inputting a pair of SAR images and visible light images into a deep image fusion network model based on self-supervised learning to obtain a fused image , the deep image fusion network model comprising two feature encoding sub-networks, a feature fusion sub-network and a feature decoding sub-network, the two feature encoding sub-networks being respectively used for extracting features from the SAR images and visible light images to obtain SAR feature maps and visible light feature maps , the feature fusion sub-network being used for gradient feature extraction on the SAR feature maps and visible light feature maps and concatenation with the original SAR feature maps and visible light feature maps in a channel dimension to obtain fused features , and the feature decoding sub-network being used for decoding the fused features to obtain the fused image ; The feature fusion sub-network comprises two fusion branches, a gradient feature construction module F_cons, a splicing module, a 3x3 convolution, and a nonlinear activation function ReLU, the two fusion branches each comprise a 3x3 convolution and a gradient feature extraction module F_grad using a Laplacian operator to extract gradient features, and the SAR feature map and the visible light feature map respectively enter one fusion branch to obtain corresponding gradient information and The gradient feature construction module F_cons combines the SAR feature map and the visible light feature map , the gradient information and to construct a gradient information feature map of the two images , the gradient information feature map of the two images and the original SAR feature map and the visible light feature map are spliced in the channel dimension after the splicing module, and then subjected to 3x3 convolution and nonlinear activation function ReLU to obtain the fusion feature ; The gradient feature construction module F_cons combines the SAR feature map and visible light feature map gradient information and Construct gradient information feature maps for two types of images The function expression is: ， wherein, is a non-linear activation function sigmoid, used to limit the output range to [0, 1], represents extracting the visible light feature map in which the visible light gradient is large and the SAR gradient at the corresponding position is small, represents extracting the SAR feature map in which the SAR gradient is large and the visible light gradient at the corresponding position is small.

2. The self-supervised learning based SAR and optical image fusion method according to claim 1, wherein, The feature encoding sub-network is composed of a 3*3 convolution, a nonlinear activation function ReLU, and three residual modules ResConv connected in series, and the input and output between the first two residual modules ResConv are connected in a way of jump splicing, the residual module ResConv includes two 3*3 convolutions, a 1*1 convolution layer, a jump connection and two nonlinear activation functions ReLU, the input features of the residual module ResConv are divided into two branches, one branch is processed by 3*3 convolution, the first nonlinear activation function ReLU and 3*3 convolution in turn, and the other branch is processed by 1*1 convolution layer in turn, and the outputs of the two branches are connected by jump connection and then activated by the second nonlinear activation function ReLU to obtain the output features of the residual module ResConv. 3.The SAR and optical image fusion method based on self-supervised learning according to claim 1, wherein, The gradient feature extraction module F_grad extracts gradient features using a Laplacian operator, which means that a Laplacian operator with a size of 3*3, a center weight of -1 and a peripheral weight of 1 / 8 is used to filter the feature map to extract gradient features in the image.

4. The self-supervised learning based SAR and optical image fusion method of claim 1, wherein, The feature decoding subnetwork includes two 3x3 convolutions and a nonlinear activation function ReLU, and the fusion features input into the feature decoding subnetwork The fusion image is generated through the first 3x3 convolution, the nonlinear activation function ReLU, and the second 3x3 convolution .

5. The self-supervised learning based SAR and optical image fusion method of claim 1, wherein, The total loss function used in the self-supervised learning training of the deep image fusion network model based on self-supervised learning has a function expression as follows: ， wherein, is the total loss function, , , is a hyper-parameter for balancing the three loss functions, is the key information loss, which is used to preserve relatively more important information in the input image; is the gradient loss, which is used to preserve the edge and texture information of the SAR and optical images to the fusion result; is the pixel loss, which is used to make the parts with relatively higher pixel values in the SAR and optical images more likely to be preserved to the fusion image; ， ， ， ， ， ， ， wherein, is a similarity function, is a fused image output by the depth image fusion network model, is a mean square error, , and is an intermediate feature, is a normalization operation, is a stretch contrast operation; is a position where pixel values in two images are greatly different, if the difference is large then the pixel value of the corresponding position is 1 otherwise 0; is a gradient extraction, is a mean absolute error, is a maximum value.

6. A SAR and optical image fusion system based on self-supervised learning, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to perform the SAR and visible light image fusion method based on self-supervised learning in any one of claims 1-5.

7. A computer-readable storage medium having stored therein a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to perform the SAR and visible light image fusion method based on self-supervised learning in any one of claims 1-5 by the processor.

8. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are programmed or configured to perform the SAR and visible light image fusion method based on self-supervised learning in any one of claims 1-5 by the processor.

Images