A Joint Interpretation Method for Optical-SAR Images Based on Multimodal Contrast Learning

Abstract

This invention relates to a joint interpretation method for optical-SAR images based on multimodal contrastive learning, comprising: constructing an optical-SAR image sample set and dividing it into a training set and a test set; performing image enhancement processing on the optical-SAR image sample set; constructing an optical-SAR image feature matching network, and training and optimizing the optical-SAR image feature matching network based on multimodal contrastive learning using the training set; transferring the optical-SAR image feature matching network to the interpretation task for fine-tuning to form an interpretation model; and using the interpretation model to perform joint interpretation and index verification on the test set. According to the scheme of this invention, the effective local information between different modalities of images for the interpretation task can be fully exploited.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field, and particularly relates to an optical-SAR image joint interpretation method based on multi-modal contrast learning. BACKGROUND

[0002] Remote sensing images play an important role in military, surveying and mapping, and forestry due to their wide field of view, fast observation, and comprehensive information acquisition. With the development of imaging technology of various sensors, the quality of remote sensing images is continuously improved, and the types are increasingly diversified, gradually developing towards high resolution and multiple types. In addition to the resolution and accuracy of remote sensing images, how to interpret remote sensing images to obtain timely and accurate information and apply it has become a challenging problem in recent years.

[0003] With the rapid development of deep learning, in the past decade, intelligent interpretation technologies such as ground feature segmentation, scene classification, target detection and recognition based on deep learning have become a frontier research direction in the field of remote sensing image processing, and have made significant breakthroughs. This is mainly due to the powerful feature extraction and representation ability of deep neural networks, which can well map remote sensing observation data into the required geographic features. However, the existing deep learning methods have the following problems in optical-SAR remote sensing image joint interpretation:

[0004] 1) Difficulty in mining multi-modal information. The mainstream remote sensing image interpretation technology is still mainly for processing single modal data, especially optical images, which leads to insufficient intelligent interpretation ability and limited application scenarios. Moreover, most of the fusion researches mainly focus on data-level fusion, and do not improve the joint interpretation ability from the feature level, which cannot fully mine the information of images between different modalities.

[0005] 2) Local features are ignored. The existing multi-modal feature extraction method only focuses on the global information of the image, but ignores the local information of the image, which makes the existing method effective in classification tasks, but still needs to be optimized for pixel-level downstream tasks such as semantic segmentation.

[0006] Due to the huge difference in geometric and radiometric features between optical images and SAR images, a method is needed that can fully mine more effective information between different modal images for interpretation tasks. SUMMARY

[0007] To solve the above technical problems in the prior art, the purpose of the present application is to provide an optical-SAR image joint interpretation method based on multi-modal contrast learning.

[0008] To achieve the above application purpose, the technical scheme of the present application is:

[0009] The embodiment of the present application provides a kind of optical-SAR image joint interpretation method based on multi-modal contrast learning, comprising:

[0010] An optical-SAR image sample set is constructed and divided into a training set and a test set;

[0011] The optical-SAR image sample set is subjected to image enhancement processing;

[0012] An optical-SAR image feature matching network is constructed, and the training set is used to train and optimize the optical-SAR image feature matching network based on multi-modal contrast learning;

[0013] The optical-SAR image feature matching network is transferred to the interpretation task for fine-tuning to form an interpretation model;

[0014] The test set is subjected to joint interpretation using the interpretation model, and the interpretation index is verified.

[0015] According to an aspect of the embodiment of the present application, the optical-SAR image sample set includes a positive sample set and a negative sample set, the positive sample set is composed of optical images and SAR image pairs of the same area, and the negative sample set is composed of optical images and SAR image pairs of different areas.

[0016] According to an aspect of the embodiment of the present application, the image enhancement processing of the optical-SAR image sample set comprises:

[0017] The optical images in the optical-SAR image sample set are subjected to image enhancement in the manner of random cropping, flipping, color distortion, conversion to grayscale and random noise;

[0018] The SAR images in the optical-SAR image sample set are subjected to image enhancement in the manner of random cropping, flipping and random noise.

[0019] According to an aspect of the embodiment of the present application, the optical-SAR image feature matching network includes a feature extraction network, a 1x1 convolution layer, a 4x4 average pooling layer and a linear layer,

[0020] The feature extraction network adopts the backbone network of ResNet50, and is used to extract the features of optical images and SAR images in the optical-SAR image sample set, and output feature maps;

[0021] The feature maps are sequentially projected into n-dimensional common feature representations through the 1x1 convolution layer, the 4x4 average pooling layer and the linear layer;

[0022] The linear layer has a sigmoid activation function.

[0023] According to an aspect of an embodiment of the present application, the training and optimization of the optical-SAR image feature matching network based on multi-modal contrastive learning using the training set comprises:

[0024] Initializing a memory bank for storing global features and local features of optical images and SAR images in the training set, respectively;

[0025] Extracting global features and local features of optical image and SAR image pairs in the training set using the optical-SAR image feature matching network;

[0026] Constructing global loss functions and local loss functions of the optical-SAR image feature matching network;

[0027] Obtaining a loss function based on the global loss functions and the local loss functions, updating parameters of the optical-SAR image feature matching network by minimizing the loss function and combining a stochastic gradient descent optimization algorithm for iteration;

[0028] Updating the memory bank based on dynamically calculated global features and local features of the optical image and SAR image pairs;

[0029] Repeating the above steps to traverse all optical image and SAR image pairs in the training set.

[0030] According to an aspect of an embodiment of the present application, the construction of the global loss functions of the optical-SAR image feature matching network comprises:

[0031] Using cosine similarity as a discriminant function to calculate the similarity of global features of optical image and SAR image pairs, and using a temperature coefficient τ to adjust the concentration of global features in the representation space;

[0032] Fixing the SAR image, enumerating the positive sample pairs and negative sample pairs of the SAR image, and taking them as the similarity of global feature image representation between different modalities to obtain the corresponding loss function

[0033] Fixing the optical image, enumerating the positive sample pairs and negative sample pairs of the optical image, and taking them as the similarity of global feature image representation between different modalities to obtain the corresponding loss function

[0034] The global loss function is obtained as Wherein, X o represents an optical image, and X s represents a SAR image.

[0035] According to an aspect of the embodiment of the present application, the local features of the optical image and the SAR image pair in the training set are extracted by using the optical-SAR image feature matching network, and the method comprises the following steps:

[0036] The local regions of the optical image and the SAR image are intercepted respectively to obtain the local region images corresponding to the optical image and the SAR image;

[0037] The local region images corresponding to the optical image and the SAR image are input into the optical-SAR image feature matching network, and the local features of the optical image and the SAR image are output.

[0038] According to an aspect of the embodiment of the present application, the local loss function of the optical-SAR image feature matching network is constructed, and the method comprises the following steps:

[0039] The cosine similarity is used as a discriminant function to calculate the similarity of the local features of the optical image and the SAR image pair, and the temperature coefficient τ is used to adjust the concentration degree of the local features in the representation space;

[0040] The local region of the SAR image is fixed, the positive sample pair and the negative sample pair of the SAR image are enumerated, and the similarity of the local feature images between different modalities is obtained, thereby obtaining the corresponding loss function

[0041] The local region of the optical image is fixed, the positive sample pair and the negative sample pair of the optical image are enumerated, and the similarity of the local feature images between different modalities is obtained, thereby obtaining the corresponding loss function

[0042] The local loss function is obtained as

[0043] According to an aspect of the embodiment of the present application, the loss function obtained based on the global loss function and the local loss function is:

[0044] L=λ·L global +(1-λ)·L local

[0045] Wherein, λ is a hyperparameter for controlling the proportion of the global loss function L global and the local loss function L local .

[0046] The global feature and the local feature momentum update memory warehouse of the optical image and the SAR image pair are obtained based on the dynamic calculation, and the formula is as follows:

[0047] d=←γ·d t-1 +(1-γ)·d t

[0048] wherein, d t-1 represents the feature before updating, d t represents the current updated feature, memory warehouse d∈{x s , x o , u s , u o}, N is the number of sample pairs in the training set, x o and x s respectively represent the global features of the optical image and the SAR image, u o and u s respectively represent the local features of the optical image and the SAR image, and γ∈[0, 1) is a momentum factor.

[0049] According to an aspect of an embodiment of the present application, the interpretation index includes the intersection over union and recognition accuracy of each type of target, the overall recognition accuracy, the average recognition accuracy, and the average intersection over union.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] According to the scheme of the embodiment of the present application, a joint interpretation framework of optical images and SAR images is constructed, which can effectively improve the general feature extraction capability of optical images and SAR images, thereby improving the accuracy of specific intelligent interpretation tasks of optical-SAR images or SAR images or optical images and widening the application scenarios.

[0052] By adopting a multi-modal contrast learning method combining local and global, and combining image enhancement, feature extraction, global contrast loss, local contrast loss, and memory warehouse to learn the common representation between optical images and SAR images, the feature extraction network pays more attention to local representation, thereby improving the accuracy of pixel-level downstream interpretation tasks. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0054] Figure 1 a flowchart schematically showing an optical-SAR image joint interpretation method based on multi-modal contrast learning according to an embodiment of the present application;

[0055] Figure 2 a schematic diagram showing the overall framework of optical-SAR image joint interpretation according to an embodiment of the present application;

[0056] Figure 3 A schematic representation of a backbone network diagram of ResNet50 disclosed in embodiments of the present application;

[0057] Figure 4 A schematic representation of a local-global combined multi-modal contrastive learning framework diagram disclosed in embodiments of the present application;

[0058] Figure 5 A schematic representation of image pairs with different matching results based on local-global combined multi-modal contrastive learning in a test subset of the MSAW dataset disclosed in embodiments of the present application;

[0059] Figure 6 A semantic segmentation visualization result of a pre-training model based on local-global combined multi-modal contrastive learning disclosed in embodiments of the present application. DETAILED DESCRIPTION

[0060] The description of the embodiments of the present application should be considered in conjunction with the accompanying drawings, which are incorporated into and form a part of the present application. In the drawings, the shapes or thicknesses of the embodiments can be exaggerated and simplified for the purpose of convenience or clarity. Furthermore, parts of the structures in the drawings will be described separately, and it should be noted that the elements not shown or described in the drawings are in the form known to those skilled in the art.

[0061] The description of the embodiments herein, any reference to direction or position, is only for the convenience of description, and cannot be understood as any limitation on the scope of protection of the present application. The following description of the preferred embodiments will refer to combinations of features, which can exist independently or in combination, and the present application is not particularly limited to the preferred embodiments. The scope of the present application is defined by the claims.

[0062] In order to fully exploit the effective information between different modal images for interpretation tasks, the present embodiment discloses an optical-SAR image joint interpretation method based on multi-modal contrastive learning, which extracts common features of optical images and SAR images by maximizing mutual information between different modalities, for pre-training model to better learn image-level representation. Moreover, the supervision information of local region is introduced to learn the representation of local region features, so that the pre-training model can be migrated to downstream interpretation tasks, and better results can be achieved in pixel-level downstream tasks, and the application scenarios are widened.

[0063] The optical-SAR image joint interpretation method and its use process in the SAR image semantic segmentation task are illustrated by taking the MSAW dataset as an example in the present embodiment.

[0064] As Figure 1 and Figure 2As shown, the optical-SAR image joint interpretation method based on multi-modal contrast learning disclosed in the embodiments of the present application specifically comprises the following steps:

[0065] S110, constructing an optical-SAR image sample set and dividing it into a training set and a test set;

[0066] S120, performing image enhancement processing on the optical-SAR image sample set;

[0067] S130, constructing an optical-SAR image feature matching network, and training and optimizing the optical-SAR image feature matching network based on multi-modal contrast learning using the training set;

[0068] S140, migrating the optical-SAR image feature matching network to an interpretation task for fine-tuning to form an interpretation model;

[0069] S150, using the interpretation model to perform joint interpretation on the test set and verifying interpretation indicators.

[0070] In some embodiments, the optical-SAR image sample set in step S110 includes a positive sample set and a negative sample set, the positive sample set is composed of optical image and SAR image pairs of the same area, and the negative sample set is composed of optical image and SAR image pairs of different areas. The above optical-SAR image sample set is composed of N optical image and SAR image matching sample pairs {X s ,X o}, wherein, is a set of SAR images, is a set of optical images. Image pairs from the same area constitute the positive sample set, and image pairs from different areas constitute the negative sample set.

[0071] In this embodiment, the optical-SAR image sample set is an MSAW data set. The MSAW data set combines SAR images and photoelectric images to capture building target labels under various weather conditions for computer vision and deep learning. The MSAW data set contains 3401 groups of multi-modal images, the optical images are provided by Maxar Worldview-2 satellite, and the SAR images are provided by CapellaSpace's four-polarization X-band (HH, HV, VH and VV) aerial sensors. The pixels of each image are 900x900, and the spatial resolution is 0.5m x 0.5m. In the multi-modal image matching task, the first three channels of the SAR image are taken as the input picture, and the MSAW data set is randomly divided into a training set and a test set at a ratio of 1:1.

[0072] In some embodiments, the specific process of performing image enhancement processing on the optical-SAR image sample set in step S120 includes the following steps: performing image enhancement on the optical images in the optical-SAR image sample set by using random cropping, flipping, color distortion, conversion to grayscale image, and random noise; and performing image enhancement on the SAR images in the optical-SAR image sample set by using random cropping, flipping, and random noise. The above image enhancement methods construct spatial and temporal transformations of images, so that the model learns the spatiotemporal invariance of images.

[0073] In the SAR image semantic segmentation task, the size of all input images is scaled to 512x512 in this embodiment.

[0074] In some embodiments, as shown in FIG. 13, the optical-SAR image feature matching network in step S130 includes a feature extraction network, a 1x1 convolutional layer, a 4x4 average pooling layer, and a linear layer. The feature extraction network uses the backbone network of ResNet50 to extract features of the optical images and the SAR images in the optical-SAR image sample set, and outputs feature maps. The feature maps are sequentially projected into an n-dimensional common feature representation through the 1x1 convolutional layer, the 4x4 average pooling layer, and the linear layer. The linear layer has a sigmoid activation function. Figure 3

[0075] In some embodiments, as shown in FIG. 13, the specific process of training and optimizing the optical-SAR image feature matching network based on multi-modal contrastive learning using the training set in step S130 includes the following steps: Figure 4

[0076] S131, initializing a memory bank for storing global features and local features of the optical images and the SAR images in the training set, respectively;

[0077] S132, extracting global features and local features of the optical image and SAR image pairs in the training set using the optical-SAR image feature matching network;

[0078] S133, constructing global loss functions and local loss functions of the optical-SAR image feature matching network;

[0079] S134, obtaining a loss function based on the global loss functions and the local loss functions, updating parameters of the optical-SAR image feature matching network by minimizing the loss function and combining a stochastic gradient descent optimization algorithm;

[0080] S135, updating the memory bank based on the dynamically calculated global features and local features of the optical image and SAR image pairs; ​​

[0081] S136, repeat the above steps S132-S135, traverse all the optical image and SAR image pairs in the training set.

[0082] In this embodiment, the optical image and SAR image mentioned in the above step S130 are all enhanced images after image enhancement processing. For the enhanced optical image and SAR image respectively use the feature extraction network f o (·, θ s ) and f o (·, θ o ) with parameters θ s and θ s extract the features of the optical image and SAR image , respectively, to obtain the global features of the optical image and the global features of the SAR image

[0083] Specifically, the process of constructing the global loss function of the optical-SAR image feature matching network in step S133 includes the following steps:

[0084] For global features z o and z s , the cosine similarity is used as the discriminant function to calculate the similarity of the global features of the optical image and SAR image pair, and the temperature coefficient τ is used to adjust the concentration of the global features in the representation space, as follows,

[0085]

[0086] Wherein, the similarity of the positive sample pair is high, and the discriminant function score is high; while the negative sample pair has low similarity, and the corresponding discriminant function score is also low.

[0087] Construct a global image pair set (y i is a sample pair, where is m negative sample pairs), correctly select the positive sample pair y0 from the set Ω, and construct the loss function as follows:

[0088]

[0089] In order to construct Ω, fix the SAR image Enumerate the positive sample pairs and negative sample pairs of the SAR image, which are used as the similarity of the global feature image representation between different modalities, to obtain the corresponding loss function It can be expressed as:

[0090]

[0091] The above formula regards the SAR image as an anchor point to enumerate the optical image;

[0092] Similarly, the fixed optical image Enumerate the positive and negative sample pairs of the optical image, and regard them as the similarity of the global feature image representation between different modalities, to obtain the corresponding loss function

[0093] Add them together to obtain the global loss function Where, X o represents the set of optical images, X s represents the set of SAR images.

[0094] Specifically, the process of extracting local features of the optical and SAR image pairs in the training set by using the optical-SAR image feature matching network in step S132 includes the following steps:

[0095] The local regions of the optical image x o and the SAR image x s are intercepted respectively, to obtain the corresponding local region maps {p o , p s} of the optical and SAR images;

[0096] The corresponding local region maps of the optical and SAR images are input into the optical-SAR image feature matching network, and the local features of the optical image and the SAR image

[0097] Further, the specific process of constructing the local loss function (similar to the definition of the global loss function) of the optical-SAR image feature matching network in step S133 includes the following steps:

[0098] The cosine similarity is used as the discriminant function to calculate the similarity of the local features of the optical and SAR image pairs, and the temperature coefficient τ is used to adjust the concentration degree of the local features in the representation space;

[0099] The local region of the SAR image is fixed, and the positive and negative sample pairs of the SAR image are enumerated, which are regarded as the similarity of the local feature image representation between different modalities, to obtain the corresponding loss function which can be represented as:

[0100]

[0101] In the above formula, the local region of the fixed SAR image is regarded as an anchor point;

[0102] The local region of the optical image is fixed, and the positive sample pairs and negative sample pairs of the optical image are enumerated as the similarity of the local feature image representation between different modalities, to obtain a corresponding loss function

[0103] The local loss function is obtained by adding the two

[0104] In step S134, the loss function obtained based on the global loss function and the local loss function is:

[0105] L = λ·L global +(1-λ)·L local

[0106] Wherein, λ is a hyperparameter that controls the proportion of the global loss function L global and the local loss function L local . In the specific calculation process, λ takes a constant 0.5.

[0107] The random gradient descent optimization algorithm is iterated for 200 steps, and the batch size is 20. The initial learning rate is set to 0.01. The SAR image and the optical image are encoded into a 128-dimensional subspace.

[0108] In step S135, the global feature and local feature momentum update memory warehouse of the optical image and SAR image pair dynamically calculated at each iteration are updated as follows:

[0109] d = ← γ·d t-1 +(1-γ)·d t

[0110] Wherein, d t-1 represents the feature before updating, d t represents the feature after updating, and the memory warehouse d ∈ {x s , x o , u s , u o}, N is the number of sample pairs in the training set, x o and x s respectively represent the global features of the optical image and the SAR image, u o and u s respectively represent the local features of the optical image and the SAR image, and γ ∈ [0, 1) is a momentum factor. In this embodiment, γ takes 0.5.

[0111] By maintaining a memory bank to store the latent features of each training sample extracted by the backbone network, m negative samples can be quickly and efficiently retrieved from the memory bank to pair with each positive sample without the need to recalculate the features. The momentum update makes the features in the memory bank update smoothly, and the memory bank has the advantage of calculating more negative sample pair losses with a smaller memory cost.

[0112] In some embodiments, the interpretation indicators in step S150 include the intersection over union and recognition accuracy of each category target, the overall recognition accuracy, the average recognition accuracy, and the average intersection over union.

[0113] To verify the effectiveness of the local-global combined multi-modal contrast learning method (LGMC method) of the present application, the optical-SAR image matching task and the SAR image semantic segmentation task are performed on the MSAW dataset. First, based on the contrast learning combined with local loss, a common representation with local information is learned on the matching task of optical images and SAR images, and then the trained backbone model is migrated to the backbone network of the downstream SAR image building segmentation model to improve the segmentation effect, as shown in Figure 5 .

[0114] During training, due to the overall dark of the SAR original image, the image is subjected to percentage linear stretching, and after the image is stretched to [0, 255], it is input to the classic DeepLabv3+ segmentation network. Using the stochastic gradient descent method, each small batch has 2 images, the weight decay is 0.0001 and the momentum is 0.9. In the first 500 iterations, the learning rate is linearly increased to 0.02 using the warm-up strategy, and the model is trained for 120,000 iterations using the polynomial learning rate decay strategy. The size of all input images is scaled to 512x512, and only random flipping and rotation are used as data augmentation.

[0115] After training, the model based on the ResNet50 backbone network uses local-global combined multi-modal contrast learning for feature matching on the MSAW dataset, achieving a matching accuracy of 0.8864, indicating that the LGMC method can learn useful common representations and well predict the relationship between SAR images and optical images.

[0116] After the local-global combined multi-modal contrastive learning is performed on the training data of the MSAW dataset, the global and local of different modal images are learned by the feature extraction network, and the SAR feature extraction backbone network at this time contains the rich texture features from the optical image and the local features of the SAR image. The trained SAR feature extraction network is migrated to the downstream pixel-level semantic segmentation task, used as the initialization model of the feature extraction backbone network in the semantic segmentation model, and fine-tuned on the downstream SAR image semantic segmentation task to complete the training process.

[0117] In the test phase, the No Pretrain (NP), the method based on ImageNet pre-training, the method based on multi-modal contrastive learning (CMC) and the local-global combined multi-modal contrastive learning method (LGMC) of the present application are compared, including the IoU of each category, the accuracy, and the overall precision, the average precision and the average intersection over union on the MSAW dataset. Compared with other methods, the LGMC method achieves the highest precision in each index, and can achieve the highest average intersection over union of 0.8837. These results prove that the local-global combined multi-modal contrastive learning method of the present application can use the optical image and the local features to improve the feature extraction capability of the SAR semantic segmentation network, and even without using more pre-training data and training methods, the effect of semantic segmentation in the SAR image can be significantly improved. Part of the visual results are shown in Figure 6

[0118]

[0119] The serial numbers of the above steps involved in the method of the present application do not mean the order of the execution of the method, and the execution order of each step should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0120] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.​

Claims

1. An optical-SAR image joint interpretation method based on multi-modal contrast learning, comprising: constructing an optical-SAR image sample set and dividing it into a training set and a test set; performing image enhancement processing on the optical-SAR image sample set; constructing an optical-SAR image feature matching network, and training and optimizing the optical-SAR image feature matching network based on multi-modal contrast learning using the training set; migrating the optical-SAR image feature matching network to the interpretation task for fine-tuning to form an interpretation model; using the interpretation model to perform joint interpretation on the test set and verifying interpretation indicators; the optical-SAR image sample set includes a positive sample set and a negative sample set, the positive sample set is composed of optical image and SAR image pairs of the same area, and the negative sample set is composed of optical image and SAR image pairs of different areas; the training and optimization of the optical-SAR image feature matching network based on multi-modal contrast learning using the training set comprises: initializing a memory bank for storing global features and local features of optical images and SAR images in the training set, respectively; extracting global features and local features of optical image and SAR image pairs in the training set using the optical-SAR image feature matching network; constructing a global loss function and a local loss function of the optical-SAR image feature matching network; obtaining a loss function based on the global loss function and the local loss function, minimizing the loss function, and iteratively updating parameters of the optical-SAR image feature matching network combined with a stochastic gradient descent optimization algorithm; updating the memory bank based on dynamically calculated global features and local features of the optical image and SAR image pairs; repeating the above steps to traverse all optical image and SAR image pairs in the training set; wherein the construction of the global loss function of the optical-SAR image feature matching network comprises: using cosine similarity as a discriminant function to calculate the similarity of global features of optical image and SAR image pairs, and using a temperature coefficient τ to adjust the concentration of global features in the representation space; Fix the SAR image, enumerate the positive sample pairs and negative sample pairs of the SAR image, and take the similarity of the global feature image representation between different modalities as the corresponding loss function Fix the optical image, enumerate the positive and negative sample pairs of the optical image, and take the similarity of the global feature image representation between different modalities as the loss function The global loss function is obtained as where X o represents the optical image, X s represents the SAR image.

2. The method of claim 1, wherein, the image enhancement processing of the optical-SAR image sample set comprises: performing image enhancement on optical images in the optical-SAR image sample set in the form of random cropping, flipping, color distortion, conversion to grayscale, and random noise; performing image enhancement on SAR images in the optical-SAR image sample set in the form of random cropping, flipping, and random noise.

3. The method of claim 1, wherein, The optical-SAR image feature matching network comprises a feature extraction network, a 1x1 convolution layer, a 4x4 average pooling layer, and a linear layer, the feature extraction network adopts a ResNet50 backbone network for extracting features of optical images and SAR images in the optical-SAR image sample set and outputting feature maps; the feature maps are projected into n-dimensional common feature representations in turn through the 1x1 convolution layer, the 4x4 average pooling layer, and the linear layer; the linear layer has a sigmoid activation function.

4. The method of claim 1, wherein, The local features of the optical image and the SAR image pair in the training set are extracted by using the optical-SAR image feature matching network, and the method comprises the following steps: Respectively intercepting the local area of the optical image and the SAR image to obtain the local area graph corresponding to the optical image and the SAR image; Inputting the local area graph corresponding to the optical image and the SAR image into the optical-SAR image feature matching network to output the local features of the optical image and the SAR image.

5. The method of claim 4, wherein, The local loss function of the optical-SAR image feature matching network is constructed, and the method comprises the following steps: Using cosine similarity as a discriminant function to calculate the similarity of the local features of the optical image and the SAR image pair, and using a temperature coefficient τ to adjust the concentration of the local features in the representation space; Fixing a local region of a SAR image, enumerating positive sample pairs and negative sample pairs of the SAR image, and taking the similarity of local feature image representations between different modalities as a loss function Fix the local region of the optical image, enumerate the positive and negative sample pairs of the optical image, and take the similarity of the local feature image representation between different modalities as the corresponding loss function The local loss function is obtained as 6. The method of claim 5, wherein, The loss function obtained based on the global loss function and the local loss function is: L = λ · L global + (1 - λ) · L local wherein λ is a control parameter of the global loss function L global and the local loss function L local the proportion of the hyperparameter; The global feature and the local feature momentum of the optical image and the SAR image pair are updated based on the dynamic calculation, and the memory warehouse is as follows: d = d t-1 + (1 - y) - d t where d t-1 represents the feature before updating, d t represents the current updated feature, and the memory warehouse M= d∈{x s ,x o ,u s ,u o}, N is the number of sample pairs in the training set, x o and x s respectively represent the global features of the optical image and the SAR image, u o and u s respectively represent the local features of the optical image and the SAR image, and β∈[0,1) is the momentum factor.

7. The method of claim 1, wherein, The interpretation index comprises the intersection over union and the correct recognition rate of each type of target, the overall recognition accuracy, the average recognition accuracy and the average intersection over union.

Images