A sar-visible light-infrared cross-modal image matching method

Abstract

The application relates to a SAR-visible light-infrared cross-modal image matching method, and relates to the technical field of image matching.The application is aimed at solving the problem of low matching precision and efficiency of the existing cross-modal image matching method.The application comprises the following steps: obtaining a preprocessed to-be-matched vehicle image and a preprocessed candidate matching vehicle image set; inputting the preprocessed to-be-matched vehicle image and the preprocessed candidate matching vehicle image set into an image matching model to obtain a to-be-matched vehicle image feature vector and a candidate matching vehicle image feature vector; obtaining the Euclidean distance between each candidate matching vehicle image feature vector and the to-be-matched vehicle image feature vector, and sorting the candidate matching vehicle images corresponding to the candidate matching vehicle image feature vectors from large to small according to the Euclidean distance; and obtaining a vehicle image matching result according to a preset similarity threshold value.The application is used for cross-modal vehicle image matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image matching, in particular to a SAR-optical-infrared cross-modal image matching method. BACKGROUND

[0002] With the rapid development of remote sensing technology, the types and quantities of remote sensing equipment and sensors are increasing, and the acquired remote sensing image data is showing explosive growth. These image data not only include traditional visible light images, but also cover infrared images, synthetic aperture radar images and other modalities. Multi-modal remote sensing images provide multi-dimensional feature information for the same target or region, greatly enhancing human ability to observe and analyze the earth. Vehicle image matching aims to confirm the identity of the same vehicle by comparing vehicle features in different images. This technology can efficiently utilize massive remote sensing image resources and has wide application prospects in intelligent transportation, urban management, public safety and other aspects.

[0003] Traditional vehicle image matching methods mainly rely on visible light images captured by fixed cameras. These methods perform well under good lighting conditions, but in complex real-world scenarios, especially in low-light or adverse weather conditions, the effect is significantly reduced. Synthetic aperture radar (SAR) as an active remote sensing technology has the advantage of not relying on external light sources and can work in all-weather, various weather conditions. SAR images can penetrate clouds and vegetation and obtain detailed information of the ground, so they can complement the information between traditional optical images. However, SAR images and visible light images have significant differences in imaging mechanisms, resulting in low cross-modal image matching accuracy. In addition, the significant differences in imaging mechanisms between SAR images and visible light images also affect information fusion efficiency, ultimately leading to low image matching efficiency. SUMMARY

[0004] The present application aims to solve the problem of low matching accuracy and efficiency of existing cross-modal image matching methods, and proposes a SAR-optical-infrared cross-modal image matching method.

[0005] A SAR-optical-infrared cross-modal image matching method, the specific process is as follows:

[0006] Step one, obtaining a to-be-matched vehicle image and a candidate matching vehicle image set, and preprocessing the to-be-matched vehicle image and the candidate matching vehicle image to obtain a preprocessed to-be-matched vehicle image and a preprocessed candidate matching vehicle image set;

[0007] The to-be-matched vehicle image and the candidate matching vehicle image belong to two different modal images;

[0008] The modalities of the images include: visible light images, synthetic aperture radar images and infrared images;

[0009] Step two, input the pre-processed to-be-matched vehicle image and the pre-processed candidate matching vehicle image set into the image matching model respectively, and obtain the to-be-matched vehicle image feature vector and the candidate matching vehicle image feature vector respectively;

[0010] Step three, combine all the candidate matching vehicle image feature vectors to obtain a candidate matching vehicle image feature vector group, obtain the Euclidean distance between each candidate matching vehicle image feature vector in the candidate matching vehicle image feature vector group and the to-be-matched vehicle image feature vector, and sort the candidate matching vehicle images corresponding to the candidate matching vehicle image feature vectors according to the Euclidean distance from small to large, to obtain a vehicle image matching result;

[0011] Wherein, the smaller the Euclidean distance value, the more similar the to-be-matched vehicle image and the current candidate matching vehicle image.

[0012] Step four, compare the Euclidean distance between each candidate matching vehicle image feature vector and the to-be-matched vehicle image feature vector with the preset similarity threshold value, if the Euclidean distance is less than the preset similarity threshold value, it indicates that the current candidate matching vehicle image and the to-be-matched vehicle image are the same target vehicle.

[0013] Further, the pre-processing of the to-be-matched vehicle image and the candidate matching vehicle image in step one is specifically:

[0014] The vehicle target is cropped in the to-be-matched vehicle image and the candidate matching vehicle image respectively, and the vehicle target image is unified to the same size;

[0015] The to-be-matched vehicle image and the candidate matching vehicle image are all three-channel images.

[0016] Further, the image matching model is obtained by the following way:

[0017] S1, pre-processing the airborne remote sensing image data to obtain a multi-modal vehicle target image set, specifically:

[0018] Obtain the airborne remote sensing image data, detect and crop the vehicle target in the airborne remote sensing image data to obtain a vehicle target image, then take the vehicle class as the label of the vehicle target image, and all the vehicle target images and the corresponding labels form a multi-modal vehicle target image set;

[0019] The airborne remote sensing image data includes: visible light images, synthetic aperture radar images and infrared images;

[0020] S2, obtain a training set and a test set by using a multi-modal vehicle target image set, and train and test the modal implicit mutual learning network CAIL by using the training set and the test set, to obtain an image matching model.

[0021] Further, in the S2, the training set and the test set are obtained by using the multi-modal vehicle target image set, and the modal implicit mutual learning network CAIL is trained and tested by using the training set and the test set, to obtain the image matching model, specifically:

[0022] S201, pre-process the multi-modal vehicle target image set to obtain a multi-modal aerial image data set:

[0023] Firstly, the SAR images in the multi-modal vehicle target image set are converted into jpg format;

[0024] Then, the visible light images, the infrared images and the SAR images converted into the format are cropped into a unified preset size;

[0025] Then, data augmentation is performed on all vehicle target images; the data augmentation includes random horizontal flipping and random erasing;

[0026] Finally, all vehicle target images are converted into numpy format files and stored;

[0027] S202, divide the multi-modal aerial image data set into a training set and a test set;

[0028] S203, extract a training group from the training set, take one training group as one training batch, train the modal implicit mutual learning network CAIL by using the training set, and obtain the trained modal implicit mutual learning network CAIL;

[0029] S204, test the trained modal implicit mutual learning network CAIL by using the test set, to obtain an image matching model.

[0030] Further, in the S203, the training group is extracted from the training set, and one training group is taken as one training batch, which is obtained by the following way, specifically:

[0031] Randomly select two different modalities A and B, select b vehicle categories, and extract a vehicle target image a of each vehicle category in the training set, wherein a vehicle target image is modal A, a vehicle target image is modal B, so as to form a training group, and one training group is taken as one training batch;

[0032] The modal A image and the modal B image are images of any two different modalities in the visible light image, the synthetic aperture radar image and the infrared image.

[0033] Further, the modal implicit mutual learning network CAIL in S203 comprises: a modal A image processing branch, a modal B image processing branch, a feature merging module, a multi-modal implicit mutual learning module MTL, a deep feature extraction module, and a feature vector output module;

[0034] The modal A image processing branch comprises: a modal A first stage processing unit STAGE1 and a modal A second stage processing unit STAGE2.

[0035] The input of the modal A first stage processing unit STAGE1 in the modal A image processing branch is any modal vehicle target image in the training set, the output of the modal A first stage processing unit STAGE1 is input to the modal A second stage processing unit STAGE2, and the output of the modal A second stage processing unit STAGE2 is a vector f a ;

[0036] The modal A first stage processing unit STAGE1 comprises: a first convolutional layer and a first LN layer.

[0037] The first convolutional layer is a convolutional layer with a kernel size of 4x4 and a stride of 4; the first convolutional layer is used to obtain a vehicle target feature map.

[0038] The first LN layer is used to normalize the vehicle target feature map to obtain a multi-modal feature map.

[0039] The modal A second stage processing unit STAGE2 comprises: a first ConvNeXt Block subunit, a first down-sampling layer, and a second ConvNeXt Block subunit.

[0040] The first ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series.

[0041] Each ConvNeXt Block block comprises: a second convolutional layer, a third convolutional layer, a first activation function layer, a fourth convolutional layer, a fifth convolutional layer, a Layer Scale operation layer, and a sixth convolutional layer.

[0042] The second convolutional layer is a convolutional layer with a kernel size of 7x7; the output of the second convolutional layer is input to the third convolutional layer.

[0043] The third convolutional layer is a convolutional layer with a kernel size of 1x1; the output of the third convolutional layer is input to the first activation function layer.

[0044] The first activation function layer is a GELU activation function; the output of the first activation function layer is the input of the fourth convolutional layer.

[0045] The fourth convolution is a convolution kernel size of 1x1 convolution layer; the output of the fourth convolution layer is the input of the fifth convolution layer;

[0046] The fifth convolution layer is a convolution kernel size of 1x1 convolution layer; the output of the fifth convolution layer is the input of the LayerScale operation layer;

[0047] The output of the Layer Scale operation layer is the input of the sixth convolution layer;

[0048] The sixth convolution layer is a convolution kernel size of 2x2 and a step size of 2 convolution layer, and the output feature map of the sixth convolution layer is the output feature map of the ConvNeXt Block block;

[0049] The second ConvNeXt Block subunit is composed of 3 ConvNeXt Block blocks in series; the second ConvNeXt Block subunit has the same structure as the first ConvNeXt Block subunit; the input of the second ConvNeXt Block subunit is the output of the first down-sampling layer;

[0050] The modality B image processing branch includes: a modality B first stage processing unit STAGE1' and a modality B second stage processing unit STAGE2';

[0051] The input of the modality B first stage processing unit STAGE1' in the modality B image processing branch is the vehicle target image of another modality in the training set, and the output of the modality B first stage processing unit STAGE1' is the input of the modality B second stage processing unit STAGE2', and the output of the modality B second stage processing unit STAGE2' is the feature vector f b ;

[0052] The modality B first stage processing unit STAGE1' has the same structure as the modality A first stage processing unit STAGE1;

[0053] The modality B second stage processing unit STAGE2' has the same structure as the modality A second stage processing unit STAGE2;

[0054] The feature merging module is used to merge the feature vector f a output by the modality A image processing branch and the feature vector f b output by the modality B image processing branch, to obtain a feature map f ab ;

[0055] The multi-modal implicit mutual learning module MTL is used to process the feature map f ab , so as to extract features and obtain a feature map F ab ;

[0056] The deep feature extraction module utilizes the feature map F ab obtain a multi-modal fusion feature map;

[0057] The deep feature extraction module sequentially comprises a third stage processing unit STAGE3 and a fourth stage processing unit STAGE4; the output of the third stage processing unit STAGE3 is the input of the fourth stage processing unit STAGE4;

[0058] The third stage processing unit STAGE3 comprises a third down-sampling layer and a third ConvNeXt Block subunit;

[0059] The third ConvNeXt Block subunit is composed of 9 ConvNeXt Block blocks in series;

[0060] The fourth stage processing unit STAGE4 comprises a fourth down-sampling layer and a fourth ConvNeXt Block subunit; the fourth stage processing unit STAGE4 outputs a multi-modal fusion feature map;

[0061] The fourth ConvNeXt Block subunit is composed of 3 ConvNeXt Block blocks in series;

[0062] The feature vector output module utilizes the multi-modal fusion feature map to obtain a feature vector F of the modal A vehicle target image and a feature vector F' of the modal B vehicle target image;

[0063] The feature vector output module comprises a second average pooling layer, a batch normalization layer and a feature splitting layer;

[0064] The second average pooling layer is used for average pooling the output multi-modal fusion feature map of the fourth stage processing unit STAGE4;

[0065] The batch normalization layer is used for batch normalization of the output of the average pooling layer to obtain a fusion feature vector;

[0066] The feature splitting layer is used for splitting the fusion feature vector into two same feature vectors to obtain the feature vector F of the modal A vehicle target image and the feature vector F' of the modal B vehicle target image.

[0067] Further, the multi-modal implicit mutual learning module MTL comprises a feature splitting adjustment unit, a first MTT unit, a first full connection layer unit, a second MTT unit, a second full connection layer unit, a first feature fusion unit, a second feature fusion unit and a feature connection unit;

[0068] The feature split adjustment unit comprises a feature split layer, a first feature transformation layer and a second feature transformation layer.

[0069] The feature split layer is configured to split the feature map f ab into two same feature vectors, to obtain feature vectors f a and f b .

[0070] The first feature transformation layer is a convolution layer with a convolution kernel size of 1x1, configured to process the feature vectors f a to obtain a feature map f a '.

[0071] The second feature transformation layer is a convolution layer with a convolution kernel size of 1x1, configured to process the feature vectors f b to obtain a feature map f b '.

[0072] The first MTT unit comprises a first max-pooling layer, a first average-pooling layer, a seventh convolution layer and an eighth convolution layer, and an input of the first MTT unit is the feature map f a '.

[0073] The first max-pooling layer is configured to perform a max-pooling operation on the output f a ' of the first feature transformation layer.

[0074] The first average-pooling layer is configured to perform an average-pooling operation on the output f a ' of the first feature transformation layer.

[0075] The seventh convolution layer is a 28x28 convolution layer, and an input of the seventh convolution layer is the output f a ' of the first feature transformation layer.

[0076] The eighth convolution layer is a convolution layer with a convolution kernel size of 1x1, and an input of the eighth convolution layer is a sum of the output of the first max-pooling layer, the output of the first average-pooling layer and the output of the seventh convolution layer.

[0077] The second MTT unit has the same structure as the first MTT unit, and an input feature map of the second MTT unit is f b '.

[0078] An input of the first full connection layer unit is the output of the first MTT unit, to obtain a feature map f

[0079] An input of the second full connection layer unit is the output of the second MTT unit, to obtain a feature map f

[0080] The first feature fusion unit is configured to obtain fa the dot product result of

[0081] the second feature fusion unit is configured to obtain f b the dot product result of

[0082] the feature connection layer is configured to connect f and f , and perform residual connection on the connection result and f ab to obtain a feature map F ab .

[0083] Further, the training of the modal implicit mutual learning network CAIL in S203 using the training set adopts the following loss function:

[0084]

[0085] wherein, is a total loss function, is an MCD loss function, L triplet is a triplet loss function, L CE is a cross-entropy loss function, i is a sample class label, and N is the total number of sample classes, is the probability of the i-th class in the prediction result, y i is the true value of the current sample; y i is 0 or 1, and 1 indicates that the current sample is the i-th class, and 0 indicates that the current sample is not the i-th class.

[0086] Further, the triplet loss function L triplet is as follows:

[0087]

[0088] wherein, L triplet is a triplet loss function, is an anchor sample feature, is a positive sample feature of the same class as the anchor sample, and ‖·‖2 represents the Euclidean distance, is a negative sample feature of a different class from the anchor sample, and [z] + represents the positive part, z represents a variable, a is a distance threshold constant greater than 0, N' is the total number of triplets, i' is a triplet label, and the triplet includes an anchor sample, a positive sample and a negative sample; the positive sample is a sample of the same class as the anchor sample, and the negative sample is a sample of a different class from the anchor sample.

[0089] Further, the MCD loss function ​​Obtained by the following way:

[0090] First, in any training batch, P different vehicle categories are randomly selected, each vehicle category contains K samples to form a sample set, and the centroid features of the Ith vehicle category in the two modalities are obtained by using the sample set:

[0091]

[0092] Wherein, P is the total number of vehicle categories in the sample set, K is the total number of samples of a vehicle category in the sample set, j'∈[1,K], j' is the sample number of a vehicle category in the sample set, I is the vehicle category number in the sample set, Respectively represent the sample features of two modalities, M1 is modality A, and M2 is modality B, is the centroid feature of the Ith vehicle category in modality A, is the centroid feature of the Ith vehicle category in modality B;

[0093] Then, the main modality is obtained by using the centroid features of the Ith vehicle category in the two modalities:

[0094] The corresponding modality of M is the main modality M main , M is obtained by the following way:

[0095]

[0096] Wherein, is the centroid feature of modality A, is the centroid feature of modality B, M main is the main modality, D(·,·) is the calculation of Euclidean distance, is the internal distance of and , is the internal distance of and ;

[0097] Finally, assuming that the main modality M main is the corresponding modality of P C1 , the loss function

[0098]

[0099] Wherein, is the MCD loss, m is the marginal term, P i C1 is the centroid feature representing the i-th sample in the main modality. is the centroid feature of the j-th negative sample in the main modality, is the centroid feature of the jth negative sample in the non-main modal, and G is the set of all vehicle categories in the same modal.

[0100] The beneficial effects of the present application are:

[0101] The present application first trains an efficient modal implicit mutual learning network CAIL, which can extract deep features from synthetic aperture radar (SAR) images, visible light images and infrared images in remote sensing data, and generate embedding vectors in the Euclidean space respectively. Subsequently, by accurately calculating the Euclidean distance between these embedding vectors, effective association and recognition between the same type of targets across modalities are achieved, which makes up for the differences in imaging mechanisms of different modal images, thereby improving the cross-modal vehicle target image matching accuracy and vehicle target image matching efficiency. The present application also proposes a strategy of using multiple loss functions for joint optimization, which effectively alleviates the problem of model optimization difficulty caused by modal differences. By fine-tuning the weights and combination of different loss functions, the robustness and discriminability of feature representation are strengthened, thereby significantly improving the overall model effect and improving the accuracy and efficiency of vehicle target image matching. BRIEF DESCRIPTION OF DRAWINGS

[0102] Figure 1 is the flowchart of the present application;

[0103] Figure 2 is the flowchart of the CAIL network training process;

[0104] Figure 3 is the schematic diagram of the multi-modal implicit mutual learning module MIL and the multi-level channel fusion module MTT;

[0105] Figure 4 is the schematic diagram of the modal center alignment loss MCD. DETAILED DESCRIPTION

[0106] Embodiment one: the specific process of the SAR-visible light-infrared cross-modal image matching method of the present embodiment is as follows:

[0107] Step one, obtain the to-be-matched vehicle image and the candidate matching vehicle image set, and pre-process the to-be-matched vehicle image and the candidate matching vehicle image to obtain the pre-processed to-be-matched vehicle image and the pre-processed candidate matching vehicle image set, specifically:

[0108] Crop the vehicle target image from the to-be-matched vehicle image and the candidate matching vehicle image respectively, and unify the vehicle target image to the same size;

[0109] The to-be-matched vehicle image and the candidate matching vehicle image are both three-channel images;

[0110] The to-be-matched vehicle image and the candidate matching vehicle image are images of two different modalities;

[0111] Step 2: Input the preprocessed to-be-matched vehicle image and the preprocessed candidate matching vehicle image set into the image matching model to obtain the to-be-matched vehicle image feature vector and the candidate matching vehicle image feature vector respectively;

[0112] Step 3: Merge all candidate matching vehicle image feature vectors to obtain a candidate matching vehicle image feature vector group, obtain the Euclidean distance between each candidate matching vehicle image feature vector in the candidate matching vehicle image feature vector group and the vehicle image feature vector to be matched, and sort the candidate matching vehicle images corresponding to the candidate matching vehicle image feature vectors from small to large according to the Euclidean distance to obtain a vehicle image matching result;

[0113] Among them, the smaller the Euclidean distance value is, the more similar the vehicle image to be matched is to the current candidate matching vehicle image;

[0114] Step 4: Compare the Euclidean distance between the feature vector of each candidate matching vehicle image and the feature vector of the vehicle image to be matched with a preset similarity threshold. If the Euclidean distance is less than the preset similarity threshold, it means that the current candidate matching vehicle image and the vehicle image to be matched are the same target vehicle.

[0115] In order to quantify the similarity between images of different modalities, this step uses a dot product operation to calculate the Euclidean distance between two eigenvectors. By traversing each pair of eigenvectors and calculating the Euclidean distance between them, a distance matrix is ​​obtained, which reflects the feature differences between all image pairs. Finally, the calculated Euclidean distances are sorted. The image pairs with smaller distances indicate that their positions in the feature space are closer, that is, the similarity is higher. Based on this sorting result, it can be inferred which image pairs are likely to belong to targets of the same identity. In the actual matching process of the present invention, the vehicle image to be matched and the candidate matching vehicle image set are separately input into the image matching model. Therefore, during the processing process, only the modality A image processing branch or the modality B image processing branch is run, and the feature vector output at the end is not processed after passing through the feature splitting layer, and the feature vector of the current modality image can be directly output.

[0116] Specific implementation method 2: Figure 1 As shown, the image matching model is obtained in the following way:

[0117] S1. Preprocess the airborne remote sensing image data to obtain a multimodal vehicle target image set:

[0118] The airborne remote sensing image data is acquired, vehicle targets in the airborne remote sensing image data are detected and cropped to obtain vehicle target images, and then a vehicle class is taken as a label of the vehicle target images, and all the vehicle target images and the corresponding labels form a multi-modal vehicle target image set;

[0119] The airborne remote sensing image data includes visible light images, synthetic aperture radar (SAR) images and infrared images, so as to ensure the data quality in subsequent processing;

[0120] If the input is video data, according to the characteristics of dynamic changes of targets in the video, a suitable frame extraction interval is selected according to experience, the video is subjected to frame extraction sampling processing, so as to be converted into image data form. Then, the obtained image data set is preprocessed to obtain a multi-modal vehicle target image set, so as to improve the data quality and the efficiency of subsequent processing.

[0121] This step aims to effectively preprocess the obtained multi-modal remote sensing data (visible light images, infrared images and SAR images). Specifically, when the input data is in the form of an image, the vehicles in the image are first detected and cropped, so as to accurately extract vehicle targets, and corresponding labels are given for different types of vehicles, and the class of the label is the vehicle class. Subsequently, the overall data set is divided into a training data set and a test data set, so as to ensure the independence and effectiveness of model training and evaluation

[0122] S2, a training set and a test set are obtained by using the multi-modal vehicle target image set, and a modal implicit mutual learning network CAIL is trained and tested by using the training set and the test set, so as to obtain an image matching model, specifically:

[0123] S201, the multi-modal vehicle target image set is preprocessed to obtain a multi-modal aerial image data set:

[0124] Firstly, the SAR images in the multi-modal vehicle target image set are converted from tif format to jpg format which is more widely supported for subsequent processing;

[0125] Then, the SAR images, visible light images and infrared images after format conversion are uniformly cropped to 224x224 pixel size to ensure the consistency of input data;

[0126] Then, mainstream data enhancement techniques such as random horizontal flipping and random erasing are applied to all vehicle target images to increase the diversity of training samples and improve the generalization ability and recognition accuracy of the model;

[0127] Finally, all the images are converted into numpy format files for storage, so as to speed up the data loading process in the training stage, improve the training efficiency, and obtain a multi-modal aerial image data set;

[0128] S202, divide the multi-modal aerial image dataset into a training set and a test set;

[0129] S203, extract a training group from the training set, take one training group as one training batch, train the modal implicit mutual learning network CAIL using the training set, and obtain the trained modal implicit mutual learning network CAIL;

[0130] In order to enhance the data diversity, the training group is extracted from the training set, specifically: two different modalities A and B and four vehicle categories are randomly selected, and 12 vehicle target images of each vehicle category are extracted from the training set, of which 6 vehicle target images are modal A and 6 vehicle target images are modal B, thereby forming a training group; each training group has 4x6x2=48 vehicle target images; each training group is a training batch;

[0131] The modal A image and the modal B image are images of any two different modalities in visible light images, synthetic aperture radar (SAR) images and infrared images;

[0132] As shown in Figure 2 The modal implicit mutual learning network CAIL is an improved network based on the first four stages of the ConvNeXt network, which adopts a double-branch architecture and can perform parallel processing on the input multi-modal data, including: a modal A image processing branch, a modal B image processing branch, a feature merging module, a multi-modal implicit mutual learning module MTL, a deep feature extraction module, and a feature vector output module.

[0133] The modal A image processing branch includes a modal A first stage processing unit STAGE1 and a modal A second stage processing unit STAGE2; the input of the modal A image processing branch is any modal vehicle target image in the training group, and the output is a vector f a .

[0134] The modal A first stage processing unit STAGE1 includes a first convolutional layer and a first LN layer.

[0135] The first convolutional layer is a convolutional layer with a kernel size of 4x4 and a stride of 4, which is used to perform preliminary down-sampling on the input image to reduce the data dimension while retaining key information and obtain a vehicle target feature map;

[0136] The first LN (Layer Normalization) layer is used to normalize the vehicle target feature map to ensure the stability of the data distribution and generate a set of 56x56xC multi-modal feature maps, where C is the number of multi-modal feature map channels.

[0137] The modal A second stage processing unit STAGE2 includes a first ConvNeXt Block subunit, a first down-sampling layer, and a second ConvNeXt Block subunit.

[0138] The first ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series.

[0139] Each ConvNeXt Block block includes a second convolutional layer, a third convolutional layer, a first activation function layer, a fourth convolutional layer, a fifth convolutional layer, a Layer Scale operation layer, and a sixth convolutional layer.

[0140] The second convolutional layer is a convolutional layer with a kernel size of 7x7 (a stride of 1 and a padding of 3), and the second convolutional layer independently convolves each input channel.

[0141] The third convolutional layer is a convolutional layer with a kernel size of 1x1, which is used to mix channel information and obtain a feature map of 56x56x4C.

[0142] The first activation function is a GELU activation function, which is used to perform nonlinear transformation on the feature map.

[0143] The fourth convolutional layer is a standard convolutional layer with a kernel size of 1x1, which is used to reduce the number of feature map channels.

[0144] The fifth convolutional layer is a standard convolutional layer with a kernel size of 1x1, which is used to further reduce the number of feature map channels and output a feature map with a size of 56x56xC.

[0145] The Layer Scale operation layer is used to finely adjust the feature map output by the fifth convolutional layer, which finely scales the data of each channel by multiplying a trainable parameter, thereby improving the feature expression capability.

[0146] The sixth convolutional layer is a convolutional layer with a kernel size of 2x2 and a stride of 2, which is used to further down-sample the feature map output by the Layer Scale operation layer and obtain the ConvNeXt Block block output feature map.

[0147] The second ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series, and the structure of the second ConvNeXt Block subunit is the same as that of the first ConvNeXt Block subunit.

[0148] The modality B image processing branch includes: a modality B first stage processing unit STAGE1' and a modality B second stage processing unit STAGE2'; the input of the modality B image processing branch is a vehicle target image of another modality in the training group, and the output is a feature vector f b ;

[0149] The first stage processing unit STAGE1' of mode B has the same structure as the first stage processing unit STAGE1 of mode A;

[0150] The second stage processing unit STAGE2' of mode B has the same structure as the second stage processing unit STAGE2 of mode A;

[0151] The feature merging module is used to output the feature vector f of the modality A image processing branch a The output feature vector f of the image processing branch of modality B b Merge to obtain feature map f ab ;

[0152] The multimodal implicit mutual learning module MTL is used to process the feature map f ab Thus, features are extracted and feature map F is obtained. ab ;

[0153] The multimodal implicit mutual learning module MTL includes: a feature splitting and adjustment unit, a first MTT unit, a first fully connected layer unit, a second MTT unit, a second fully connected layer unit, a first feature fusion unit, a second feature fusion unit, and a feature connection unit;

[0154] The feature splitting and adjusting unit includes: a feature splitting layer, a first feature transformation layer, and a second feature transformation layer;

[0155] The feature splitting layer is used to transform the feature map f ab Split into two identical eigenvectors and obtain the eigenvector and

[0156] Where h is f a 、f b The height of f a 、f b The width of f a 、f b The number of channels;

[0157] The first feature transformation layer is a convolution layer ω with a convolution kernel size of 1×1. a , used to adjust Feature dimension or feature transformation, while promoting the interaction between different channels in the feature map, to obtain the feature map f a ';

[0158] The second feature transformation layer is a convolution layer ω with a convolution kernel size of 1×1. b , used to adjust Feature dimension or feature transformation to obtain feature map f b ';

[0159] The first MTT unit includes: a first maximum pooling layer, a first average pooling layer, a seventh convolutional layer, and an eighth convolutional layer; the input of the first MTT unit is the feature map f a ';

[0160] The first maximum pooling layer is used to capture the key features within the modality. The input of the first maximum pooling layer is the output f of the first feature transformation layer. a ';

[0161] The first average pooling layer is used to summarize the common features between channels. The input of the first average pooling layer is the output f of the first feature transformation layer. a ';

[0162] The seventh convolutional layer is a 28×28 convolutional layer; the input of the seventh convolutional layer is the output f of the first feature transformation layer. a ';

[0163] The eighth convolutional layer is a convolutional layer with a convolution kernel size of 1×1, which is used to deeply explore local details of the image; the input of the eighth convolutional layer is the sum of the output of the first maximum pooling layer, the output of the first average pooling layer, and the output of the seventh convolutional layer;

[0164] The second MTT unit has the same structure as the first MTT unit, and the input feature map is f b ';

[0165] The input of the first fully connected layer unit is the output of the first MTT unit, and the feature map is obtained.

[0166] The input of the second fully connected layer unit is the output of the second MTT unit, and the feature map is obtained.

[0167] The first feature fusion unit is used to obtain f a 'and The dot product result of

[0168] The second feature fusion unit is used to obtain f b 'and The dot product result of

[0169] The feature connection layer is used to and connect, the connection result is compared with f ab perform residual connection, obtain feature map F ab ;

[0170] The deep feature extraction module utilizes the feature map F ab obtain a multi-modal fusion feature map;

[0171] The deep feature extraction module sequentially includes a third stage processing unit STAGE3 and a fourth stage processing unit STAGE4; the output of the third stage processing unit STAGE3 is the input of the fourth stage processing unit STAGE4;

[0172] The third stage processing unit STAGE3 includes a third down-sampling layer and a third ConvNeXt Block subunit;

[0173] The third ConvNeXt Block subunit is composed of nine ConvNeXt Block blocks connected in series;

[0174] The fourth stage processing unit STAGE4 includes a fourth down-sampling layer and a fourth ConvNeXt Block subunit; the fourth stage processing unit outputs a group of feature maps with a size of 32x32x2048; the fourth stage processing unit outputs a group of multi-modal fusion feature maps; the multi-modal fusion feature map is a 32x32x2048 feature map;

[0175] The fourth ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series;

[0176] The feature vector output module utilizes the multi-modal fusion feature map to obtain a feature vector F of a modal A vehicle target image and a feature vector F' of a modal B vehicle target image

[0177] The feature vector output module includes a second average pooling layer, a batch normalization layer, and a feature splitting layer;

[0178] The second average pooling layer is configured to perform average pooling on the multi-modal fusion feature map output by the fourth stage processing unit;

[0179] The batch normalization layer is configured to perform batch normalization on the output of the average pooling layer to obtain a fusion feature vector; the fusion feature vector is a 1x1x2048 feature vector;

[0180] The feature splitting layer is used to split the fused feature vector into two 1x1x1024 feature vectors, and this splitting operation is based on the division of modalities, which ensures that each 1x1x1024 feature vector represents a set of features of an independent modality, and obtains the feature vector F of the modality A vehicle target image in the multi-modal aerial image data set and the feature vector F' of the modality B vehicle target image.

[0181] In S204, the trained modal implicit mutual learning network CAIL is tested by using the test set to obtain an image matching model.

[0182] In this embodiment, in the multi-modal implicit mutual learning network CAIL (Center-Aligned Implicit Learning Network) based on aerial images, the model aims to deeply mine and extract the essential features of images in a high-dimensional Euclidean space to form representative feature vectors. This multi-branch architecture not only efficiently fuses features of multiple modalities, but also greatly improves the adaptability of multi-modal encoding channels to dimensional changes, significantly enhancing the accuracy of feature recognition and analysis. In order to deal with the gradient vanishing problem that may be caused by deep networks, the CAIL network adopts a residual structure between multiple ConvNeXt Block feature extraction units to ensure effective information transmission in deep networks. The entire network is divided into four stages (STAGE1-STAGE4), each stage containing 0, 6, 9, and 3 encoder units respectively. This design not only guarantees the depth of the network, but also considers the computational efficiency and performance. During the forward propagation stage in the training process, the outputs of the modality A second stage processing unit STAGE2 and the modality B second stage processing unit STAGE2' are merged together, so that the shared network parameters involved can be trained during the training process. Then, f ab is directly split into two parts, which respectively represent the features of the two modalities, and the feature vectors and To further enhance the learning and fusion ability of features, an MTL module is inserted between STAGE2 and STAGE3. The MTL module can make full use of the complementary information in multi-modal data, and directly map cross-modal image data to a common representation space through a multi-modal implicit learning method, further improving the accuracy and robustness of cross-modal vehicle image matching. In order to more effectively learn the implicit representation of cross-modal, improve the representation ability and generalization performance of the model. The present application specially introduces MTT (Multi-level Channel Fusion) module. The MTT module, i.e. multi-level channel fusion module, focuses on deeply capturing and integrating statistical features at the modal channel level. The module is composed of three parallel branches, each branch is responsible for extracting different types of feature information. By applying different pooling layers and convolution kernels, these branches can obtain global statistics, maximum statistics and local details of the modal features. After extracting the features, the features from different branches are integrated through a fusion mechanism. Then, the cross-modal implicit information extracted by the MTT module is multiplied with the corresponding features as channel attention to obtain enhanced features and By integrating the MTT module into the MIL module, effective interaction between multi-modal features is realized. Finally, the processed multi-modal features are spliced, and a residual structure is used to improve the robustness of the entire module.

[0183] Specific implementation method three: the processing process of the first MTT unit, the first full connection layer unit, the second MTT unit and the second full connection layer unit can be summarized as follows:

[0184] A1, the specific processing process of the first MTT unit and the second MTT unit is:

[0185] f o =θ 1×1 (MaxPool(f' r )+AvgPool(f' r )+Conv2D(f' r ))

[0186] Where f' r takes f a ' or f b ', θ 1×1 is a weight matrix of a 1x1 convolution kernel, when f' r takes f a ', f o is the output of the first MTT unit; when f' r takes f b ', f o is the output of the second MTT unit;

[0187] A2, the first full connection layer unit, and the second full connection layer unit specifically process the following:

[0188]

[0189] Wherein, L is a full connection layer for integrating the fused information, and sigma is a Sigmoid activation function for introducing a nonlinear modeling capability, when f o is the output of the first MTT unit is a feature map When f o is the output of the second MTT unit is a feature map

[0190] In this embodiment, in order to further strengthen the model's ability in cross-modal implicit representation learning, the present application proposes an MTT module, which is optimized in depth at the level of modal channel, aiming to comprehensively capture the statistical characteristics of different modalities. Figure 3 The light red part in the formula is the MTT module, and the inputs of the two MTT modules are f a ' and f b ', each MTT module is composed of three parallel feature extraction branches with the same input, and each branch focuses on extracting specific types of feature information. Through the differential pooling strategy and the convolution kernel design, the module can efficiently obtain the global statistical characteristics, the maximum statistical characteristics and the local detail features of the modalities. After the completion of feature extraction, through the advanced fusion mechanism, these features from different branches are organically combined, which not only retains the unique information of each branch, but also realizes the complementation and enhancement between the features. Therefore, the MTT module can effectively represent the meaning of cross-modal implicit information in the original signal.

[0191] Specific implementation four: training the modal implicit mutual learning network CAIL using the training set adopts the following loss function:

[0192] B1, cross-entropy loss (Cross-Entropy Loss, CE Loss) is used as the basic supervision signal to strengthen the overall compactness of the feature embedding space and the separability between classes. CE Loss directly optimizes the error between the class prediction and the true label, effectively constraining the feature learning process, and its definition is as follows:

[0193]

[0194] Wherein, L CE is the cross-entropy loss function, i is the sample class label, N is the total number of sample classes, is the probability of the i-th class in the prediction result, y iis the true value of the current sample; y i is 1 or 0, 1 represents that the current sample is the i-th category, and 0 represents that the current sample is not the i-th category;

[0195] B2, in order to further improve the discrimination ability of the feature space, the application introduces a triplet loss function (Triplet Loss), which enhances the intra-class compactness and inter-class difference by optimizing the relative distance between the anchor sample, the positive sample (of the same category as the anchor) and the negative sample (of a different category from the anchor). Its definition is described in detail as follows:

[0196]

[0197] wherein, L triplet is a triplet loss function, is the feature of the anchor sample, is the feature of the positive sample of the same category as the anchor sample, and ‖·‖2 represents the Euclidean distance number, is the feature of the negative sample of a different category from the anchor sample, and [z] + represents the positive part, i.e. (max(z,0)), z represents a variable, and alpha is a distance threshold constant greater than 0, which is set to 0.3 here, N' is the total number of triplets, i' is the triplet label, and the triplet includes the anchor sample, the positive sample and the negative sample;

[0198] B3, as shown in Figure 4 , in view of the inherent differences between different modal data, the application proposes a modal center alignment (Modal Center Alignment, MCD) loss, which aims to dynamically adjust and optimize the adaptability of the neural network to the heterogeneity of different modalities, specifically:

[0199] In each training batch, P different vehicle categories are randomly selected, each vehicle category contains K samples to form a sample set, and the feature center of each modality under each identity is calculated. The feature center of the i-th vehicle category in the two modalities is obtained using the sample set:

[0200]

[0201] wherein, P is the total number of vehicle categories in the sample set, K is the total number of samples of a vehicle category in the sample set, j'∈[1,K], j' is the sample number of a vehicle category in the sample set, I is the vehicle category number in the sample set, respectively represent the sample features of the two modalities, M1 is modality A, and M2 is modality B, is the feature center of the i-th vehicle category in modality A, is the feature center of the i-th vehicle category in modality B.

[0202] Then, the centroid feature and the internal distance of each modality are calculated, and the modality with a larger sum of internal distances is called the main modality, because it may be more critical in describing the identity feature, specifically:

[0203]

[0204] The corresponding modality of M is the main modality M main .

[0205] wherein, is the centroid feature of modality A, is the centroid feature of modality B, M main is the main modality, D(·,·) is the Euclidean distance, is the internal distance between and , is the internal distance between and ;

[0206] Taking the centroid of the main modality as the anchor point (here it is assumed that P C1 corresponding modality is the main modality), a modality centroid alignment: MCD (Modality Centroid Dispersion) loss function is designed to reduce the Euclidean distance between the same class features, while increasing the Euclidean distance between the different class features. In order to balance this optimization process, a margin term m is introduced to ensure that the distance between different class features is large enough, while avoiding excessive aggregation between the same class features. In this way, the CAIL network can be more effectively optimized, and its performance in the multi-modal image recognition task can be improved. Finally, the definition of the MCD loss is as follows:

[0207]

[0208] wherein, is the MCD loss, m is the margin term, is the centroid feature of the i-th class sample in the main modality. is the centroid feature of the j-th negative sample in the main modality, is the centroid feature of the j-th negative sample in the non-main modality, and G is the set of all vehicle classes under the same modality.

[0209] B4, using the MCD loss triplet loss function L triplet and cross-entropy loss function L CE to obtain the overall loss:

[0210]

[0211] wherein, is the total loss function.

[0212] The embodiment combines CE Loss, Triplet Loss and MCD Loss to construct a multi-level loss function joint optimization system, aiming to comprehensively optimize the feature representation capability of the neural network. Especially in processing multi-modal data, the compactness, discriminability and cross-modal consistency of the feature space can be significantly improved. This multi-loss function joint optimization strategy can comprehensively consider the performance requirements of different aspects, further improve the recognition accuracy and generalization ability of the network.

[0213] Specific implementation five: the training set is used as a training batch to train the modal implicit mutual learning network CAIL, and the specific parameter settings are as follows:

[0214] The present application adopts the method of stage attenuation, and the initial learning rate is set to 1x10 -1 , then attenuated to 1x10 -2 , 1x10 -3 and 1x10 -4 at the 20th, 60th and 100th training stages respectively, until a total of 120 training cycles are completed. In addition, the SGD optimizer is selected for training, and the momentum parameter is set to 0.9 to accelerate the model convergence process.

[0215] The present application effectively realizes high-precision identification of vehicle target images in complex scenes, and the identification accuracy is stably maintained at more than 70%.

Claims

1. A SAR-visible-infrared cross-modal image matching method, characterized by: The specific process of the method is: Step 1: Obtain a to-be-matched vehicle image and a set of candidate matching vehicle images, and preprocess the to-be-matched vehicle image and the candidate matching vehicle images to obtain the preprocessed to-be-matched vehicle image and the preprocessed set of candidate matching vehicle images; The to-be-matched vehicle image and the candidate matching vehicle image are images of two different modalities; Image modalities include: visible light images, synthetic aperture radar images, and infrared images; Step 2: Input the preprocessed to-be-matched vehicle image and the preprocessed candidate matching vehicle image set into the image matching model to obtain the to-be-matched vehicle image feature vector and the candidate matching vehicle image feature vector respectively; The image matching model is obtained in the following way: S1. Preprocess the airborne remote sensing image data to obtain a multimodal vehicle target image set; S2. Using the multimodal vehicle target image set to obtain a training set and a test set, and using the training set and the test set to train and test a modality implicit inter-learning network (CAIL) to obtain an image matching model. The modality implicit mutual learning network CAIL includes: a modality A image processing branch, a modality B image processing branch, a feature merging module, a multimodal implicit mutual learning module MTL, a deep feature extraction module and a feature vector output module; The modality A image processing branch includes: a modality A first stage processing unit STAGE1 and a modality A second stage processing unit STAGE2; The input of the first stage processing unit STAGE1 of the modality A in the modality A image processing branch is the vehicle target image of any modality in the training group, and the output of the first stage processing unit STAGE1 of the modality A is input to the second stage processing unit STAGE2 of the modality A. The output of the second stage processing unit STAGE2 of the modality A is a vector ; The first stage processing unit STAGE1 of the modality A includes: a first convolutional layer and a first LN layer; The first convolutional layer is a convolutional layer with a convolution kernel size of 4×4 and a stride of 4; the first convolutional layer is used to obtain a vehicle target feature map; The first LN layer is used to normalize the vehicle target feature map to obtain a multimodal feature map; The second stage processing unit STAGE2 of the modality A includes: a first ConvNeXt Block subunit, a first downsampling layer and a second ConvNeXt Block subunit; The first ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series; Each ConvNeXt Block includes: the second convolution layer, the third convolution layer, the first activation function layer, the fourth convolution layer, the fifth convolution layer, the Layer Scale operation layer and the sixth convolution layer; The second convolutional layer is a convolutional layer with a convolution kernel size of 7×7; the output of the second convolutional layer is input to the third convolutional layer; The third convolutional layer is a convolutional layer with a convolution kernel size of 1×1; the output of the third convolutional layer is input to the first activation function layer; The first activation function layer is a GELU activation function; the output of the first activation function layer is the input of the fourth convolutional layer; The fourth convolution is a convolution layer with a convolution kernel size of 1×1; the output of the fourth convolution layer is the input of the fifth convolution layer; The fifth convolutional layer is a convolutional layer with a convolution kernel size of 1×1; the output of the fifth convolutional layer is the input of the Layer Scale operation layer; The output of the Layer Scale operation layer is the input of the sixth convolutional layer; The sixth convolutional layer is a convolutional layer with a convolution kernel size of 2×2 and a stride of 2, and the output feature map of the sixth convolutional layer is a ConvNeXt Block output feature map; The second ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series; the second ConvNeXtBlock subunit has the same structure as the first ConvNeXt Block subunit; the input of the second ConvNeXt Block subunit is the output of the first downsampling layer; The modality B image processing branch includes: a modality B first stage processing unit STAGE1 ′ and a modality B second stage processing unit STAGE2 ′; The input of the first stage processing unit STAGE1' of the modality B in the modality B image processing branch is the vehicle target image of another modality in the training group, the output of the first stage processing unit STAGE1' of the modality B is the input of the second stage processing unit STAGE2' of the modality B, and the second stage processing unit STAGE2' of the modality B outputs the feature vector ; The first stage processing unit STAGE1' of mode B has the same structure as the first stage processing unit STAGE1 of mode A; The second stage processing unit STAGE2' of mode B has the same structure as the second stage processing unit STAGE2 of mode A; The feature merging module is used to output the feature vector of the modality A image processing branch The output feature vector of the image processing branch of modality B is Merge to obtain feature map ; The multimodal implicit mutual learning module MTL is used to process the feature map Thus extracting features and obtaining feature maps ; The deep feature extraction module uses feature maps Obtain multimodal fusion feature map; The deep feature extraction module includes: a third stage processing unit STAGE3 and a fourth stage processing unit STAGE4; the output of the third stage processing unit STAGE3 is the input of the fourth stage processing unit STAGE4; The third stage processing unit STAGE3 includes: a third downsampling layer and a third ConvNeXt Block subunit; The third ConvNeXt Block subunit is composed of 9 ConvNeXt Block blocks connected in series; The fourth stage processing unit STAGE4 includes: a fourth downsampling layer and a fourth ConvNeXt Block subunit; the fourth stage processing unit STAGE4 outputs a multimodal fusion feature map; The fourth ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series; The feature vector output module uses the multimodal fusion feature map to obtain the feature vector of the vehicle target image in modality A. , the feature vector of the vehicle target image of modality B ; The feature vector output module includes: a second average pooling layer, a batch normalization layer and a feature splitting layer; The second average pooling layer is used to perform average pooling on the output multimodal fusion feature map of the fourth stage processing unit STAGE4; The batch normalization layer is used to perform batch normalization on the output of the average pooling layer to obtain a fused feature vector; The feature splitting layer is used to split the fused feature vector into two identical feature vectors to obtain the feature vector of the modal A vehicle target image. , the feature vector of the vehicle target image of modality B ; Step 3: Merge all candidate matching vehicle image feature vectors to obtain a candidate matching vehicle image feature vector group, obtain the Euclidean distance between each candidate matching vehicle image feature vector in the candidate matching vehicle image feature vector group and the vehicle image feature vector to be matched, and sort the candidate matching vehicle images corresponding to the candidate matching vehicle image feature vectors from small to large according to the Euclidean distance to obtain a vehicle image matching result; Among them, the smaller the Euclidean distance value is, the more similar the vehicle image to be matched is to the current candidate matching vehicle image; Step 4: Compare the Euclidean distance between the feature vector of each candidate matching vehicle image and the feature vector of the vehicle image to be matched with a preset similarity threshold. If the Euclidean distance is less than the preset similarity threshold, it means that the current candidate matching vehicle image and the vehicle image to be matched are the same target vehicle.

2. The SAR-visible-light-infrared cross-modal image matching method according to claim 1, characterized in that: The pre-processing of the to-be-matched vehicle image and the candidate matching vehicle image in step 1 is specifically as follows: The vehicle targets are respectively cropped out from the to-be-matched vehicle image and the candidate matching vehicle image, and the vehicle target images are unified into the same size; The to-be-matched vehicle image and the candidate matching vehicle image are both three-channel images.

3. The SAR-visible-light-infrared cross-modal image matching method according to claim 2, characterized in that: The image matching model is obtained in the following way: S1. Preprocess the airborne remote sensing image data to obtain a multimodal vehicle target image set, specifically: Acquire airborne remote sensing image data, detect and crop vehicle targets in the airborne remote sensing image data to obtain vehicle target images, and then use vehicle categories as labels for vehicle target images. All vehicle target images and corresponding labels constitute a multimodal vehicle target image set; The airborne remote sensing image data includes: visible light images, synthetic aperture radar images and infrared images; S2. Use the multimodal vehicle target image set to obtain a training set and a test set, and use the training set and the test set to train and test the modal implicit inter-learning network CAIL to obtain an image matching model.

4. The SAR-visible-light-infrared cross-modal image matching method according to claim 3, characterized in that: In S2, a training set and a test set are obtained by using a multimodal vehicle target image set, and a modality implicit inter-learning network CAIL is trained and tested by using the training set and the test set to obtain an image matching model, specifically: S201. Preprocess the multimodal vehicle target image set to obtain a multimodal aerial image dataset: First, the SAR images in the multimodal vehicle target image set are converted into jpg format; Then, the visible light image, infrared image and converted SAR image are cropped to a uniform preset size; Then, data augmentation is performed on all vehicle target images; The data enhancement includes: random horizontal flipping and random erasing; Finally, all vehicle target images are converted into numpy format files and stored; S202, dividing the multimodal aerial image dataset into a training set and a test set; S203, extracting a training group from the training set, taking one training group as a training batch, and using the training set to train a modality implicit mutual learning network CAIL to obtain a trained modality implicit mutual learning network CAIL; S204: Use the test set to test the trained modality implicit inter-learning network CAIL to obtain an image matching model.

5. The SAR-visible-infrared cross-modal image matching method according to claim 4, characterized in that: The training group is extracted from the training set in S203, and one training group is regarded as a training batch, which is obtained by the following method: Randomly select two different modes A and B, and choose vehicle categories and extract vehicle target images of each vehicle category in the training set Zhang, among them The vehicle target image is modality A, The vehicle target image is modality B, thus forming a training group, and a training group is regarded as a training batch; The modality A image and the modality B image are any two images of different modalities among visible light images, synthetic aperture radar images, and infrared images.

6. The SAR-visible-light-infrared cross-modal image matching method according to claim 5, characterized in that: The multimodal implicit mutual learning module MTL includes: a feature splitting and adjustment unit, a first MTT unit, a first fully connected layer unit, a second MTT unit, a second fully connected layer unit, a first feature fusion unit, a second feature fusion unit and a feature connection unit; The feature splitting and adjusting unit includes: a feature splitting layer, a first feature transformation layer, and a second feature transformation layer; The feature splitting layer is used to split the feature map Split into two identical eigenvectors and obtain the eigenvector and ; The first feature transformation layer is a convolution layer with a convolution kernel size of 1×1, which is used to process the feature vector Get feature map ; The second feature transformation layer is a convolution layer with a convolution kernel size of 1×1, which is used to process the feature vector Get feature map ; The first MTT unit includes: a first maximum pooling layer, a first average pooling layer, a seventh convolutional layer and an eighth convolutional layer; the input of the first MTT unit is a feature map ; The first maximum pooling layer is used to transform the output of the first feature transformation layer Perform maximum pooling operation; The first average pooling layer is used to average the output of the first feature transformation layer Perform average pooling operation; The seventh convolutional layer is a 28×28 convolutional layer; the input of the seventh convolutional layer is the output of the first feature transformation layer ; The eighth convolutional layer is a convolutional layer with a convolution kernel size of 1×1; the input of the eighth convolutional layer is the sum of the output of the first maximum pooling layer, the output of the first average pooling layer, and the output of the seventh convolutional layer; The second MTT unit has the same structure as the first MTT unit, and the input feature map of the second MTT unit is ; The input of the first fully connected layer unit is the output of the first MTT unit, and the feature map is obtained. ; The input of the second fully connected layer unit is the output of the second MTT unit, and the feature map is obtained. ; The first feature fusion unit is used to obtain and The dot product result of ; The second feature fusion unit is used to obtain and The dot product result of ; The feature connection unit is used to and Connect and connect the result with Perform residual connection to obtain feature map .

7. The SAR-visible-light-infrared cross-modal image matching method according to claim 6, characterized in that: The following loss function is used to train the modality implicit mutual learning network CAIL using the training set in S203: in, is the overall loss function, is the MCD loss function, is the triplet loss function, is the cross entropy loss function, is the sample category label, is the total number of sample categories, is the probability of the i-th category in the prediction result, is the true value of the current sample; It takes 0 or 1, 1 means the current sample belongs to the i-th category, and 0 means the current sample does not belong to the i-th category.

8. The SAR-visible-light-infrared cross-modal image matching method according to claim 7, characterized in that: The triplet loss function as follows: in, is the triplet loss function, is the anchor sample feature, is the positive sample feature of the same category as the anchor sample, represents the Euclidean distance number, is the negative sample feature of a different category from the anchor sample, Indicates the positive part, z indicates the variable, is a distance threshold constant greater than 0, is the total number of triplets, is a triplet label, which includes anchor samples, positive samples and negative samples; positive samples are samples of the same category as the anchor samples, and negative samples are samples of different categories from the anchor samples.

9. The SAR-visible-infrared cross-modal image matching method according to claim 8, characterized in that: The MCD loss function Obtained through: First, in any training batch, randomly select P different vehicle categories. Each vehicle category contains K samples to form a sample set. The sample set is used to obtain the centroid features of the I-th vehicle category in the two modes: Among them, P is the total number of vehicle categories in the sample set, K is the total number of samples of a certain vehicle category in the sample set, , is the sample number of a vehicle category in the sample set, is the vehicle category number in the sample set, 、 Represent the sample features of the two modalities respectively, is mode A, is mode B, is the centroid feature of the I-th vehicle category in mode A, is the centroid feature of the I-th vehicle category in mode B; Then, the main mode is obtained by using the centroid features of the I-th vehicle category in the two modes: The mode corresponding to M is the main mode , M is obtained as follows: in, is the centroid characteristic of mode A, is the centroid characteristic of mode B, is the main mode, is to calculate the Euclidean distance, yes and The internal distance of yes and Internal distance of Finally, assuming the main mode for Corresponding mode, obtain the loss function : in, is the MCD loss, is a marginal term, It indicates the first The centroid features of class samples, Is the first in the main mode The centroid features of negative samples, Is the first in the non-main mode The centroid features of negative samples, is the set of all vehicle categories under the same modality.

Images