SAR-visible light-infrared cross-modal image matching method

Abstract

The invention discloses an SAR-visible light-infrared cross-modal image matching method, and relates to the technical field of image matching. The invention aims to solve the problem that the existing cross-modal image matching method is low in matching precision and efficiency. The method comprises the following steps: obtaining a preprocessed to-be-matched vehicle image and a preprocessed candidate matched vehicle image set; inputting the preprocessed to-be-matched vehicle image and the preprocessed candidate matching vehicle image set into an image matching model to obtain a to-be-matched vehicle image feature vector and a candidate matching vehicle image feature vector; and obtaining the Euclidean distance between each candidate matching vehicle image feature vector and the to-be-matched vehicle image feature vector, sorting the candidate matching vehicle images corresponding to the candidate matching vehicle image feature vectors according to the descending order of the Euclidean distances, and obtaining a vehicle image matching result according to a preset similarity threshold. The method is used for cross-modal vehicle image matching.

Description

Technical Field

[0001] The present invention relates to the technical field of image matching, and in particular to a SAR-visible light-infrared cross-modal image matching method. Background Art

[0002] With the rapid development of remote sensing technology, the types and numbers of remote sensing equipment and sensors are increasing, and the acquired remote sensing image data is showing explosive growth. These image data include not only traditional visible light images, but also multiple modalities such as infrared images and synthetic aperture radar images. Multimodal remote sensing images provide multi-dimensional feature information for the same target or area, greatly enhancing human's ability to observe and analyze the earth. Vehicle image matching aims to confirm the identity of the same vehicle by comparing vehicle features in different images. This technology can efficiently utilize massive remote sensing image resources and has broad application prospects in intelligent transportation, urban management, public safety and other aspects.

[0003] Traditional vehicle image matching methods mainly rely on visible light images taken by fixed cameras. These methods perform well under good lighting conditions, but the effect is significantly reduced in complex real-world scenarios, especially in low light or bad weather conditions. Synthetic aperture radar (SAR), as an active remote sensing technology, has the advantage of not relying on external light sources and can work in all weather conditions and in various meteorological conditions. SAR images can penetrate clouds and vegetation to obtain detailed information about the surface, so they can complement traditional optical images well. However, there are significant differences in the imaging mechanism between SAR images and visible light images, resulting in low cross-modal image matching accuracy. In addition, the significant differences in the imaging mechanism between SAR images and visible light images will also affect the efficiency of information fusion, ultimately resulting in low image matching efficiency. Summary of the invention

[0004] The purpose of the present invention is to solve the problems of low matching accuracy and efficiency in existing cross-modal image matching methods, and propose a SAR-visible light-infrared cross-modal image matching method.

[0005] A SAR-visible light-infrared cross-modal image matching method, the specific process is as follows:

[0006] Step 1: obtaining a to-be-matched vehicle image and a set of candidate matching vehicle images, and preprocessing the to-be-matched vehicle image and the candidate matching vehicle image to obtain the preprocessed to-be-matched vehicle image and the preprocessed set of candidate matching vehicle images;

[0007] The to-be-matched vehicle image and the candidate matching vehicle image are images of two different modalities;

[0008] Image modalities include: visible light images, synthetic aperture radar images, and infrared images;

[0009] Step 2: input the preprocessed to-be-matched vehicle image and the preprocessed candidate matching vehicle image set into the image matching model respectively, and obtain the to-be-matched vehicle image feature vector and the candidate matching vehicle image feature vector respectively;

[0010] Step 3: merge all candidate matching vehicle image feature vectors to obtain a candidate matching vehicle image feature vector group, obtain the Euclidean distance between each candidate matching vehicle image feature vector in the candidate matching vehicle image feature vector group and the to-be-matched vehicle image feature vector, and sort the candidate matching vehicle images corresponding to the candidate matching vehicle image feature vectors from small to large according to the Euclidean distance to obtain a vehicle image matching result;

[0011] Among them, the smaller the Euclidean distance value is, the more similar the to-be-matched vehicle image is to the current candidate matching vehicle image.

[0012] Step 4: Compare the Euclidean distance between each candidate matching vehicle image feature vector and the feature vector of the vehicle image to be matched with a preset similarity threshold. If the Euclidean distance is less than the preset similarity threshold, it means that the current candidate matching vehicle image and the vehicle image to be matched are the same target vehicle.

[0013] Furthermore, the preprocessing of the to-be-matched vehicle image and the candidate matching vehicle image in step 1 is specifically as follows:

[0014] The vehicle targets are respectively cropped out from the to-be-matched vehicle image and the candidate matching vehicle image, and the vehicle target images are unified into the same size;

[0015] The to-be-matched vehicle image and the candidate matching vehicle image are both three-channel images.

[0016] Furthermore, the image matching model is obtained by:

[0017] S1. Preprocess the airborne remote sensing image data to obtain a multi-modal vehicle target image set, specifically:

[0018] Acquire airborne remote sensing image data, detect and crop vehicle targets in the airborne remote sensing image data to obtain vehicle target images, and then use the vehicle category as a label of the vehicle target image. All vehicle target images and corresponding labels constitute a multimodal vehicle target image set;

[0019] The airborne remote sensing image data includes: visible light images, synthetic aperture radar images and infrared images;

[0020] S2. Use the multimodal vehicle target image set to obtain a training set and a test set, and use the training set and the test set to train and test the modal implicit mutual learning network CAIL to obtain an image matching model.

[0021] Furthermore, the multimodal vehicle target image set is used to obtain a training set and a test set in S2, and the training set and the test set are used to train and test the modal implicit mutual learning network CAIL to obtain an image matching model, specifically:

[0022] S201, preprocessing the multimodal vehicle target image set to obtain a multimodal aerial image dataset:

[0023] First, the SAR images in the multimodal vehicle target image set are converted into jpg format;

[0024] Then, the visible light image, infrared image and converted SAR image are cropped to a uniform preset size;

[0025] Then, data enhancement is performed on all vehicle target images; the data enhancement includes: random horizontal flipping and random erasing;

[0026] Finally, all vehicle target images are converted into numpy format files and stored;

[0027] S202, dividing the multimodal aerial image dataset into a training set and a test set;

[0028] S203, extracting a training group from the training set, taking one training group as a training batch, and using the training set to train the modality implicit mutual learning network CAIL to obtain a trained modality implicit mutual learning network CAIL;

[0029] S204, using the test set to test the trained modality implicit mutual learning network CAIL to obtain an image matching model.

[0030] Furthermore, the step of extracting a training group from the training set in S203, taking a training group as a training batch, is obtained by the following method, specifically:

[0031] Randomly select two different modes A and B, select b vehicle categories, and extract a vehicle target image of each vehicle category in the training set, where The vehicle target image is modality A, The vehicle target images are modality B, thus forming a training group, and a training group is regarded as a training batch;

[0032] The modality A image and the modality B image are any two images of different modalities among visible light images, synthetic aperture radar images and infrared images.

[0033] Furthermore, the modality implicit mutual learning network CAIL in S203 includes: a modality A image processing branch, a modality B image processing branch, a feature merging module, a multimodal implicit mutual learning module MTL, a deep feature extraction module and a feature vector output module;

[0034] The modality A image processing branch includes: a modality A first stage processing unit STAGE1 and a modality A second stage processing unit STAGE2;

[0035] The input of the first stage processing unit STAGE1 of modality A in the modality A image processing branch is the vehicle target image of any modality in the training group, and the output of the first stage processing unit STAGE1 of modality A is input to the second stage processing unit STAGE2 of modality A, and the output of the second stage processing unit STAGE2 of modality A is the vector f a ;

[0036] The first stage processing unit STAGE1 of the modality A includes: a first convolutional layer and a first LN layer;

[0037] The first convolution layer is a convolution layer with a convolution kernel size of 4×4 and a step size of 4; the first convolution layer is used to obtain a vehicle target feature map;

[0038] The first LN layer is used to normalize the vehicle target feature map to obtain a multimodal feature map;

[0039] The second stage processing unit STAGE2 of the modality A includes: a first ConvNeXt Block subunit, a first downsampling layer, and a second ConvNeXt Block subunit;

[0040] The first ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series;

[0041] Each ConvNeXt Block includes: the second convolution layer, the third convolution layer, the first activation function layer, the fourth convolution layer, the fifth convolution layer, the Layer Scale operation layer and the sixth convolution layer;

[0042] The second convolution layer is a convolution layer with a convolution kernel size of 7×7; the output of the second convolution layer is input to the third convolution layer;

[0043] The third convolution layer is a convolution layer with a convolution kernel size of 1×1; the output of the third convolution layer is input to the first activation function layer;

[0044] The first activation function layer is a GELU activation function; the output of the first activation function layer is the input of the fourth convolutional layer;

[0045] The fourth convolution is a convolution layer with a convolution kernel size of 1×1; the output of the fourth convolution layer is the input of the fifth convolution layer;

[0046] The fifth convolution layer is a convolution layer with a convolution kernel size of 1×1; the output of the fifth convolution layer is the input of the LayerScale operation layer;

[0047] The output of the Layer Scale operation layer is the input of the sixth convolutional layer;

[0048] The sixth convolutional layer is a convolutional layer with a convolution kernel size of 2×2 and a stride of 2, and the output feature map of the sixth convolutional layer is a ConvNeXt Block output feature map;

[0049] The second ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series; the second ConvNeXt Block subunit has the same structure as the first ConvNeXt Block subunit; the input of the second ConvNeXt Block subunit is the output of the first downsampling layer;

[0050] The modality B image processing branch includes: a modality B first stage processing unit STAGE1' and a modality B second stage processing unit STAGE2';

[0051] The input of the first stage processing unit STAGE1' of the modality B in the modality B image processing branch is the vehicle target image of another modality in the training group, the output of the first stage processing unit STAGE1' of the modality B is the input of the second stage processing unit STAGE2' of the modality B, and the second stage processing unit STAGE2' of the modality B outputs the feature vector f b ;

[0052] The first stage processing unit STAGE1' of mode B has the same structure as the first stage processing unit STAGE1 of mode A;

[0053] The second stage processing unit STAGE2' of mode B has the same structure as the second stage processing unit STAGE2 of mode A;

[0054] The feature merging module is used to output the feature vector f of the modality A image processing branch a The image processing branch of modality B outputs the feature vector f b Merge to obtain feature map f ab ;

[0055] The multimodal implicit mutual learning module MTL is used to process the feature map f ab Thus, features are extracted and feature map F is obtained. ab ;

[0056] The deep feature extraction module uses the feature map F ab Obtain multimodal fusion feature map;

[0057] The deep feature extraction module includes: a third stage processing unit STAGE3 and a fourth stage processing unit STAGE4 in sequence; the output of the third stage processing unit STAGE3 is the input of the fourth stage processing unit STAGE4;

[0058] The third stage processing unit STAGE3 includes: a third downsampling layer and a third ConvNeXt Block subunit;

[0059] The third ConvNeXt Block subunit is composed of 9 ConvNeXt Block blocks connected in series;

[0060] The fourth stage processing unit STAGE4 includes: a fourth downsampling layer and a fourth ConvNeXt Block subunit; the fourth stage processing unit STAGE4 outputs a multimodal fusion feature map;

[0061] The fourth ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series;

[0062] The feature vector output module uses the multimodal fusion feature map to obtain the feature vector F of the vehicle target image in modality A and the feature vector F' of the vehicle target image in modality B;

[0063] The feature vector output module includes: a second average pooling layer, a batch normalization layer and a feature splitting layer;

[0064] The second average pooling layer is used to average pool the output multimodal fusion feature map of the fourth stage processing unit STAGE4;

[0065] The batch normalization layer is used to perform batch normalization on the output of the average pooling layer to obtain a fused feature vector;

[0066] The feature splitting layer is used to split the fused feature vector into two identical feature vectors to obtain the feature vector F of the modality A vehicle target image and the feature vector F' of the modality B vehicle target image.

[0067] Further, the multimodal implicit mutual learning module MTL includes: a feature splitting adjustment unit, a first MTT unit, a first fully connected layer unit, a second MTT unit, a second fully connected layer unit, a first feature fusion unit, a second feature fusion unit and a feature connection unit;

[0068] The feature splitting and adjusting unit comprises: a feature splitting layer, a first feature transformation layer, and a second feature transformation layer;

[0069] The feature splitting layer is used to transform the feature map f ab Split into two identical eigenvectors and obtain the eigenvector f a and f b ;

[0070] The first feature transformation layer is a convolution layer with a convolution kernel size of 1×1, which is used to process the feature vector f a Get the feature map f a ′;

[0071] The second feature transformation layer is a convolution layer with a convolution kernel size of 1×1, which is used to process the feature vector f b Get the feature map f b ′;

[0072] The first MTT unit includes: a first maximum pooling layer, a first average pooling layer, a seventh convolutional layer and an eighth convolutional layer; the input of the first MTT unit is the feature map f a ′;

[0073] The first maximum pooling layer is used to transform the output f of the first feature transformation layer. a 'Perform maximum pooling operation;

[0074] The first average pooling layer is used to average the output f of the first feature transformation layer. a 'Perform average pooling operation;

[0075] The seventh convolutional layer is a 28×28 convolutional layer; the input of the seventh convolutional layer is the output f of the first feature transformation layer. a ′;

[0076] The eighth convolutional layer is a convolutional layer with a convolution kernel size of 1×1; the input of the eighth convolutional layer is the sum of the output of the first maximum pooling layer, the output of the first average pooling layer, and the output of the seventh convolutional layer;

[0077] The second MTT unit has the same structure as the first MTT unit, and the input feature map of the second MTT unit is f b ′;

[0078] The input of the first fully connected layer unit is the output of the first MTT unit, and the feature map is obtained.

[0079] The input of the second fully connected layer unit is the output of the second MTT unit, and the feature map is obtained.

[0080] The first feature fusion unit is used to obtain fa 'and The dot product result of

[0081] The second feature fusion unit is used to obtain f b 'and The dot product result of

[0082] The feature connection layer is used to and Connect and connect the result with f ab Perform residual connection to obtain feature map F ab .

[0083] Furthermore, the following loss function is used in the step S203 to train the modality implicit mutual learning network CAIL using the training set:

[0084]

[0085] in, is the overall loss function, is the MCD loss function, L triplet is the triplet loss function, L CE is the cross entropy loss function, i is the sample category label, N is the total number of sample categories, is the probability of the i-th category in the prediction result, y i is the true value of the current sample; y i It takes 0 or 1, 1 means the current sample belongs to the i-th category, and 0 means the current sample does not belong to the i-th category.

[0086] Furthermore, the triple loss function L triplet as follows:

[0087]

[0088] Among them, L triplet is the triplet loss function, is the anchor sample feature, is the positive sample feature of the same category as the anchor sample, ‖·‖2 represents the Euclidean distance number, is the negative sample feature of a different category from the anchor sample, [z] + Indicates taking the positive part, z represents a variable, α is a distance threshold constant greater than 0, N' is the total number of triplets, i' is the triplet label, and the triplet includes anchor samples, positive samples, and negative samples; positive samples are samples of the same category as the anchor samples, and negative samples are samples of different categories from the anchor samples.

[0089] Furthermore, the MCD loss function Obtained through:

[0090] First, in any training batch, P different vehicle categories are randomly selected, each of which contains K samples to form a sample set. The sample set is used to obtain the centroid features of the I-th vehicle category in the two modes:

[0091]

[0092] Where P is the total number of vehicle categories in the sample set, K is the total number of samples of a certain vehicle category in the sample set, j'∈[1,K], j' is the sample number of a certain vehicle category in the sample set, I is the vehicle category number in the sample set, Represent the sample features of the two modes respectively, M1 is mode A, M2 is mode B, is the centroid feature of the I-th vehicle category in mode A, is the centroid feature of the I-th vehicle category in mode B;

[0093] Then, the main mode is obtained by using the centroid features of the I-th vehicle category in the two modes:

[0094] The mode corresponding to M is the main mode M main , M is obtained as follows:

[0095]

[0096] in, is the centroid characteristic of mode A, is the centroid characteristic of mode B, M main is the main mode, D(·,·) is the Euclidean distance, yes and The internal distance of yes and The internal distance of

[0097] Finally, assuming that the main mode M main P C1 Corresponding mode, get the loss function

[0098]

[0099] in, is the MCD loss, m is the marginal term, P i C1 It represents the centroid feature of the i-th sample in the main mode. is the centroid feature of the j-th negative sample in the main modality, is the centroid feature of the j-th negative sample in the non-main mode, and G is the set of all vehicle categories in the same mode.

[0100] The beneficial effects of the present invention are:

[0101] The present invention first trains an efficient modal implicit mutual learning network CAIL, which can perform deep feature extraction on synthetic aperture radar (SAR) images, visible light images and infrared images in remote sensing data, and generate their respective embedding vectors in Euclidean space. Subsequently, by accurately calculating the Euclidean distance between these embedded vectors, effective association and recognition between similar targets across modalities are achieved, which makes up for the problem of differences in imaging mechanisms of different modal images, thereby improving the accuracy of cross-modal vehicle target image matching and the efficiency of vehicle target image matching. The present invention also proposes a strategy for joint optimization using multiple loss functions, which effectively alleviates the problem of model optimization difficulties caused by modal differences. The present invention strengthens the robustness and discriminability of feature representation by finely adjusting the weights and combinations of different loss functions, thereby significantly improving the effect of the overall model and improving the accuracy and efficiency of vehicle target image matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 It is a flow chart of the present invention;

[0103] Figure 2 This is a flowchart of the CAIL network training process;

[0104] Figure 3 Schematic diagram of the multimodal implicit mutual learning module MIL and the multi-level channel fusion module MTT;

[0105] Figure 4 Schematic diagram of the modal center alignment loss MCD. DETAILED DESCRIPTION

[0106] Specific implementation method 1: The specific process of a SAR-visible light-infrared cross-modal image matching method in this implementation method is as follows:

[0107] Step 1: Obtain a to-be-matched vehicle image and a set of candidate matching vehicle images, and preprocess the to-be-matched vehicle image and the candidate matching vehicle image to obtain the preprocessed to-be-matched vehicle image and the preprocessed set of candidate matching vehicle images, specifically:

[0108] The vehicle target images are respectively cropped out from the to-be-matched vehicle image and the candidate matching vehicle image, and the vehicle target images are unified into the same size;

[0109] The to-be-matched vehicle image and the candidate matching vehicle image are both three-channel images;

[0110] The to-be-matched vehicle image and the candidate matching vehicle image are images of two different modalities;

[0111] Step 2: input the preprocessed to-be-matched vehicle image and the preprocessed candidate matching vehicle image set into the image matching model respectively, and obtain the to-be-matched vehicle image feature vector and the candidate matching vehicle image feature vector respectively;

[0112] Step 3: merge all candidate matching vehicle image feature vectors to obtain a candidate matching vehicle image feature vector group, obtain the Euclidean distance between each candidate matching vehicle image feature vector in the candidate matching vehicle image feature vector group and the to-be-matched vehicle image feature vector, and sort the candidate matching vehicle images corresponding to the candidate matching vehicle image feature vectors from small to large according to the Euclidean distance to obtain a vehicle image matching result;

[0113] Among them, the smaller the Euclidean distance value is, the more similar the image of the vehicle to be matched is to the current candidate matching vehicle image;

[0114] Step 4: Compare the Euclidean distance between each candidate matching vehicle image feature vector and the feature vector of the vehicle image to be matched with a preset similarity threshold. If the Euclidean distance is less than the preset similarity threshold, it means that the current candidate matching vehicle image and the vehicle image to be matched are the same target vehicle.

[0115] In order to quantify the similarity between images of different modalities, this step uses a dot multiplication operation to calculate the Euclidean distance between two feature vectors. By traversing each pair of feature vectors and calculating the Euclidean distance between them, a distance matrix is ​​obtained, which reflects the feature differences between all image pairs. Finally, the calculated Euclidean distances are sorted. The smaller the distance, the closer their positions in the feature space are, that is, the higher the similarity. Based on this sorting result, it can be inferred which image pairs are likely to belong to the same identity target. In the actual matching process of the present invention, the vehicle image to be matched and the candidate matching vehicle image set are separately input into the image matching model. Therefore, during the processing, only the modality A image processing branch or the modality B image processing branch is run, and the feature vector output at the end is not processed after the feature splitting layer, and the feature vector of the current modality image can be directly output.

[0116] Specific implementation method 2: Figure 1 As shown, the image matching model is obtained in the following way:

[0117] S1. Preprocess the airborne remote sensing image data to obtain a multi-modal vehicle target image set:

[0118] Acquire airborne remote sensing image data, detect and crop vehicle targets in the airborne remote sensing image data to obtain vehicle target images, and then use the vehicle category as a label of the vehicle target image. All vehicle target images and corresponding labels constitute a multimodal vehicle target image set;

[0119] The airborne remote sensing image data includes: visible light images, synthetic aperture radar (SAR) images and infrared images, ensuring the data quality for subsequent processing;

[0120] If the input is video data, according to the characteristics of the dynamic changes of the target in the video, the appropriate frame sampling interval is selected according to experience, and the video is frame sampled and processed to convert it into image data. Then, the obtained image data set is preprocessed to obtain a multimodal vehicle target image set to improve data quality and the efficiency of subsequent processing.

[0121] This step aims to effectively preprocess the acquired multimodal remote sensing data (visible light images, infrared images, SAR images). Specifically, when the input data is in the form of an image, the vehicle in the image is first detected and cropped to accurately extract the vehicle target, and corresponding labels are given for different types of vehicles. The category of the label is the vehicle category. Subsequently, the entire data set is divided into a training data set and a test data set to ensure the independence and effectiveness of model training and evaluation.

[0122] S2. Using the multimodal vehicle target image set to obtain a training set and a test set, and using the training set and the test set to train and test the modal implicit mutual learning network CAIL to obtain an image matching model, specifically:

[0123] S201, preprocessing the multimodal vehicle target image set to obtain a multimodal aerial image dataset:

[0124] Firstly, the SAR images in the multimodal vehicle target image set are converted from tif format to the more widely supported jpg format for subsequent processing;

[0125] Then, the converted SAR images, visible light images, and infrared images were uniformly cropped to 224 × 224 pixel size to ensure the consistency of the input data;

[0126] Then, mainstream data enhancement techniques such as random horizontal flipping and random erasing are applied to all vehicle target images to increase the diversity of training samples and improve the generalization ability and recognition accuracy of the model;

[0127] Finally, all images are converted to numpy format files for storage, which speeds up the data loading process in the training phase, improves training efficiency, and obtains a multimodal aerial image dataset.

[0128] S202, dividing the multimodal aerial image dataset into a training set and a test set;

[0129] S203, extracting a training group from the training set, taking one training group as a training batch, and using the training set to train the modality implicit mutual learning network CAIL to obtain a trained modality implicit mutual learning network CAIL;

[0130] In order to enhance data diversity, the training group is extracted from the training set as follows: two different modalities A and B and four vehicle categories are randomly selected, and 12 vehicle target images of each vehicle category are extracted from the training set, of which 6 vehicle target images are modality A and 6 vehicle target images are modality B, thus forming a training group; each training group has 4×6×2=48 vehicle target images; each training group is a training batch;

[0131] The modality A image and the modality B image are any two images of different modalities among visible light images, synthetic aperture radar (SAR) images, and infrared images;

[0132] like Figure 2 As shown, the modality implicit mutual learning network CAIL is an improved network based on the first four stages of the ConvNeXt network. The network adopts a dual-division architecture and can process the input multimodal data in parallel, including: modality A image processing branch, modality B image processing branch, feature merging module, multimodal implicit mutual learning module MTL, deep feature extraction module, and feature vector output module;

[0133] The modality A image processing branch includes: a modality A first stage processing unit STAGE1 and a modality A second stage processing unit STAGE2; the input of the modality A image processing branch is a vehicle target image of any modality in the training group, and the output is a vector f a ;

[0134] The first stage processing unit STAGE1 of the modality A includes: a first convolutional layer and a first LN layer;

[0135] The first convolutional layer is a convolutional layer with a convolution kernel size of 4×4 and a step size of 4, which is used to perform preliminary downsampling of the input image to reduce the data dimension while retaining key information to obtain a vehicle target feature map;

[0136] The first LN (Layer Normalization) layer is used to normalize the vehicle target feature map to ensure the stability of data distribution and generate a set of 56×56×C multimodal feature maps, where C is the number of multimodal feature map channels;

[0137] The second stage processing unit STAGE2 of the modality A includes: a first ConvNeXt Block subunit, a first downsampling layer, and a second ConvNeXt Block subunit;

[0138] The first ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series;

[0139] Each ConvNeXt Block includes: the second convolution layer, the third convolution layer, the first activation function layer, the fourth convolution layer, the fifth convolution layer, the Layer Scale operation layer, and the sixth convolution layer;

[0140] The second convolutional layer is a convolutional layer with a convolution kernel size of 7×7 (a step size of 1 and a padding of 3), and the second convolutional layer performs convolution on each input channel independently;

[0141] The third convolutional layer is a convolutional layer with a convolution kernel size of 1×1, which is used to mix channel information to obtain a 56×56×4C feature map;

[0142] The first activation function is a GELU activation function, which is used to perform nonlinear transformation on the feature map;

[0143] The fourth convolution is a standard convolution layer with a convolution kernel size of 1×1, which is used to reduce the number of feature map channels;

[0144] The fifth convolutional layer is a standard convolutional layer with a convolution kernel size of 1×1, which is used to further reduce the number of channels of the feature map, and the output feature map size is 56×56×C;

[0145] The Layer Scale operation layer is used to fine-tune the feature map output by the fifth convolutional layer, and fine-scales the data of each channel by multiplying a trainable parameter to improve the feature expression capability;

[0146] The sixth convolution layer is a convolution layer with a convolution kernel size of 2×2 and a step size of 2, and is used to further downsample the output feature map of the Layer Scale operation layer to obtain the ConvNeXt Block output feature map;

[0147] The second ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series; the second ConvNeXt Block subunit has the same structure as the first ConvNeXt Block subunit.

[0148] The modality B image processing branch includes: a modality B first stage processing unit STAGE1' and a modality B second stage processing unit STAGE2'; the input of the modality B image processing branch is a vehicle target image of another modality in the training group, and the output is a feature vector f b ;

[0149] The first stage processing unit STAGE1' of mode B has the same structure as the first stage processing unit STAGE1 of mode A;

[0150] The second stage processing unit STAGE2' of mode B has the same structure as the second stage processing unit STAGE2 of mode A;

[0151] The feature merging module is used to output the feature vector f of the modality A image processing branch a The image processing branch of modality B outputs the feature vector f b Merge to obtain feature map f ab ;

[0152] The multimodal implicit mutual learning module MTL is used to process the feature map f ab Thus, features are extracted and feature map F is obtained. ab ;

[0153] The multimodal implicit mutual learning module MTL includes: a feature splitting adjustment unit, a first MTT unit, a first fully connected layer unit, a second MTT unit, a second fully connected layer unit, a first feature fusion unit, a second feature fusion unit, and a feature connection unit;

[0154] The feature splitting and adjusting unit comprises: a feature splitting layer, a first feature transformation layer, and a second feature transformation layer;

[0155] The feature splitting layer is used to transform the feature map f ab Split into two identical eigenvectors and obtain the eigenvector and

[0156] Where h is f a 、f b The height of f a 、f b The width of f a 、f b The number of channels;

[0157] The first feature transformation layer is a convolution layer ω with a convolution kernel size of 1×1. a , used to adjust The feature dimension or feature transformation is performed, and the interaction between different channels in the feature map is promoted to obtain the feature map f a ′;

[0158] The second feature transformation layer is a convolution layer ω with a convolution kernel size of 1×1 b , used to adjust Feature dimension or feature transformation to obtain feature map f b ′;

[0159] The first MTT unit includes: a first maximum pooling layer, a first average pooling layer, a seventh convolutional layer, and an eighth convolutional layer; the input of the first MTT unit is a feature map f a ′;

[0160] The first maximum pooling layer is used to capture the key features within the modality. The input of the first maximum pooling layer is the output f of the first feature transformation layer. a ′;

[0161] The first average pooling layer is used to summarize the common features between channels. The input of the first average pooling layer is the output f of the first feature transformation layer. a ′;

[0162] The seventh convolutional layer is a 28×28 convolutional layer; the input of the seventh convolutional layer is the output f of the first feature transformation layer. a ′;

[0163] The eighth convolutional layer is a convolutional layer with a convolution kernel size of 1×1, which is used to deeply explore local details of the image; the input of the eighth convolutional layer is the sum of the output of the first maximum pooling layer, the output of the first average pooling layer, and the output of the seventh convolutional layer;

[0164] The second MTT unit has the same structure as the first MTT unit, and the input feature map is f b ′;

[0165] The input of the first fully connected layer unit is the output of the first MTT unit, and the feature map is obtained.

[0166] The input of the second fully connected layer unit is the output of the second MTT unit, and the feature map is obtained.

[0167] The first feature fusion unit is used to obtain f a 'and The dot product result of

[0168] The second feature fusion unit is used to obtain f b 'and The dot product result of

[0169] The feature connection layer is used to and Connect and connect the result with f ab Perform residual connection to obtain feature map F ab ;

[0170] The deep feature extraction module uses the feature map F ab Obtain multimodal fusion feature map;

[0171] The deep feature extraction module includes: a third stage processing unit STAGE3 and a fourth stage processing unit STAGE4 in sequence; the output of the third stage processing unit STAGE3 is the input of the fourth stage processing unit STAGE4;

[0172] The third stage processing unit STAGE3 includes: a third downsampling layer, a third ConvNeXt Block subunit;

[0173] The third ConvNeXt Block subunit is composed of 9 ConvNeXt Block blocks connected in series;

[0174] The fourth stage processing unit STAGE4 includes: a fourth downsampling layer, a fourth ConvNeXt Block subunit; the fourth stage processing unit outputs a set of feature maps with a size of 32×32×2048; the fourth stage processing unit outputs a set of multimodal fusion feature maps; the multimodal fusion feature map is a 32×32×2048 feature map;

[0175] The fourth ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series;

[0176] The feature vector output module uses the multimodal fusion feature map to obtain the feature vector F of the vehicle target image in modality A and the feature vector F' of the vehicle target image in modality B.

[0177] The feature vector output module includes: a second average pooling layer, a batch normalization layer, and a feature splitting layer;

[0178] The second average pooling layer is used to average pool the output multimodal fusion feature map of the fourth stage processing unit;

[0179] The batch normalization layer is used to perform batch normalization on the output of the average pooling layer to obtain a fused feature vector; the fused feature vector is a feature vector of 1×1×2048;

[0180] The feature splitting layer is used to split the fused feature vector into two 1×1×1024 feature vectors. This splitting operation is based on the division of modalities, ensuring that each 1×1×1024 feature vector represents a feature set of an independent modality, and obtaining the feature vector F of the vehicle target image of modality A and the feature vector F' of the vehicle target image of modality B in the multimodal aerial image dataset.

[0181] S204, using the test set to test the trained modality implicit mutual learning network CAIL to obtain an image matching model.

[0182] In this embodiment, in the multimodal implicit mutual learning network CAIL (Center-Aligned Implicit Learning Network) based on aerial images, the model aims to deeply explore and extract the essential features of the image in the high-dimensional Euclidean space to form a representative feature vector. This multi-branch architecture not only efficiently integrates the features of multiple modes, but also greatly improves the adaptability of the multimodal coding channel to dimensional changes, and significantly enhances the accuracy of feature recognition and analysis. In order to deal with the gradient vanishing problem that may be caused by deep networks, the CAIL network adopts a residual structure between multiple ConvNeXt Block feature extraction units to ensure the effective transmission of information in the deep network. The entire network is divided into four stages (STAGE1-STAGE4), each stage contains 0, 6, 9, and 3 encoder units respectively. This design not only ensures the depth of the network, but also takes into account the computational efficiency and performance. In the forward propagation stage of the training process, the outputs of the second stage processing unit STAGE2 of modality A and the second stage processing unit STAGE2' of modality B are merged together, so that the parameters of the network involved can be shared during the training process. Then, it is necessary to convert f ab Directly split into two parts, these two parts represent the characteristics of the two modes respectively, and obtain the feature vector and In order to further enhance the learning and fusion capabilities of features, the MTL module is inserted between STAGE2 and STAGE3. The MTL module can make full use of the complementary information in multimodal data, and directly map the cross-modal image data to a common representation space through a multimodal implicit learning method, thereby further improving the accuracy and robustness of cross-modal vehicle image matching. In order to more effectively learn the implicit representation of cross-modality and improve the representation ability and generalization performance of the model. The present invention particularly introduces the MTT (Multi-level Channel Fusion) module. The MTT module, i.e., the multi-level channel fusion module, focuses on deeply capturing and integrating statistical features at the modal channel level. The module consists of three parallel branches, each of which is responsible for extracting different types of feature information. By applying different pooling layers and convolution kernels, these branches can obtain features such as global statistics, maximum statistics, and local details of the modality. After extracting the features, these features from different branches are integrated through a fusion mechanism. Afterwards, the cross-modal implicit information extracted by the MTT module is used as channel attention and multiplied with the corresponding features to obtain enhanced features. as well as By integrating the MTT module into the MIL module, effective interaction between multimodal features is achieved. Finally, the processed multimodal features are concatenated and the input features are transformed into residual structures to improve the robustness of the entire module.

[0183] Specific implementation method three: The processing process of the first MTT unit, the first fully connected layer unit, the second MTT unit, and the second fully connected layer unit can be summarized as the following formula:

[0184] A1, the first MTT unit, and the second MTT unit have the following specific processing steps:

[0185] f o =θ 1×1 (MaxPool(f' r )+AvgPool(f' r )+Conv2D(f' r ))

[0186] Among them, f' r Take f a ′ or f b ′,θ 1×1 is the weight matrix of the 1×1 convolution kernel, when f' r Take f a ′, f o is the output of the first MTT unit; when f' r Take f b ′, f o is the output of the second MTT unit;

[0187] The specific processing process of A2, the first fully connected layer unit, and the second fully connected layer unit is as follows:

[0188]

[0189] Among them, L is the fully connected layer used to integrate the fused information, σ is the Sigmoid activation function used to introduce nonlinear modeling capabilities, when f o is the output of the first MTT unit It is a feature map When f o is the output of the second MTT unit It is a feature map

[0190] In this embodiment, in order to further enhance the model's ability in cross-modal implicit representation learning, the present invention proposes an MTT module, which is deeply optimized at the modal channel level to comprehensively capture the statistical characteristics of different modalities. Figure 3 The light red part in the figure is the MTT module. The inputs of the two MTT modules are f a ′ and f b ′, each MTT module consists of three parallel feature extraction branches with exactly the same input, and each branch focuses on extracting a specific type of feature information. Through differentiated pooling strategies and convolution kernel designs, the module can efficiently obtain the global statistical characteristics, maximum statistical characteristics, and local detail characteristics of the modality. After the feature extraction is completed, these features from different branches are organically combined through advanced fusion mechanisms, which not only retains the unique information of each branch, but also achieves complementarity and enhancement between features. Therefore, the MTT module can effectively represent the meaning of cross-modal implicit information in the original signal.

[0191] Specific implementation method 4: Using the training set to train the modality implicit mutual learning network CAIL adopts the following loss function:

[0192] B1. Cross-Entropy Loss (CE Loss) is used as the basic supervisory signal to enhance the overall compactness of the feature embedding space and the separability between categories. CE Loss effectively constrains the feature learning process by directly optimizing the error between the category prediction and the true label. It is defined as follows:

[0193]

[0194] Among them, L CE is the cross entropy loss function, i is the sample category label, N is the total number of sample categories, is the probability of the i-th category in the prediction result, y iis the true value of the current sample; y i It takes 0 or 1, 1 means the current sample belongs to the i-th category, and 0 means the current sample does not belong to the i-th category;

[0195] B2. In order to further improve the discriminative ability of feature space, the present invention introduces the triplet loss function, which enhances the intra-class compactness and inter-class difference by optimizing the relative distance between anchor samples, positive samples (same category as anchors) and negative samples (different categories from anchors). Its definition is as follows:

[0196]

[0197] Among them, L triplet is the triplet loss function, is the anchor sample feature, is the positive sample feature of the same category as the anchor sample, ‖·‖2 represents the Euclidean distance number, is the negative sample feature of a different category from the anchor sample, [z] + Indicates taking the positive part, that is, (max(z,0)), z represents a variable, α is a distance threshold constant greater than 0, which is set to 0.3 here, N' is the total number of triplets, i' is the triplet number, and the triplets include anchor samples, positive samples, and negative samples;

[0198] B3, such as Figure 4 As shown, in view of the inherent differences between different modal data, the present invention proposes a modal center alignment (MCD) loss, which aims to dynamically adjust and optimize the adaptability of the neural network to different modal heterogeneities, specifically:

[0199] In each training batch, P different vehicle categories are randomly selected, each vehicle category contains K samples to form a sample set, and the feature centroid of each mode under each identity is calculated. The sample set is used to obtain the centroid features of the I-th vehicle category in the two modes:

[0200]

[0201] Where P is the total number of vehicle categories in the sample set, K is the total number of samples of a certain vehicle category in the sample set, j'∈[1,K], j' is the sample number of a certain vehicle category in the sample set, I is the vehicle category number in the sample set, Represent the sample features of the two modes respectively, M1 is mode A, M2 is mode B, is the centroid feature of the I-th vehicle category in mode A, is the centroid feature of the I-th vehicle category in mode B;

[0202] Subsequently, the centroid features and internal distances of each mode are calculated, and the mode with a larger sum of internal distances is called the main mode because it may be more critical in describing identity features, specifically:

[0203]

[0204] The mode corresponding to M is the main mode M main .

[0205] in, is the centroid characteristic of mode A, is the centroid characteristic of mode B, M main is the main mode, D(·,·) is the Euclidean distance, yes and The internal distance of yes and The internal distance of

[0206] The centroid of the main mode is used as the anchor point (here we assume that P C1 The corresponding modality is the main modality), and the modality center alignment: MCD (Modality Centroid Dispersion) loss function is designed to reduce the Euclidean distance between similar features and increase the Euclidean distance between heterogeneous features. In order to balance this optimization process, a marginal term m is introduced to ensure that the distance between heterogeneous features is large enough while avoiding excessive aggregation of similar features. In this way, the CAIL network can be optimized more effectively and its performance in multimodal image recognition tasks can be improved. Finally, the definition of MCD loss is as follows:

[0207]

[0208] in, is the MCD loss, m is the marginal term, It represents the centroid feature of the i-th sample in the main mode. is the centroid feature of the j-th negative sample in the main modality, is the centroid feature of the j-th negative sample in the non-main mode, and G is the set of all vehicle categories in the same mode.

[0209] B4. Using MCD loss Triplet loss function L triplet And the cross entropy loss function L CE Get the overall loss:

[0210]

[0211] in, is the overall loss function.

[0212] This implementation combines CE Loss, Triplet Loss and MCD Loss to construct a multi-level loss function joint optimization system, aiming to comprehensively optimize the feature representation ability of the neural network. Especially when processing multimodal data, it can significantly improve the compactness, discriminability and cross-modal consistency of the feature space. This multi-loss function joint optimization strategy can comprehensively consider the performance requirements of different aspects and further improve the recognition accuracy and generalization ability of the network.

[0213] Specific implementation method 5: The method uses a training group as a training batch, and uses the training set to train the modality implicit mutual learning network CAIL. The specific parameters are set as follows:

[0214] The present invention adopts a phased decay method, and the initial learning rate is set to 1×10 -1 , and then decayed to 1×10 at the 20th, 60th, and 100th training stages, respectively. -2 , 1×10 -3 and 1×10 -4 , until a total of 120 training cycles are completed. In addition, the SGD optimizer is selected for training, and the momentum parameter is set to 0.9 to accelerate the model convergence process.

[0215] The present invention effectively realizes high-precision recognition of vehicle target images in complex scenes, and its recognition accuracy rate is stably maintained at above 70%.

Claims

1. A SAR-visible light-infrared cross-modal image matching method, characterized by: The specific process of the method is: Step 1: obtaining a to-be-matched vehicle image and a set of candidate matching vehicle images, and preprocessing the to-be-matched vehicle image and the candidate matching vehicle image to obtain the preprocessed to-be-matched vehicle image and the preprocessed set of candidate matching vehicle images; The to-be-matched vehicle image and the candidate matching vehicle image are images of two different modalities; Image modalities include: visible light images, synthetic aperture radar images, and infrared images; Step 2: input the preprocessed to-be-matched vehicle image and the preprocessed candidate matching vehicle image set into the image matching model respectively, and obtain the to-be-matched vehicle image feature vector and the candidate matching vehicle image feature vector respectively; Step 3: merge all candidate matching vehicle image feature vectors to obtain a candidate matching vehicle image feature vector group, obtain the Euclidean distance between each candidate matching vehicle image feature vector in the candidate matching vehicle image feature vector group and the to-be-matched vehicle image feature vector, and sort the candidate matching vehicle images corresponding to the candidate matching vehicle image feature vectors from small to large according to the Euclidean distance to obtain a vehicle image matching result; Among them, the smaller the Euclidean distance value is, the more similar the image of the vehicle to be matched is to the current candidate matching vehicle image; Step 4: Compare the Euclidean distance between each candidate matching vehicle image feature vector and the feature vector of the vehicle image to be matched with a preset similarity threshold. If the Euclidean distance is less than the preset similarity threshold, it means that the current candidate matching vehicle image and the vehicle image to be matched are the same target vehicle.

2. The SAR-visible light-infrared cross-modal image matching method according to claim 1, characterized in that: The preprocessing of the to-be-matched vehicle image and the candidate matching vehicle image in step 1 is specifically as follows: The vehicle targets are respectively cropped out from the to-be-matched vehicle image and the candidate matching vehicle image, and the vehicle target images are unified into the same size; The to-be-matched vehicle image and the candidate matching vehicle image are both three-channel images.

3. The SAR-visible light-infrared cross-modal image matching method according to claim 2, characterized in that: The image matching model is obtained in the following way: S1. Preprocess the airborne remote sensing image data to obtain a multi-modal vehicle target image set, specifically: Acquire airborne remote sensing image data, detect and crop vehicle targets in the airborne remote sensing image data to obtain vehicle target images, and then use the vehicle category as a label of the vehicle target image. All vehicle target images and corresponding labels constitute a multimodal vehicle target image set; The airborne remote sensing image data includes: visible light images, synthetic aperture radar images and infrared images; S2. Use the multimodal vehicle target image set to obtain a training set and a test set, and use the training set and the test set to train and test the modal implicit mutual learning network CAIL to obtain an image matching model.

4. The SAR-visible light-infrared cross-modal image matching method according to claim 3, characterized in that: The step S2 uses a multimodal vehicle target image set to obtain a training set and a test set, and uses the training set and the test set to train and test a modality implicit mutual learning network CAIL to obtain an image matching model, specifically: S201, preprocessing the multimodal vehicle target image set to obtain a multimodal aerial image dataset: First, the SAR images in the multimodal vehicle target image set are converted into jpg format; Then, the visible light image, infrared image and converted SAR image are cropped to a uniform preset size; Then, data augmentation is performed on all vehicle target images; The data enhancement includes: random horizontal flipping and random erasing; Finally, all vehicle target images are converted into numpy format files and stored; S202, dividing the multimodal aerial image dataset into a training set and a test set; S203, extracting a training group from the training set, taking one training group as a training batch, and using the training set to train the modality implicit mutual learning network CAIL to obtain a trained modality implicit mutual learning network CAIL; S204, using the test set to test the trained modality implicit mutual learning network CAIL to obtain an image matching model.

5. The SAR-visible light-infrared cross-modal image matching method according to claim 4, characterized in that: The training group is extracted from the training set in S203, and one training group is used as a training batch, which is obtained by: Randomly select two different modes A and B, select b vehicle categories, and extract a vehicle target image of each vehicle category in the training set, where The vehicle target image is modality A, The vehicle target images are modality B, thus forming a training group, and a training group is regarded as a training batch; The modality A image and the modality B image are any two images of different modalities among visible light images, synthetic aperture radar images and infrared images.

6. The SAR-visible light-infrared cross-modal image matching method according to claim 5, characterized in that: The modality implicit mutual learning network CAIL in S203 includes: a modality A image processing branch, a modality B image processing branch, a feature merging module, a multi-modality implicit mutual learning module MTL, a deep feature extraction module and a feature vector output module; The modality A image processing branch includes: a modality A first stage processing unit STAGE1 and a modality A second stage processing unit STAGE2; The input of the first stage processing unit STAGE1 of modality A in the modality A image processing branch is the vehicle target image of any modality in the training group, and the output of the first stage processing unit STAGE1 of modality A is input to the second stage processing unit STAGE2 of modality A, and the output of the second stage processing unit STAGE2 of modality A is the vector f a ; The first stage processing unit STAGE1 of the modality A includes: a first convolutional layer and a first LN layer; The first convolution layer is a convolution layer with a convolution kernel size of 4×4 and a step size of 4; the first convolution layer is used to obtain a vehicle target feature map; The first LN layer is used to normalize the vehicle target feature map to obtain a multimodal feature map; The second stage processing unit STAGE2 of the modality A includes: a first ConvNeXt Block subunit, a first downsampling layer, and a second ConvNeXt Block subunit; The first ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series; Each ConvNeXt Block includes: the second convolution layer, the third convolution layer, the first activation function layer, the fourth convolution layer, the fifth convolution layer, the Layer Scale operation layer and the sixth convolution layer; The second convolution layer is a convolution layer with a convolution kernel size of 7×7; the output of the second convolution layer is input to the third convolution layer; The third convolution layer is a convolution layer with a convolution kernel size of 1×1; the output of the third convolution layer is input to the first activation function layer; The first activation function layer is a GELU activation function; the output of the first activation function layer is the input of the fourth convolutional layer; The fourth convolution is a convolution layer with a convolution kernel size of 1×1; the output of the fourth convolution layer is the input of the fifth convolution layer; The fifth convolution layer is a convolution layer with a convolution kernel size of 1×1; the output of the fifth convolution layer is the input of the Layer Scale operation layer; The output of the Layer Scale operation layer is the input of the sixth convolutional layer; The sixth convolutional layer is a convolutional layer with a convolution kernel size of 2×2 and a stride of 2, and the output feature map of the sixth convolutional layer is a ConvNeXt Block output feature map; The second ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series; the second ConvNeXtBlock subunit has the same structure as the first ConvNeXt Block subunit; the input of the second ConvNeXt Block subunit is the output of the first downsampling layer; The modality B image processing branch includes: a modality B first stage processing unit STAGE1' and a modality B second stage processing unit STAGE2'; The input of the first stage processing unit STAGE1' of the modality B in the modality B image processing branch is the vehicle target image of another modality in the training group, the output of the first stage processing unit STAGE1' of the modality B is the input of the second stage processing unit STAGE2' of the modality B, and the second stage processing unit STAGE2' of the modality B outputs the feature vector f b ; The first stage processing unit STAGE1' of mode B has the same structure as the first stage processing unit STAGE1 of mode A; The second stage processing unit STAGE2' of mode B has the same structure as the second stage processing unit STAGE2 of mode A; The feature merging module is used to output the feature vector f of the modality A image processing branch a The image processing branch of modality B outputs the feature vector f b Merge to obtain feature map f ab ; The multimodal implicit mutual learning module MTL is used to process the feature map f ab Thus, features are extracted and feature map F is obtained. ab ; The deep feature extraction module uses the feature map F ab Obtain multimodal fusion feature map; The deep feature extraction module includes: a third stage processing unit STAGE3 and a fourth stage processing unit STAGE4 in sequence; the output of the third stage processing unit STAGE3 is the input of the fourth stage processing unit STAGE4; The third stage processing unit STAGE3 includes: a third downsampling layer and a third ConvNeXt Block subunit; The third ConvNeXt Block subunit is composed of 9 ConvNeXt Block blocks connected in series; The fourth stage processing unit STAGE4 includes: a fourth downsampling layer and a fourth ConvNeXt Block subunit; the fourth stage processing unit STAGE4 outputs a multimodal fusion feature map; The fourth ConvNeXt Block subunit is composed of three ConvNeXt Block blocks connected in series; The feature vector output module uses the multimodal fusion feature map to obtain the feature vector F of the vehicle target image in modality A and the feature vector F' of the vehicle target image in modality B; The feature vector output module includes: a second average pooling layer, a batch normalization layer and a feature splitting layer; The second average pooling layer is used to average pool the output multimodal fusion feature map of the fourth stage processing unit STAGE4; The batch normalization layer is used to perform batch normalization on the output of the average pooling layer to obtain a fused feature vector; The feature splitting layer is used to split the fused feature vector into two identical feature vectors to obtain the feature vector F of the modality A vehicle target image and the feature vector F' of the modality B vehicle target image.

7. The SAR-visible light-infrared cross-modal image matching method according to claim 6, characterized in that: The multimodal implicit mutual learning module MTL includes: a feature splitting adjustment unit, a first MTT unit, a first fully connected layer unit, a second MTT unit, a second fully connected layer unit, a first feature fusion unit, a second feature fusion unit and a feature connection unit; The feature splitting and adjusting unit comprises: a feature splitting layer, a first feature transformation layer, and a second feature transformation layer; The feature splitting layer is used to transform the feature map f ab Split into two identical eigenvectors and obtain the eigenvector f a and f b ; The first feature transformation layer is a convolution layer with a convolution kernel size of 1×1, which is used to process the feature vector f a Get the feature map f a ′; The second feature transformation layer is a convolution layer with a convolution kernel size of 1×1, which is used to process the feature vector f b Get the feature map f b ′; The first MTT unit includes: a first maximum pooling layer, a first average pooling layer, a seventh convolutional layer and an eighth convolutional layer; the input of the first MTT unit is the feature map f a ′; The first maximum pooling layer is used to transform the output f of the first feature transformation layer. a 'Perform maximum pooling operation; The first average pooling layer is used to average the output f of the first feature transformation layer. a 'Perform average pooling operation; The seventh convolutional layer is a 28×28 convolutional layer; the input of the seventh convolutional layer is the output f of the first feature transformation layer. a ′; The eighth convolutional layer is a convolutional layer with a convolution kernel size of 1×1; the input of the eighth convolutional layer is the sum of the output of the first maximum pooling layer, the output of the first average pooling layer, and the output of the seventh convolutional layer; The second MTT unit has the same structure as the first MTT unit, and the input feature map of the second MTT unit is f b ′; The input of the first fully connected layer unit is the output of the first MTT unit, and the feature map is obtained. The input of the second fully connected layer unit is the output of the second MTT unit, and the feature map is obtained. The first feature fusion unit is used to obtain f a 'and The dot product result of The second feature fusion unit is used to obtain f b 'and The dot product result of The feature connection layer is used to and Connect and connect the result with f ab Perform residual connection to obtain feature map F ab .

8. The SAR-visible light-infrared cross-modal image matching method according to claim 7, characterized in that: The following loss function is used for training the modality implicit mutual learning network CAIL using the training set in S203: in, is the overall loss function, is the MCD loss function, L triplet is the triplet loss function, L CE is the cross entropy loss function, i is the sample category label, N is the total number of sample categories, is the probability of the i-th category in the prediction result, y i is the true value of the current sample; y i It takes 0 or 1, 1 means the current sample belongs to the i-th category, and 0 means the current sample does not belong to the i-th category.

9. The SAR-visible light-infrared cross-modal image matching method according to claim 8, characterized in that: The triple loss function L triplet as follows: Among them, L triplet is the triplet loss function, is the anchor sample feature, is the positive sample feature of the same category as the anchor sample, ‖·‖2 represents the Euclidean distance number, is the negative sample feature of a different category from the anchor sample, [z] + Indicates taking the positive part, z represents a variable, α is a distance threshold constant greater than 0, N' is the total number of triplets, i' is the triplet label, and the triplet includes anchor samples, positive samples, and negative samples; positive samples are samples of the same category as the anchor samples, and negative samples are samples of different categories from the anchor samples.

10. The SAR-visible light-infrared cross-modal image matching method according to claim 9, characterized in that: The MCD loss function Obtained through: First, in any training batch, P different vehicle categories are randomly selected, each of which contains K samples to form a sample set. The sample set is used to obtain the centroid features of the I-th vehicle category in the two modes: M 1: M2: Where P is the total number of vehicle categories in the sample set, K is the total number of samples of a certain vehicle category in the sample set, j'∈[1,K], j' is the sample number of a certain vehicle category in the sample set, I is the vehicle category number in the sample set, Represent the sample features of the two modes respectively, M1 is mode A, M2 is mode B, is the centroid feature of the I-th vehicle category in mode A, is the centroid feature of the I-th vehicle category in mode B; Then, the main mode is obtained by using the centroid features of the I-th vehicle category in the two modes: The mode corresponding to M is the main mode M main , M is obtained as follows: in, is the centroid characteristic of mode A, is the centroid characteristic of mode B, M main is the main mode, D(·,·) is the Euclidean distance, yes and The internal distance of yes and The internal distance of Finally, assuming that the main mode M main P C1 Corresponding mode, get the loss function in, is the MCD loss, m is the marginal term, P i C1 It represents the centroid feature of the i-th sample in the main mode. is the centroid feature of the j-th negative sample in the main modality, is the centroid feature of the j-th negative sample in the non-main mode, and G is the set of all vehicle categories in the same mode.

Images