A multi-scale reinforcement learning cross-modal image matching method

Abstract

The application discloses a kind of multi-scale reinforcement learning cross-modal image matching methods, comprising the following steps: S1.in neural network model input benchmark and real-time graph;S2.based on ResNet network, the features of the benchmark and real-time graph are extracted, and the relevant area of the real-time graph feature in the benchmark feature is obtained by the area recommendation module in the 2nd, 3rd and 4th layers of ResNet network;S3.the features extracted by ResNet network in step S2 are fused by the transformation feature fusion module;S4.the fusion features in step S3 are input into the feature decoding module, and the position of real-time graph in benchmark is obtained;S5.the position matching result obtained in step S4 is output.The benchmark and real-time graph of different modalities input by the application can obtain the accurate position of real-time graph in benchmark, whether SAR image, visible light image or far infrared image can obtain better matching result, compared with traditional matching method, the method has great potential, strong robustness and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and machine learning, and specifically relates to a cross-modal image matching method based on multi-scale reinforcement learning. Background Art

[0002] Currently proposed algorithms for cross-modal imaging primarily focus on the modality invariance of feature similarity measures. Currently, the main methods proposed for measuring cross-modal image similarity include template matching, grayscale correlation, and fast Fourier transform (FFT). These methods leverage the statistical information of pixel grayscale values ​​to measure image similarity and employ specific search methods to maximize the similarity measure, thereby achieving cross-modal image matching. Similarity algorithms for cross-modal images generally employ mutual information, gradient, and self-similarity as similarity measures, offering advantages of simplicity and ease of implementation. However, these methods suffer from a large number of parameters and computational complexity, which aligns with the trend toward lightweight algorithms in the field of artificial intelligence. Furthermore, these methods are sensitive to grayscale distortion and object deformation, which are common in multimodal images. If the distortion rate and deformation detected in two images are high, the feature reproducibility rate will be very low, compromising the reliability of feature matching. Therefore, studying cross-modal image matching is more complicated than traditional visible light image matching, because the imaging principles of various sensors are different, the differences between multimodal images are much greater than those of single visible light images, and multimodal images are affected by many factors such as different shooting perspectives, time, and deformation.

[0003] As one of the most important foundational technologies in artificial intelligence, deep learning has gradually extended its reach into the field of imaging in recent years, with its application in scenarios such as image classification, object recognition, and detection becoming increasingly widespread. Convolutional neural networks, as a component of deep learning models, can be trained to learn features such as texture, edges, shape, and image topology, providing a more comprehensive representation of features. Furthermore, feature extraction algorithms based on convolutional neural networks can leverage neural networks to extract high-level image features. These features have also been shown to share similarities across heterogeneous images, demonstrating greater potential than traditional algorithms and avoiding the inherent robustness and inefficiency of traditional matching algorithms. Summary of the Invention

[0004] In order to solve the problems in the prior art, the present invention provides a cross-modal image matching method based on multi-scale reinforcement learning, which includes the following steps:

[0005] S1. Input the baseline image and the real-time image into the neural network model;

[0006] S2. Feature extraction is performed on the reference image and the real-time image based on the ResNet network, and the 2nd, 3rd, and 4th layer features in the ResNet network are passed through the region recommendation module to obtain the relevant areas of the real-time image features at three scales of the 2nd, 3rd, and 4th layer features in the reference image features;

[0007] S3. Fusion of features extracted by the ResNet network in step S2 through the Transformer feature fusion module;

[0008] S4. The fusion feature in step S3 is input into the feature decoding module to obtain the position of the real-time image in the reference image;

[0009] S5. Output the position matching result obtained in step S4.

[0010] Furthermore, in step S2, a real-time image feature extraction network and a reference image feature extraction network are obtained based on the ResNet network, and residual block features in the real-time image feature extraction network and the reference image feature extraction network are extracted, and a cross-correlation operation is performed using the region recommendation module to obtain the relevant area of ​​the real-time image in the reference image.

[0011] Furthermore, the specific calculation process of the cross-correlation operation is as follows:

[0012] The real-time graph feature is represented as: The feature representation of the i-th real-time image residual block is:

[0013] The benchmark image feature is expressed as: The feature representation of the residual block of the i-th reference image is:

[0014] Among them, B is the batch size of the training data, c is the feature dimension, h i 、w i and H i 、W i is the feature size, and h i <H i , w i <W i , H is the height of the input reference image, W is the width of the input reference image, h is the height of the input real-time image, w is the width of the input real-time image,

[0015] At this time, the correlation characteristics It can be expressed as:

[0016]

[0017] Among them, i represents the i-th residual block; Represents convolution-related operations, including convolution, Batch Normalization, and ReLU; the symbol * represents cross-correlation operation.

[0018] Furthermore, the real-time image feature is used as a convolution kernel, a convolution operation with a step size of 1 is performed on the reference image feature, and a correlation calculation is performed correspondingly in each feature channel to obtain a correlation feature.

[0019]

[0020] Furthermore, the real-time graph feature First, upsample to the same size as the baseline image feature by linear interpolation Post- and base map features The input feature f of the Transformer feature fusion module is obtained by concatenating and transposing in the feature dimension in ,

[0021] in, D = 2 × C,

[0022] Where N is the number of features after spatial flattening, H is the height of the input reference image, W is the width of the input reference image, h is the height of the input real-time image, and c is the dimension of the feature.

[0023] Furthermore, the input feature f in is segmented into a sequence of fixed-size 2D patches, and then a learnable positional encoding is added To add position information, we get f, which is the feature obtained by adding a position code to each input feature.

[0024] f=[f1;f : ;f3;…;f N ]+E pos .

[0025] Furthermore, the backbone network of the transformer in the Transformer feature fusion module consists of L blocks, each of which consists of a multi-head self-attention layer (MSA) and a feed-forward network (FFN), where the single-head attention is calculated as follows:

[0026]

[0027] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, d k is the scale factor,

[0028] In order to obtain more effective attention results for different feature subspaces, multi-head self-attention concatenates the outputs of multiple single-head self-attentions and projects them onto another parameter matrix:

[0029]

[0030]

[0031] in, is the parameter matrix in the i-th attention head of the l-th transformer block, f l The output of MSA is sent to the feedforward neural network composed of MLP to generate the output f of the transformer block. l+1 ,

[0032] The calculation process is as follows:

[0033] f l ′=MSA(f l )+f l ,f l+1 =FFN(f l ′)+f l '

[0034] The final feature is f out After the transposition operation, the features input into the feature decoding module are obtained

[0035] Furthermore, the feature decoding module includes five decoding layers, namely the first decoding layer, the second decoding layer, the third decoding layer, the fourth decoding layer and the fifth decoding layer. Each decoding layer must undergo convolution, batch normalization, ReLU activation and upsampling operation steps to obtain the output features of each decoding layer.

[0036] Furthermore, the specific steps of decoding features by using the feature decoding module are as follows:

[0037] 1) The fused features The features are obtained through the first decoding layer Then the correlation feature obtained from the fourth residual block Up-sample to get The same size of h and w, so we get After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1, and then The output feature of the first decoding layer is obtained by performing residual connection with the activated value:

[0038] 2) Input the output features of the first decoding layer into the second decoding layer to obtain features Then the correlation features obtained by the third residual block Up-sample to get The same size of h and w, so we get After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1. Perform residual connection with the activated value to obtain the output feature of the second decoding layer

[0039] 3) Input the output features of the second decoding layer into the third decoding layer to obtain features Similarly, the correlation features obtained by the second residual block Up-sample and get The same size of h and w, so we get After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1. Perform residual connection with the activated value to obtain the output features of the third decoding layer

[0040] 4) Input the output features of the third decoding layer into the fourth decoding layer to obtain features Then the features Input to the fifth decoding layer to obtain features

[0041] 5) Features Perform convolution operation to reduce the dimension of the feature to 1, and then perform Sigmoid activation operation on the feature to make the feature value distributed between 0 and 1 to obtain the output feature

[0042] The output feature f out The size of has the same two-dimensional matrix as the output feature size of the input reference image, and the peak position in the matrix represents the position of the real-time image in the reference image.

[0043] Furthermore, the convolution kernel size of the convolution operation is 3, the step size is 1, the padding is 1, and the output dimension is 1 / 2 of the input dimension; the magnification scale of the upsampling operation is 2.

[0044] The present invention provides a cross-modal image matching method with multi-scale reinforcement learning. Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) The present invention uses the ResNet18 network to realize feature extraction, and innovatively uses the region recommendation module to perform cross-correlation operations to obtain the relevant areas of the real-time image in the reference image.

[0046] (2) The present invention adopts a Transformer feature fusion module to calculate the correlation between the same parts of two features through the self-attention mechanism, and obtains the fused features through the MLP network and residual connection.

[0047] (3) The present invention can achieve cross-modal matching, and the model can adapt to inputs of different modalities. Whether it is SAR images, visible light images or far-infrared images, good matching results can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the cross-modal image matching method with multi-scale reinforcement learning of the present invention;

[0049] Figure 2 This is the overall flow chart of the ResNet18 network operation during the implementation of the present invention;

[0050] Figure 3 is a diagram of the calculation process of the region recommendation module during the implementation of the present invention;

[0051] Figure 4 This is a diagram of the calculation process of the Transformer feature fusion module in the implementation process of the present invention;

[0052] Figure 5 This is a diagram of a feature decoding module guided by similarity in the implementation process of the present invention;

[0053] Figure 6 It is a matching result graph of the center of the real-time graph and its accurate position in the reference graph in the present invention and the position of the real-time graph in the reference graph predicted by the model. DETAILED DESCRIPTION

[0054] The present invention is further explained below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments in this application without making creative efforts are within the scope of protection of this application.

[0055] The present invention provides a cross-modal image matching method based on multi-scale reinforcement learning, which mainly includes a region recommendation module, a feature fusion module, and a feature decoding module. By inputting reference images and real-time images of different modalities into a neural network model, and processing them through the region recommendation module, the feature fusion module, and the feature decoding module, the accurate position of the real-time image in the reference image is obtained.

[0056] according to Figure 1The flowchart of the cross-modal matching method of multi-scale reinforcement learning of the present invention uses a real-time image in SAR mode and a reference image in visible light mode, and outputs a matching probability map, in which the peak position is the exact position of the real-time image in the reference image. It is worth noting that the model can adapt to inputs of different modalities, and can obtain good matching results regardless of whether it is a SAR image, a visible light image, or a far-infrared image. The specific implementation steps are as follows: S1. Input the reference image and the real-time image of different modalities into the neural network model; S2. Based on the ResNet network, feature extraction is performed on the reference image and the real-time image, and the ResNet 2nd, 3rd, and 4th layer features are passed through the region recommendation module to obtain the relevant regions of the real-time image at three scales of the 2nd, 3rd, and 4th layer features in the reference image; S3. The features extracted by ResNet in step S2 are fused through the Transforme feature fusion module; S4. The fused features in step S3 are input into the feature decoding module to obtain the position of the real-time image in the reference image; S5. The matching result obtained in step S4 is output.

[0057] According to an embodiment of the present invention, the present invention performs feature extraction based on the ResNet18 network. Figure 2 The following is a flowchart of the overall ResNet18 implementation process of the present invention. Each residual block consists of the corresponding convolution operation and the residual connection mentioned above. In this method, the ResNet18 network is mainly used for feature extraction. Therefore, during implementation, the final fully connected layer and Softmax classification in the extraction network can be removed. If other networks are used for feature extraction, the fully connected layer and Softmax classification steps in the extraction network need to be considered.

[0058] According to an embodiment of the present invention, this method takes into account the influence of features extracted from different extraction layers, that is, features of different scales, on the matching results during the extraction process. Figure 1 As shown in Figure 2, the features after residual blocks 2, 3, and 4 in the real-time image feature extraction network and the benchmark image feature extraction network are extracted respectively, and the cross-correlation operation is performed using the region recommendation module to obtain the relevant area of ​​the real-time image in the benchmark image. The specific calculation process is as follows Figure 3 shown.

[0059] According to an embodiment of the present invention, Figure 3 The real-time graph features extracted at each stage are recorded as The real-time graph features extracted at each stage are recorded as Where B represents the batch size of the training data, c is the feature dimension, h i 、w i and H i 、W irepresents the feature size, and h i <H i , w i <W i The correlation characteristics It can be expressed as:

[0060]

[0061] Formula i=2,3,4 represents the i-th residual block; Represents convolution-related operations, including convolution, BatchNormalization, and ReLU; the symbol * represents cross-correlation operation.

[0062] Specifically, the features of the real-time image are used as convolution kernels to perform convolution operations with a step size of 1 on the features of the reference image, and correlation calculations are performed on each feature channel. The resulting correlation features are The size of (H i -h i +1)×(W i -w i +1).

[0063] According to an embodiment of the present invention, Figure 4 The calculation process diagram of the Transformer feature fusion module in the implementation process of the present invention is shown, and the real-time graph features extracted by the feature extraction network are and the base map features First, they are concatenated and transposed in the feature dimension to obtain the input of the Transformer fusion module in D = 2 × C, where N is the number of features after spatial flattening, H is the height of the input reference image, W is the width of the input reference image, h is the height of the input real-time image, and C is the dimension of the feature.

[0064] According to an embodiment of the present invention, the Transformer feature fusion module is similar to the traditional visual Transformer, which takes a slice patch sequence of image features as input and uses a stacked multi-head self-attention layer and a feedforward neural network to model its features. Specifically, for the input feature f in , is first segmented into a sequence of fixed-size 2D patches and a learnable positional encoding is added To add location information.

[0065] f=[f1;f : ;f3;…;f N ]+E pos

[0066] Similar to the visual transformer, the backbone network of the visual transformer consists of L blocks, each of which consists of a multi-head self-attention layer (MSA) and a feed-forward network (FFN). Among them, the single-head attention is calculated as follows:

[0067]

[0068] Among them, Q, K, and V are the query matrix, key matrix, and value matrix, respectively. k Is a scaling factor. In order to obtain more effective attention results for different feature subspaces, multi-head self-attention concatenates the outputs of multiple single-head attentions and projects them onto another parameter matrix:

[0069]

[0070]

[0071] in, is the parameter matrix in the i-th attention head of the l-th transformer block, f l The output of MSA is then sent to the feedforward neural network composed of MLP to generate the output f of the transformer block. l+1 The overall calculation process is as follows:

[0072] f l ′=MSA(f l )+f l ,f l+1 =FFN(f l ′)+f l '

[0073] The final feature is

[0074] f out After the transposition operation, the features input into the decoding module are obtained

[0075] According to an embodiment of the present invention, the implementation steps of the decoding feature module are:

[0076] The feature decoding module contains a total of five decoding layers, each of which consists of convolution, batch normalization, ReLU activation, and upsampling operations. The convolution kernel size is 3, the stride is 1, the padding is 1, and the output dimension is 1 / 2 of the input dimension; the upsampling scale is 2. That is, the features are processed through convolution, batch normalization, ReLU activation, and upsampling to obtain the output of each decoding layer. Through these four operations, the feature dimension is reduced by half, and the width and height are doubled.

[0077] First, the fused features First, the features are obtained through the first decoding layer Secondly, the correlation characteristics of the fourth residual block are First upsample to Same size After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1. Perform a residual connection with the activated value to obtain the output of the first decoding layer

[0078] Then, the output of the first decoding layer is input into the second decoding layer to obtain the feature Similarly, the correlation characteristics of the third residual block are First upsample to Same size After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1. Perform a residual connection with the activated value to obtain the output of the second decoding layer

[0079] Next, the output of the second decoding layer is input into the third decoding layer to obtain the feature Similarly, the correlation characteristics of the second residual block are First upsample to Same size After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1. Perform a residual connection with the activated value to obtain the output of the third decoding layer

[0080] Finally, the output of the third decoding layer is input to the fourth decoding layer to obtain the feature Then Input to the fifth decoding layer to get features Finally, a convolution operation with a kernel size of 3, a step length of 1, a padding of 1, and an output dimension of 1 is performed to reduce the dimension of the feature to 1. Then a Sigmoid activation operation is performed on the feature to make the eigenvalues ​​distributed between 0 and 1, thus obtaining the final output. It is a two-dimensional matrix with the same size as the input reference image, and its peak position represents the position of the real-time image in the reference image.

[0081] According to an embodiment of the present invention, a Sigmoid function is used in the final step to make the output feature distribution between 0 and 1, so as to emphasize the position of the real-time image in the reference image.

[0082] It is worth noting that this method introduces skip connections at different stages of decoding based on previous decoding ideas, such as Figure 5and Figure 1 As shown, an example of a jump connection is given. That is, the real-time image is subjected to a convolution operation, and the convolution operation here is not only a simple convolution operation, but also includes a normalization operation, an activation operation, and a pooling operation. The obtained residual block 2, residual block 3, and residual block 4 are input into the feature decoding module through the features that need to be directly decoded in the residual block extracted by the region recommendation module. Correspondingly, the reference image is subjected to a convolution operation, and the convolution operation here is not only a simple convolution operation, but also includes a normalization operation, an activation operation, and a pooling operation. The obtained residual block 2, residual block 3, and residual block 4 are input into the feature decoding module through the features that need to be directly decoded in the residual block extracted by the region recommendation module.

[0083] According to an embodiment of the present invention, the features accessed are output from the region recommendation modules at each stage, namely, the correlation features Specifically, first, the correlation feature After the Sigmoid function is applied to make the value distributed between 0 and 1, the input of the j+1 layer decoding module is It can be calculated by the following formula:

[0084]

[0085] According to an embodiment of the present invention, Figure 6 The matching effect of the model is shown, where the upper right corner is the real-time image of the SAR mode, the red cursor is the center of the real-time image and its exact position in the reference image, and the green cursor is the position of the real-time image in the reference image predicted by the model. By comparison, it can be clearly seen that the center of the real-time image and its exact position in the reference image obtained by the cross-modal matching method of the present invention almost coincide with the position of the real-time image in the reference image predicted by the model, and even the two have an intersection. In other words, by using the model matching method of the present invention, whether SAR images, visible light images or far-infrared images are input into the model trained by the invention, a good matching effect can be obtained.

[0086] Based on the above results, the learning model and learning method of the present invention have the following characteristics: (1) efficient and fast learning and testing, which means that fewer architectural parameters can be optimized, the training process can be accelerated, and the testing process can be carried out in parallel; (2) innovative use of the region recommendation module to perform cross-correlation operations to obtain the relevant regions of the real-time image in the reference image; (3) cross-modal matching is achieved, and the model can adapt to inputs of different modalities.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal image matching method based on multi-scale reinforcement learning, characterized in that: The method comprises the following steps: S1. Input the baseline image and the real-time image into the neural network model; S2. Feature extraction is performed on the reference image and the real-time image based on the ResNet network, and the 2nd, 3rd, and 4th layer features in the ResNet network are passed through the region recommendation module to obtain the relevant areas of the real-time image features at three scales of the 2nd, 3rd, and 4th layer features in the reference image features; Based on the ResNet network, a real-time image feature extraction network and a reference image feature extraction network are obtained. The residual block features in the real-time image feature extraction network and the reference image feature extraction network are extracted, and a cross-correlation operation is performed using the region recommendation module to obtain the relevant area of ​​the real-time image in the reference image. The specific calculation process of the cross-correlation operation is as follows: The real-time graph feature is represented as: The feature representation of the i-th real-time image residual block is: The benchmark image feature is expressed as: The feature representation of the residual block of the i-th reference image is: Among them, B is the batch size of the training data, c is the feature dimension, h i 、w i and H i 、W i is the feature size, and h i <H i , w i <W i , H is the height of the input reference image, W is the width of the input reference image, h is the height of the input real-time image, w is the width of the input real-time image, At this time, the correlation characteristics It can be expressed as: Among them, i represents the i-th residual block; Represents convolution-related operations, including convolution, Batch Normalization, and ReLU; the symbol * represents cross-correlation operation; S3. Fusion of features extracted by the ResNet network in step S2 through the Transformer feature fusion module; S4. The fusion feature in step S3 and the correlation feature in step S2 are input into the feature decoding module to obtain the position of the real-time map in the reference map; S5. Output the position matching result obtained in step S4.

2. The cross-modal image matching method using multi-scale reinforcement learning according to claim 1, characterized in that: The real-time image feature is used as the convolution kernel, and a convolution operation with a step size of 1 is performed on the reference image feature. Correlation calculation is performed correspondingly on each feature channel to obtain the correlation feature.

3. The cross-modal image matching method using multi-scale reinforcement learning according to claim 2, characterized in that: The real-time graph feature First, upsample to the same size as the baseline image feature by linear interpolation Post- and base map features The input feature f of the Transformer feature fusion module is obtained by concatenating and transposing in the feature dimension i4 , in, D = 2 × C, Where N is the number of features after spatial flattening, H is the height of the input reference image, W is the width of the input reference image, h is the height of the input real-time image, and c is the dimension of the feature.

4. The cross-modal image matching method using multi-scale reinforcement learning according to claim 3, characterized in that: The input feature f i4 is segmented into a sequence of fixed-size 2D patches, and then a learnable positional encoding is added To add position information, we get f, which is the feature obtained by adding a position code to each input feature. f=[f1;f2;f3;…;f N ]+E pos 。 5. The cross-modal image matching method using multi-scale reinforcement learning according to claim 4, characterized in that: The backbone network of the transformer in the Transformer feature fusion module consists of L blocks, each of which consists of a multi-head self-attention layer and a feed-forward network. The feed-forward network is denoted as FFN, where the single-head attention is calculated as follows: Among them, Q is the query matrix, K is the key matrix, V is the value matrix, d U is the scale factor, In order to obtain more effective attention results for different feature subspaces, multi-head self-attention concatenates the outputs of multiple single-head self-attentions and projects them onto another parameter matrix: in, is the parameter matrix in the i-th attention head of the l-th transformer block, f Z The output of MSA is sent to the feedforward neural network composed of MLP to generate the output f of the transformer block. l+1 , the calculation process is as follows: f l ′=MSA(f l )+f l ,f l+1 =FFN(f l ′)+f l ′ The final feature is f out After the transposition operation, the features input into the feature decoding module are obtained 6. The cross-modal image matching method using multi-scale reinforcement learning according to claim 5, characterized in that: The feature decoding module includes five decoding layers, namely the first decoding layer, the second decoding layer, the third decoding layer, the fourth decoding layer and the fifth decoding layer. Each decoding layer must undergo convolution, batch normalization, ReLU activation and upsampling operations to obtain the output features of each decoding layer.

7. The cross-modal image matching method using multi-scale reinforcement learning according to claim 6, characterized in that: The specific steps of decoding features by using the feature decoding module are as follows: 1) The fused features After the first decoding layer, the features are obtained Then the correlation feature obtained from the fourth residual block Up-sample to get The same size of h and w, so we get After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1, and then The output feature of the first decoding layer is obtained by performing residual connection with the activated value 2) Input the output features of the first decoding layer into the second decoding layer to obtain features Then the correlation features obtained by the third residual block Up-sample to get The same size of h and w, so we get After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1. Perform residual connection with the activated value to obtain the output features of the second decoding layer 3) Input the output features of the second decoding layer into the third decoding layer to obtain features Similarly, the correlation features obtained by the second residual block Up-sample and get The same size of h and w, so we get After the Sigmoid activation function, the eigenvalues ​​are distributed between 0 and 1. Perform residual connection with the activated value to obtain the output features of the third decoding layer 4) Input the output features of the third decoding layer into the fourth decoding layer to obtain features Then the features Input to the fifth decoding layer to obtain features 5) Features Perform convolution operation to reduce the dimension of the feature to 1, and then perform Sigmoid activation operation on the feature to make the feature value distributed between 0 and 1 to obtain the output feature The output feature f @qr The size of has the same two-dimensional matrix as the output feature size of the input reference image, and the peak position in the matrix represents the position of the real-time image in the reference image.

8. The cross-modal image matching method using multi-scale reinforcement learning according to claim 7, characterized in that: The convolution operation has a kernel size of 3, a stride of 1, a padding of 1, and an output dimension of 1 / 2 the input dimension. The upsampling operation has an amplification scale of 2.

Images