A deep learning image matching method based on joint representation attention mechanism

Abstract

The present invention relates to a deep learning image matching method based on a joint representation attention mechanism. The method comprises the following steps: step 1, using a SIFT algorithm and a NN matching algorithm to establish an initial matching set; step 2, using an improved network JRA‑Net to eliminate false matches and obtain a probability set of matches as internal points; step 3, treating the probability set output by step 2 as a weight set, and using a weighted 8-point algorithm to estimate the essential matrix; step 4, regressing the accurate relative posture (rotation and translation) through the essential matrix. The method illustrates that the method has the characteristics of high precision, high efficiency and high robustness through the relevant theoretical knowledge of the joint representation attention mechanism and a large number of experiments on public indoor and outdoor data sets, and can be widely used in computer vision tasks such as three-dimensional reconstruction, remote sensing image stitching, real-time positioning and map construction.

Description

Technical Field

[0001] The present invention relates to the field of image matching, and in particular to a deep learning image matching method based on a joint representation attention mechanism. Background Art

[0002] Traditional image matching plays a very important role in computer vision tasks such as motion recovery structure, image stitching, simultaneous localization and mapping, and 3D reconstruction. It is an important topic in the fields of computer vision, pattern recognition, image analysis, security, and remote sensing. Usually, image matching includes three steps, namely feature point extraction and description, establishing an initial matching set, and false matching removal. Due to large perspective changes, illumination changes, and severe occlusion in image pairs, the initial matching set contains a large number of outliers. False matching removal, as a key post-processing step, can retain inliers and remove outliers from the initial matching set to improve the matching accuracy. Therefore, studying a high-precision, high-efficiency, and high-robust image matching method has extremely important theoretical research significance and practical application value in the era of global automation and artificial intelligence.

[0003] In the past few decades, researchers have proposed many different types of image matching methods. According to recent literature and research results, image matching methods can be divided into three categories: direct matching, indirect matching, and deep learning-based matching. The purpose of direct matching is to directly use spatial geometric relationships and optimization methods to establish the correspondence between two given feature sets, which is divided into graph matching and point set registration. The biggest difference between indirect matching and direct matching is the use of local feature descriptors, such as scale-invariant feature transform (SIFT) and speeded up robust features (SURF). First, the initial match is established through the local feature descriptor, and then the classic algorithms, such as random sampling consensus algorithm (RANSAC) and local preserving matching (LPM), are used to eliminate false matches. In recent years, due to the powerful data-driven expression ability and deep feature acquisition ability, feature matching algorithms based on deep learning have become popular and have achieved good results on large data sets. For example, LFGC proposed that due to the disordered and irregular nature of the initial matches, the deep learning network needs to have permutation and variability. To this end, they designed a module called PointCN, which mainly uses multi-layer perceptrons (MLPs) to process each match individually, and context normalization (CN) to capture global context information. OANet proposed a differentiable pooling layer and an upper pooling layer (DiffPool & DiffUnpool layer) to capture geometric context information. However, the CN operation normalizes the feature map based on the mean and variance. This approach of treating each initial match equally is unreasonable for the initial match set dominated by outliers, which will lead to suboptimal matching results. Therefore, how to treat each match discriminatively in the network learning stage is a challenging problem, which is of great significance to further improve the accuracy of image matching. Summary of the invention

[0004] The purpose of the present invention is to solve the defects of the above-mentioned background technology and provide a deep learning image matching method based on a joint representation attention mechanism.

[0005] To achieve the above object, the technical solution of the present invention is: a deep learning image matching method based on a joint representation attention mechanism, comprising the following steps:

[0006] Step 1: Use SIFT algorithm and NN matching algorithm to establish the initial matching set;

[0007] Step 2: Use the improved network JRA-Net to eliminate false matches and obtain the probability set of matching as internal points;

[0008] Step 3: Consider the probability set output from step 2 as a weight set and use the weighted 8-point algorithm to estimate the essential matrix;

[0009] Step 4: Regress the accurate relative pose through the essential matrix.

[0010] In one embodiment of the present invention, step 1 is specifically implemented as follows:

[0011] Step 1.1, extract key point coordinates and descriptors from the image pair I and I′ using the SIFT algorithm;

[0012] Step 1.2: Based on the similarity of the descriptors and the distance determined from the measurement space, establish the initial matching set C:

[0013]

[0014] where c i represents the i-th match, N represents the total number of matches, and denote the normalized coordinates of image pair I and I′ respectively.

[0015] In one embodiment of the present invention, step 2 is specifically implemented as follows:

[0016] Step 2.1: Take the N×4 initial matching set C obtained in step 1.2 as the input of the deep learning network; after passing through the three PointCN blocks in the context extraction layer, a feature map F1∈R is obtained. N×128 ; The PointCN block consists of two groups of multilayer perceptrons with the same structure. The multilayer perceptron consists of a context normalization CN layer, a Batch normalization layer, a ReLU activation function and a 128-neuron network. The CN layer is used to extract global context information, and the Batch normalization layer helps to accelerate network convergence;

[0017] Step 2.2: Send the output of step 2.1 to the DiffPool & DiffUnpool layer in the geometry extraction layer and output a feature map F2∈R N×128 ; The DiffPool&DiffUnpool layer includes a differentiable pooling layer, three order-aware filtering layers, and a differentiable up-pooling layer;

[0018] Step 2.3: Input the output results of step 2.1 and step 2.2 into the first joint representation attention block JRABlock 1, and recalibrate the output results of step 2.1 and step 2.2 respectively; the feature dimension after recalibration is still 128; JRA Block 1 contains a global attention block GA Block and a local attention block LA Block;

[0019] Step 2.4: Send the output results F″1, F″2 of step 2.3 to the second joint attention block JRA Block 2 for further and more detailed recalibration; the feature dimension after recalibration is still 128;

[0020] Step 2.5: After splicing the output results of step 2.4, pass them through the three PointCN blocks in the context extraction layer to obtain a feature map with strong expressive ability;

[0021] Step 2.6: Input the output result of step 2.5 to the probability prediction layer Prob Predictor to obtain the probability of each match being an internal point.

[0022] In one embodiment of the present invention, step 2.2 is specifically implemented as follows:

[0023] Step 2.2.1: Map the N pairs of matches in F1 into M clusters through a differentiable pooling layer to obtain F pool ∈R M×128 ;

[0024] The output results of step 2.2.2 and step 2.2.1 are passed through three sequential perceptual filtering modules to capture geometric information;

[0025] The output results of step 2.2.3 and step 2.2.2 are passed through a differentiable upper pooling layer to map the M clusters back to N pairs of matches; this process is the inverse process of the differentiable pooling layer.

[0026] In one embodiment of the present invention, step 2.3 is specifically implemented as follows:

[0027] Step 2.3.1. Design a global attention block GA Block. By learning the global joint representation of the feature maps of the two inputs, they are recalibrated for the channels respectively, so that the network pays more attention to the important channels and suppresses the unimportant channels. The details are as follows:

[0028] First, GABlock aggregates the spatial global information of the input feature maps F1 and F2 through two global pooling layers to obtain two channel statistics S1, S2∈R 1×128 ;

[0029] Then, the global joint representation S is obtained by concatenating g , the formula is as follows:

[0030] S g =MLP(Concat(S1, S2)) (2)

[0031] Where MLP represents a multi-layer perceptron, and Concat represents a concatenation operation along the channel dimension;

[0032] Next, based on the global joint representation S g , through two groups, each with two multi-layer perceptrons with channel dimension of 128, the channel recalibration weights are learned The formula is as follows:

[0033]

[0034] Where δ represents the ReLU activation function and θ represents the Sigmoid function;

[0035] Finally, the weights are recalibrated using the channel Recalibrate the input feature maps by channels respectively; get the recalibrated feature maps F′1, F′2∈R N×128 ; The formula is as follows:

[0036]

[0037] where λ i is a learnable parameter with an initial value of 0, allowing the network to learn a suitable weight for the global attention block, ⊙ represents channel-by-channel multiplication;

[0038] Step 2.3.2, design a local attention block LA Block, by learning the local joint representation of the feature maps of the two inputs, recalibrating them element by element, so that the network pays more attention to important elements and suppresses unimportant elements, as follows:

[0039] First, LABlock retains the spatial local information of the two input feature maps F′1 and F′2; the local joint representation S is obtained by concatenating and learning. l , the formula is as follows:

[0040] S l =MLP(Concat(F′1,F′2)) (5)

[0041] Then, based on the local joint representation S l , recalibrating the weights through two groups of multilayer perceptron learning elements with two channel dimensions of 128 each The formula is as follows:

[0042]

[0043] Finally, recalibrate the weights using the elements Recalibrate the input feature maps element by element respectively; get the recalibrated feature maps F″1, F″2∈R N×128 ; The formula is as follows:

[0044]

[0045] where μ i is a learnable parameter with an initial value of 0, allowing the network to learn a suitable weight for the local attention block; Represents element-wise multiplication.

[0046] In one embodiment of the present invention, step 2.6 is specifically implemented as follows:

[0047] First, the feature map output from step 2.5 is converted into a logical value o∈R through a multilayer perceptron consisting of 1 neuron N×1 ;

[0048] Then, through a ReLU and a tanh activation function, we get the probability set Q∈R N×1 ; The formula is as follows:

[0049] Q = tanh(ReLU(o)) (8)

[0050] Q=[q1,q2,......,q i ,......,q N ] (9)

[0051] where q i ∈[0, 1) represents the probability that the i-th pair of matches belongs to an interior point;

[0052] Iterate JRA-Net once, and use the network’s first output probability set Q and residual information as the second input of the input network to obtain a new probability set Q′=[q′1, q′2, ..., q′ i ......,q′ N ], where q′ i ∈[0, 1) represents the probability that the i-th pair of matches belongs to an interior point.

[0053] In one embodiment of the present invention, in step 3, the essential matrix is ​​estimated by a weighted 8-point algorithm. The formula is as follows:

[0054]

[0055] in, is the essential matrix of network estimation, g(·) is the weighted 8-point algorithm, and C is the initial matching set.

[0056] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a deep learning image matching method based on a joint representation attention mechanism, in which the present invention designs a new attention mechanism module, which includes two sub-attention blocks, namely global and local attention blocks. The attention module takes a feature map containing global context information and geometric context information as input, and captures the correlation between global context information and geometric context information through two sub-attention blocks based on the global and local joint representations of the two input features, and uses this correlation to recalibrate the channels and elements of the original feature map, enhance important information, and suppress unimportant information. The JRA-Net of the present invention can effectively remove mismatches and estimate relative poses when a large number of outliers are included in the initial matching set, which is crucial for computer vision tasks such as motion recovery structure, image splicing, simultaneous positioning and mapping, and three-dimensional reconstruction. A large number of experiments on public indoor and outdoor data sets show that the performance of the JRA-Net of the present invention in mismatch removal and relative pose estimation tasks has reached the most advanced level. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 For the present invention Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention.

[0058] Figure 2 This is a network framework diagram of an embodiment of the present invention.

[0059] Figure 3 Figure 2 is a diagram of the global attention block framework.

[0060] Figure 4 Figure 2 is a diagram of the local attention block framework.

[0061] Figure 5 The effect diagram of feature matching in the indoor and outdoor public datasets of the embodiment of the present invention. Among them, (a) and (b) are scenes in the YFCC100M dataset, and (c) and (d) are scenes in the SUN3D dataset. DETAILED DESCRIPTION

[0062] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0063] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0064] The present invention provides a deep learning image matching method based on a joint representation attention mechanism, comprising the following steps:

[0065] Step 1: Use SIFT algorithm and NN matching algorithm to establish the initial matching set;

[0066] Step 2: Use the improved network JRA-Net to eliminate false matches and obtain the probability set of matching as internal points;

[0067] Step 3: Consider the probability set output from step 2 as a weight set and use the weighted 8-point algorithm to estimate the essential matrix;

[0068] Step 4: Regress the accurate relative pose through the essential matrix.

[0069] The following is the specific implementation process of the present invention.

[0070] Reference Figure 1 It can be seen that the present invention mainly consists of four steps: Step 1, using SIFT algorithm and NN matching algorithm to establish an initial matching set C; Step 2, through the improved network JRA-Net (such as Figure 2 As shown in the figure, the false matches are eliminated to obtain the probability set Q' of the matched internal points. Step 3: The probability set obtained in step 2 is regarded as a weight set, and the essential matrix is ​​estimated by the weighted 8-point algorithm; Step 4: The accurate relative posture (translation, rotation) is restored according to the essential matrix.

[0071] Step 1: Use SIFT algorithm and NN matching to establish the initial matching set, including the following steps:

[0072] Step 1.1, extract key point coordinates and descriptors from the image pair I and I′ using the SIFT algorithm;

[0073] Step 1.2: According to the similarity of the descriptors and the distance determined from the measurement space, an initial matching set C is established: C = [c1, c2, ..., c i ,......,c N ]∈R N×4 ,

[0074] where c i represents the i-th match, N represents the total number of matches, and denote the normalized coordinates of image pair I and I′ respectively.

[0075] Step 2: Use the improved network JRA-Net to eliminate false matches and obtain the probability set of matching as internal points, as follows:

[0076] Step 2.1: Take the N×4 initial matching set C obtained in step 1.2 as the input of the deep learning network. After the three PointCN blocks in the context extraction layer, a feature map F1∈R is obtained. N×128 .

[0077] The PointCN block consists of two sets of identical structures (a context normalization (CN) layer, a batch normalization layer, a ReLU activation function, and a multi-layer perceptron consisting of 128 neurons). The CN layer is used to extract global context information, and the batch normalization layer helps accelerate network convergence.

[0078] Step 2.2: Send the output of step 2.1 to the DiffPool & DiffUnpool layer in the geometry extraction layer and output a feature map F2∈R N×128 .

[0079] The DiffPool&DiffUnpool layer includes a differentiable pooling layer, three sequence-aware filtering layers, and a differentiable up-pooling layer.

[0080] Step 2.2.1. First, through a differentiable pooling layer, the N pairs of matches in F1 are mapped into M clusters to obtain F pool ∈R M×128 .

[0081] Step 2.2.2. Then, the output of step 2.2.1 passes through three sequential perceptual filtering modules to capture geometric information.

[0082] Step 2.2.3. Finally, the output of step 2.2.2 is passed through a differentiable upper pooling layer to map the M clusters back to N pairs of matches. This process is the inverse process of the differentiable pooling layer.

[0083] Step 2.3: Input the output results of step 2.1 and step 2.2 into the first joint representation attention block (JRA Block 1) to recalibrate the output results of step 2.1 and step 2.2 respectively. The feature dimension after recalibration is still 128.

[0084] JRA Block 1 contains a global attention block (GA Block) and a local attention block (LABlock).

[0085] Step 2.3.1. The present invention designs a global attention block (GA Block), which learns the global joint representation of the feature maps of the two inputs and recalibrates their channels respectively, so that the network pays more attention to important channels and suppresses unimportant channels. Its structure is as follows Figure 3 shown.

[0086] First, the module aggregates the spatial global information of the input feature maps F1 and F2 through two global pooling layers to obtain two channel statistics S1, S2∈R 1×128 .

[0087] Then, the global joint representation S is obtained by concatenating g , the formula is as follows: g =MLP(Concat(S1, S2))

[0088] Where MLP represents a multi-layer perceptron, and Concat represents a concatenation operation along the channel dimension.

[0089] Next, based on the global joint representation S g , through two groups, each with two multi-layer perceptrons with channel dimension of 128, the channel recalibration weights are learned The formula is as follows:

[0090] Where δ represents the ReLU activation function and θ represents the Sigmoid function.

[0091] Finally, the weights are recalibrated using the channel Recalibrate the input feature maps by channels. Get the recalibrated feature maps F′1, F′2∈R N×128 The formula is as follows:

[0092] where λ i is a learnable parameter with an initial value of 0, allowing the network to learn a suitable weight for the global attention block, and ⊙ represents the channel-by-channel multiplication

[0093] Step 2.3.2, the present invention designs a local attention block (LA Block), which learns the local joint representation of the feature maps of the two inputs and recalibrates them element by element, so that the network pays more attention to important elements and suppresses unimportant elements. Its structure is as follows Figure 4 shown.

[0094] First, the module retains the spatial local information of the two input feature maps F′1, F′2. The local joint representation S is obtained by concatenating the feature maps l , the formula is as follows: l =MLP(Concat(F′1,F′2)).

[0095] Then, based on the local joint representation S l , recalibrating the weights through two groups of multilayer perceptron learning elements with two channel dimensions of 128 each The formula is as follows:

[0096] Finally, recalibrate the weights using the elements Recalibrate the input feature maps element by element. Get the recalibrated feature maps F″1, F″2∈RN×128 The formula is as follows:

[0097] where μ i is a learnable parameter with an initial value of 0, allowing the network to learn an appropriate weight for the local attention block. Represents element-wise multiplication.

[0098] Step 2.4: Send the output results F″1, F″2 of step 2.3 to the second joint attention block (JRA Block 2) for further and more detailed recalibration. The feature dimension after recalibration is still 128.

[0099] Step 2.5: After splicing the output results of step 2.4, pass them through the three PointCN blocks in the context extraction layer to obtain a feature map with strong expressive ability.

[0100] Step 2.6: Input the output of step 2.5 to the Prob Predictor layer to obtain the probability of each match being an inlier.

[0101] The Prob Predictor contains a multi-layer perceptron consisting of 1 neuron, a ReLU, and a tanh activation function.

[0102] Step 2.6.1. First, the feature map output from step 2.5 is converted into a logical value o∈R by a multilayer perceptron consisting of 1 neuron. N×1 .

[0103] Then, through a ReLU and a tanh activation function, we get the probability set Q∈R N×1 The formula is as follows: Q = tanh(ReLU(o)).

[0104] Q=[q1,q2,......,q i ,......,q N ], where q i ∈[0, 1) represents the probability that the i-th pair of matches belongs to an interior point.

[0105] Iterate JRA-Net once, and use the network’s first output probability set Q and residual information as the second input of the input network to obtain a new probability set Q′=[q′1, q′2, ..., q′ i , ..., q′ N ], where q′ i ∈[0, 1) represents the probability that the i-th pair of matches belongs to an interior point.

[0106] Step 3: Use the probability set Q′ output from step 2 as the weight set and estimate the essential matrix using the weighted 8-point algorithm

[0107] The formula is as follows:

[0108] in, is the essential matrix of the network estimation of the present invention, g(·) is the weighted 8-point algorithm, and C is the initial matching set.

[0109] Step 4: Restore the accurate relative posture (rotation and translation) according to the essential matrix obtained in step 3.

[0110] PointNet++, LFGC, DFE, ACNet, OANet, OANet++ and the JRA-Net of the present invention were selected to conduct relative pose estimation experiments on the YFCC100M and SUN3D datasets, with mean average precision (mAP) as the evaluation index and an error threshold of 5°. As shown in Table 1, the bold font indicates the best performance.

[0111] Table 1

[0112]

[0113] The method of the present invention is tested on outdoor and indoor public data sets, and the experimental results show that the present invention achieves the best effect among all the methods involved in the comparison. It can be seen from practice that the present invention can be widely used in the fields of remote sensing image stitching and three-dimensional reconstruction.

[0114] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A deep learning image matching method based on joint representation attention mechanism, characterized in that: The following steps are involved: Step 1: Use SIFT algorithm and NN matching algorithm to establish the initial matching set; Step 2: Use the improved network JRA-Net to eliminate false matches and obtain the probability set of matching as internal points; Step 3: Consider the probability set output from step 2 as a weight set and use the weighted 8-point algorithm to estimate the essential matrix; Step 4: Regress the accurate relative posture through the essential matrix; The step 1 is specifically implemented as follows: Step 1.1, extract key point coordinates and descriptors from the image pair I and I' using the SIFT algorithm; Step 1.2: Based on the similarity of the descriptors and the distance determined from the measurement space, establish the initial matching set C: where c i represents the i-th match, N represents the total number of matches, and Represent the normalized coordinates of image pairs I and I' respectively; The step 2 is specifically implemented as follows: Step 2.1: Take the N×4 initial matching set C obtained in step 1.2 as the input of the deep learning network; after passing through the three PointCN blocks in the context extraction layer, a feature map F1∈R is obtained. N×128 ; The PointCN block consists of two groups of multilayer perceptrons with the same structure. The multilayer perceptron consists of a context normalization CN layer, a Batch normalization layer, a ReLU activation function and a 128-neuron network. The CN layer is used to extract global context information, and the Batch normalization layer helps to accelerate network convergence; Step 2.2: Send the output of step 2.1 to the DiffPool & DiffUnpool layer in the geometry extraction layer and output a feature map F2∈R N×128 ; The DiffPool & DiffUnpool layer includes a differentiable pooling layer, three order-aware filtering layers, and a differentiable up-pooling layer; Step 2.3: Input the output results of step 2.1 and step 2.2 into the first joint representation attention block JRABlock 1, and recalibrate the output results of step 2.1 and step 2.2 respectively; the feature dimension after recalibration is still 128; JRABlock1 contains a global attention block GABlock and a local attention block LABlock; Step 2.4: Send the output results F”1, F”2 of step 2.3 to the second joint attention block JRABlock 2 for further and more detailed recalibration; the feature dimension after recalibration is still 128; Step 2.5: After splicing the output results of step 2.4, pass them through the three PointCN blocks in the context extraction layer to obtain a feature map with strong expressive ability; Step 2.6: Input the output result of step 2.5 to the probability prediction layer Prob Predictor to obtain the probability of each match being an internal point.

2. A deep learning image matching method based on joint representation attention mechanism according to claim 1, characterized in that: The step 2.2 is specifically implemented as follows: Step 2.2.1: Map the N pairs of matches in F1 into M clusters through a differentiable pooling layer to obtain F pool ∈R M ×128 ; The output results of step 2.2.2 and step 2.2.1 are passed through three sequential perceptual filtering modules to capture geometric information; The output results of step 2.2.3 and step 2.2.2 are passed through a differentiable upper pooling layer to map the M clusters back to N pairs of matches; this process is the inverse process of the differentiable pooling layer.

3. A deep learning image matching method based on joint representation attention mechanism according to claim 1, characterized in that: The step 2.3 is specifically implemented as follows: Step 2.3.

1. Design a global attention block GABlock. By learning the global joint representation of the feature maps of the two inputs, they are recalibrated for the channels respectively, so that the network pays more attention to the important channels and suppresses the unimportant channels. The details are as follows: First, GABlock aggregates the spatial global information of the input feature maps F1 and F2 through two global pooling layers to obtain two channel statistics S1, S2∈R 1×128 ; Then, the global joint representation S is obtained by concatenating g , the formula is as follows: S g =MLP(Concat(S1,S2)) (2) Where MLP represents a multi-layer perceptron, and Concat represents a concatenation operation along the channel dimension; Next, based on the global joint representation S g , through two groups, each with two multi-layer perceptrons with channel dimension of 128, the channel recalibration weights are learned The formula is as follows: Where δ represents the ReLU activation function and θ represents the Sigmoid function; Finally, use the channel to recalibrate the weights Recalibrate the input feature maps by channels respectively; get the recalibrated feature maps F'1, F'2∈R N×128 ; The formula is as follows: where λ i is a learnable parameter with an initial value of 0, allowing the network to learn a suitable weight for the global attention block, ⊙ represents channel-by-channel multiplication; Step 2.3.2, design a local attention block LABlock, by learning the local joint representation of the feature maps of the two inputs, recalibrating them element by element, so that the network pays more attention to important elements and suppresses unimportant elements, as follows: First, LABlock retains the spatial local information of the two input feature maps F'1, F'2; the local joint representation S is obtained by concatenating and learning l , the formula is as follows: S l =MLP(Concat(F’1,F’2)) (5) Then, based on the local joint representation S l , recalibrating the weights through two groups of multilayer perceptron learning elements with two channel dimensions of 128 each The formula is as follows: Finally, recalibrate the weights using the elements Recalibrate the input feature maps element by element respectively; get the recalibrated feature maps F”1,F”2∈R N×128 ; The formula is as follows: where μ i is a learnable parameter with an initial value of 0, allowing the network to learn a suitable weight for the local attention block; Represents element-wise multiplication.

4. The deep learning image matching method based on the joint representation attention mechanism according to claim 1 is characterized in that: The step 2.6 is specifically implemented as follows: First, the feature map output from step 2.5 is converted into a logical value o∈R through a multilayer perceptron consisting of 1 neuron N×1 ; Then, through a ReLU and a tanh activation function, we get the probability set Q∈R N×1 ; The formula is as follows: Q = tanh(ReLU(o)) (8) Q=[q1,q2,……,q i ,……,q N ] (9) where q i ∈[0,1) represents the probability that the i-th pair of matches belongs to an interior point; Iterate JRA-Net once, and use the network's first output probability set Q and residual information as the second input of the input network to obtain a new probability set Q' = [q'1, q'2, ..., q' i ,……,q' N ], where q' i ∈[0,1) represents the probability that the i-th pair of matches belongs to an interior point.

5. A deep learning image matching method based on joint representation attention mechanism according to claim 4, characterized in that: In step 3, the essential matrix is ​​estimated by the weighted 8-point algorithm. The formula is as follows: in, is the essential matrix of network estimation, g(·) is the weighted 8-point algorithm, and C is the initial matching set.

Images