Model training methods, key point matching methods, devices, equipment and media

Abstract

This application discloses a training method, apparatus, device, and medium for a keypoint matching model, relating to the field of image processing technology. The method includes: acquiring an original training image and performing image transformation on the original training image to obtain a transformed image; inputting the original training image and the transformed image into an initial keypoint matching model, so that the keypoint detection head and self-attention matching detection head in the initial keypoint matching model respectively output a first keypoint position feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint position feature map and a second keypoint matching feature map corresponding to the transformed image; constructing loss functions using the first and second keypoint position feature maps and keypoint annotation information respectively, and constructing a loss function using the first and second keypoint matching feature maps; and training the initial keypoint matching model using each loss function to obtain a trained keypoint matching model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to model training methods, key point matching methods, devices, equipment and media. Background Technology

[0002] Currently, keypoint matching methods mainly include traditional image algorithms and deep learning-based keypoint matching methods. Traditional image algorithms include methods using Euclidean distance and Hamming distance, but these methods suffer from poor robustness, high computational complexity, and dependence on specific scenes. Deep learning-based keypoint matching methods primarily use convolutional neural networks to extract image feature representations and then match feature points using matching strategies. Generally, deep learning-based keypoint matching methods outperform traditional methods, but they still face challenges such as high false detection rates, computational complexity, and difficulties in data annotation.

[0003] For deep learning, deep learning models are usually constrained by specific datasets and tasks during the training phase. In real-world scenarios, there are many scene changes, so a single dataset cannot guarantee the generalization of key point matching models in real-world environments.

[0004] In summary, improving the robustness of keypoint matching models to image scene changes in order to enhance the accuracy of subsequent keypoint matching is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a model training method, a keypoint matching method, an apparatus, a device, and a medium that can improve the robustness of the keypoint matching model to image scene changes, thereby improving the accuracy of subsequent keypoint matching. The specific solution is as follows:

[0006] Firstly, this application discloses a training method for a keypoint matching model, comprising:

[0007] Obtain the original training image and perform image transformation on the original training image to obtain the transformed image;

[0008] The original training image and the transformed image are input into the initial keypoint matching model, so that the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model can output the first keypoint position feature map and the first keypoint matching feature map corresponding to the original training image, and the second keypoint position feature map and the second keypoint matching feature map corresponding to the transformed image, respectively.

[0009] A first loss function and a second loss function are constructed using the first key point location feature map and the second key point location feature map, respectively, and the third loss function is constructed using the first key point matching feature map and the second key point matching feature map.

[0010] The initial keypoint matching model is trained using the first loss function, the second loss function, and the third loss function to obtain the trained keypoint matching model.

[0011] Optionally, the step of performing image transformation on the original training image to obtain the transformed image includes:

[0012] The original training image is transformed using a preset image transformation model and a homography transformation matrix to obtain a transformed image.

[0013] Optionally, the initial keypoint matching model includes a backbone network, the keypoint detection head, and the self-attention matching detection head;

[0014] Accordingly, the step of inputting the original training image and the transformed image into the initial keypoint matching model to output a first keypoint location feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint location feature map and a second keypoint matching feature map corresponding to the transformed image, includes:

[0015] The original training image and the transformed image are input into the backbone network to extract the first image features of the original training image and the second image features of the transformed image through the backbone network.

[0016] The first image features are input to the key point detection head and the self-attention matching detection head, respectively, so as to output the first key point position feature map through the key point detection head and the first key point matching feature map through the self-attention matching detection head;

[0017] The second image features are input to the key point detection head and the self-attention matching detection head, respectively, so that the key point detection head outputs a second key point position feature map, and the self-attention matching detection head outputs a second key point matching feature map.

[0018] Optionally, the self-attention matching detection head processes the input feature map based on a self-attention mechanism to obtain an output feature map. The processing flow of the self-attention matching detection head is as follows:

[0019] The input feature map is transformed into a query vector, a key vector, and a value vector based on a self-attention mechanism.

[0020] The query vector and the key vector are weighted using a preset weight matrix to obtain a weighted feature vector, and the weighted feature vector is then calculated using a preset activation function to obtain an image weighting matrix.

[0021] Calculate the similarity score between the image weighting matrix and the value vector, and add the similarity score to the input feature map to obtain the output feature map.

[0022] Optionally, the training method for the keypoint matching model further includes:

[0023] Construct the first loss function and the second loss function based on the preset loss function;

[0024] The third loss function is constructed based on a weighted bilateral contrastive loss function.

[0025] Optionally, the training method for the keypoint matching model further includes:

[0026] The feature extraction network is trained using the generated virtual dataset;

[0027] The trained feature extraction network is used to extract features from the original training image to obtain the key point coordinate information of the original training image, and the key point annotation information is generated based on the key point coordinate information.

[0028] Secondly, this application discloses a keypoint matching method, including:

[0029] The current frame image is acquired, and the key point matching model trained using the aforementioned disclosed key point matching model training method is used to detect the current frame image to obtain the current key point position feature map and the current key point matching feature map.

[0030] The corresponding feature association matrix is ​​calculated using the current keypoint matching feature map and the keypoint matching feature map of the previous frame image, and the feature association matrix is ​​decoded to obtain the keypoint matching result between the current frame image and the previous frame image.

[0031] Thirdly, this application discloses a training device for a keypoint matching model, comprising:

[0032] The image transformation module is used to acquire the original training image and perform image transformation on the original training image to obtain the transformed image;

[0033] The feature output module is used to input the original training image and the transformed image into an initial keypoint matching model, so as to output a first keypoint position feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint position feature map and a second keypoint matching feature map corresponding to the transformed image, respectively, through the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model.

[0034] The loss construction module is used to construct a first loss function and a second loss function by using the first key point location feature map and the second key point location feature map with the key point annotation information, respectively, and to construct a third loss function by using the first key point matching feature map and the second key point matching feature map;

[0035] The model training module is used to train the initial keypoint matching model using the first loss function, the second loss function, and the third loss function to obtain the trained keypoint matching model.

[0036] Fourthly, this application discloses an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is configured to execute the computer program to implement the steps of the aforementioned training method or keypoint matching method for the disclosed keypoint matching model.

[0039] Fifthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned training method or key point matching method for the disclosed key point matching model.

[0040] As can be seen, this application acquires the original training image and performs image transformation on the original training image to obtain a transformed image; the original training image and the transformed image are input into an initial keypoint matching model, so that the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model respectively output a first keypoint position feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint position feature map and a second keypoint matching feature map corresponding to the transformed image; the first keypoint position feature map and the second keypoint position feature map are used with keypoint annotation information to construct a first loss function and a second loss function, respectively, and the first keypoint matching feature map and the second keypoint matching feature map are used to construct a third loss function; the initial keypoint matching model is trained using the first loss function, the second loss function and the third loss function to obtain a trained keypoint matching model. Therefore, this application requires image transformation processing of the original training image to obtain a transformed image, and inputs both the original training image and the transformed image into an initial keypoint matching model. The keypoint detection head and the self-attention matching detection head in the initial keypoint matching model output the first keypoint position feature map and the first keypoint matching feature map corresponding to the original training image, and output the second keypoint position feature map and the second keypoint matching feature map corresponding to the transformed image, respectively. In this way, by introducing the transformed image during the training stage, the model obtained after training can have stronger generalization ability, be more suitable for image transformation scenarios, and have better robustness. Furthermore, by introducing the self-attention detection head into the model, this application can pay attention to global features, thereby improving matching efficiency. Furthermore, a first loss function and a second loss function are constructed using the first and second keypoint location feature maps and keypoint annotation information, respectively, to reduce the difference between the keypoint location features extracted by the keypoint matching model and the initial keypoint annotation information. A third loss function is also constructed using the first and second keypoint matching feature maps to reduce the keypoint matching difference between the original training image and the transformed image. Finally, the first, second, and third loss functions are used together to train the initial keypoint matching model, resulting in the trained keypoint matching model. This application improves the robustness of the keypoint matching model to image scene transformations and further enhances the accuracy of subsequent keypoint matching. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a flowchart of a training method for a keypoint matching model disclosed in this application;

[0043] Figure 2 This is a flowchart illustrating a specific keypoint matching model training method disclosed in this application;

[0044] Figure 3 This is a schematic diagram of the structure of an initial keypoint matching model disclosed in this application;

[0045] Figure 4 This is a schematic diagram of the structure of a key point detection head disclosed in this application;

[0046] Figure 5 This is a schematic diagram of the structure of a self-attention matching detection head disclosed in this application;

[0047] Figure 6 Here is a flowchart of a keypoint matching method disclosed in this application;

[0048] Figure 7 This is a schematic diagram of the training device structure for a keypoint matching model disclosed in this application;

[0049] Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0051] For deep learning, deep learning models are often constrained by specific datasets and tasks during the training phase. However, real-world scenarios involve numerous scene changes, making it impossible to guarantee the generalization ability of a keypoint matching model in real-world environments using a single dataset. Therefore, this application discloses a model training method, a keypoint matching method, an apparatus, a device, and a medium that can improve the robustness of the keypoint matching model to image scene changes, thereby enhancing the accuracy of subsequent keypoint matching.

[0052] See Figure 1 As shown in the embodiment of this application, a training method for a keypoint matching model is disclosed, the method comprising:

[0053] Step S11: Obtain the original training image and perform image transformation on the original training image to obtain the transformed image.

[0054] In this embodiment, after acquiring the original training image, it is necessary to perform image transformation processing on the original training image to obtain a transformed image. In a specific implementation, the above-mentioned image transformation of the original training image to obtain the transformed image includes: using a preset image transformation model and a homography transformation matrix to transform the original training image to obtain the transformed image. It is understood that this is to address situations where scene changes occur in real-world scenarios, such as changes in lighting, angle, environment, weather, and scaling, leading to mismatches between key points in consecutive frames. Specifically, the preset image transformation model in this application uses a CycleGAN (Cycle-Consistent Generative Adversarial Networks) model to generate images under different weather conditions, and then uses a homography transformation matrix to perform image transformation to simulate the situation between consecutive frames when there are corners and rotations. This ensures that key points in consecutive frames can still be correlated under different lighting conditions, different environments, and even with large corners, thereby achieving key point recognition and tracking. In addition to the CycleGAN model, other preset image transformation models that can achieve the same image transformation effects include StarGAN and pix2pix models. By introducing transformed images during the training phase, the resulting model exhibits stronger generalization ability, is more suitable for image transformation scenarios, and has better robustness.

[0055] Step S12: Input the original training image and the transformed image into the initial keypoint matching model, so that the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model can output the first keypoint position feature map and the first keypoint matching feature map corresponding to the original training image, and the second keypoint position feature map and the second keypoint matching feature map corresponding to the transformed image, respectively.

[0056] In this embodiment, both the original training image and the transformed image are input into an initial keypoint matching model. The initial keypoint matching model outputs a first keypoint position feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint position feature map and a second keypoint matching feature map corresponding to the transformed image, respectively, through a keypoint detection head and a self-attention matching detection head. This application improves matching efficiency by introducing a self-attention matching detection head into the model, which can focus on global features. It should be noted that the self-attention matching detection head in this application uses a self-attention mechanism, which is a mechanism that can focus on global features when processing image sequence data. In the self-attention detection head, a weight distribution can be obtained by calculating the similarity between different positions in the input image sequence. This weight distribution reflects the correlation between different positions. Therefore, the self-attention mechanism can effectively capture global features in the input image sequence, unlike traditional convolutional neural networks which can only focus on local features.

[0057] Step S13: Construct a first loss function and a second loss function using the first key point location feature map and the second key point location feature map, respectively, and construct a third loss function using the first key point matching feature map and the second key point matching feature map.

[0058] In this embodiment, a first loss function and a second loss function are constructed using the first keypoint location feature map and the second keypoint location feature map, respectively, and the keypoint annotation information, in order to reduce the difference between the keypoint location features extracted by the keypoint matching model and the initial keypoint annotation information. A third loss function is constructed using the first keypoint matching feature map and the second keypoint matching feature map, in order to reduce the keypoint matching difference between the original training image and the transformed image.

[0059] In a specific implementation, the above method further includes: constructing a first loss function and a second loss function based on a preset loss function; and constructing a third loss function based on a weighted bilateral contrast loss function. In this embodiment, the preset loss function used in constructing the first and second loss functions is specifically the Softmax (i.e., normalized exponential function) loss function. Alternatively, a cross-entropy function or other methods can be used. That is, this application embodiment constructs a first loss function and a second loss function based on the Softmax loss function between the first keypoint location feature map and the second keypoint location feature map and the keypoint annotation information, respectively; and constructs a third loss function based on a weighted bilateral contrast loss function between the first keypoint matching feature map and the second keypoint matching feature map.

[0060] The formula for the Softmax loss function L is as follows:

[0061]

[0062] Where C represents the number of channels, y true This indicates the actual location of the keypoint in the keypoint annotation information, y pred This represents the probability of predicting the location of a key point in the first key point location feature map and the second key point location feature map, where 'a' represents the channel number.

[0063] The formula for the weighted double-margin contrastive loss function (WDMC loss) is as follows:

[0064]

[0065] Where w1 and w2 represent coefficients used to balance the loss, y i,j This indicates that if the point is a key point, y i,j =1, otherwise 0, d i,j Let m1 and m2 represent the cosine similarity between the first keypoint location feature map i and the second keypoint location feature map j, and let N be the number of predicted keypoints.

[0066] Furthermore, the above method also includes: training a feature extraction network using a generated virtual dataset; using the trained feature extraction network to extract features from the original training image to obtain keypoint coordinate information of the original training image, and generating keypoint annotation information based on the keypoint coordinate information. In a specific implementation, this embodiment introduces the virtual dataset generation part of SuperPoint, uses this virtual dataset to train the feature extraction network, and uses the trained feature extraction network to extract features from the original training image to obtain keypoint coordinate information in the original training image, and generates keypoint annotation information based on the keypoint coordinate information. This part can be completed automatically by the program, without manual annotation, reducing manual annotation costs and lowering the probability of model errors due to labeling errors.

[0067] Step S14: Train the initial keypoint matching model using the first loss function, the second loss function, and the third loss function to obtain the trained keypoint matching model.

[0068] In this embodiment, the initial keypoint matching model is trained using a first loss function, a second loss function, and a third loss function to obtain a trained keypoint matching model. It should be noted that the training cutoff condition can be either stopping model training when the total loss function value is less than a preset threshold, or stopping model training after a predetermined number of training iterations.

[0069] As can be seen, this application acquires the original training image and performs image transformation on the original training image to obtain a transformed image; the original training image and the transformed image are input into an initial keypoint matching model, so that the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model respectively output a first keypoint position feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint position feature map and a second keypoint matching feature map corresponding to the transformed image; the first keypoint position feature map and the second keypoint position feature map are used with keypoint annotation information to construct a first loss function and a second loss function, respectively, and the first keypoint matching feature map and the second keypoint matching feature map are used to construct a third loss function; the initial keypoint matching model is trained using the first loss function, the second loss function and the third loss function to obtain a trained keypoint matching model. Therefore, this application requires image transformation processing of the original training image to obtain a transformed image, and inputs both the original training image and the transformed image into an initial keypoint matching model. The keypoint detection head and the self-attention matching detection head in the initial keypoint matching model output the first keypoint position feature map and the first keypoint matching feature map corresponding to the original training image, and output the second keypoint position feature map and the second keypoint matching feature map corresponding to the transformed image, respectively. In this way, by introducing the transformed image during the training stage, the model obtained after training can have stronger generalization ability, be more suitable for image transformation scenarios, and have better robustness. Furthermore, by introducing the self-attention detection head into the model, this application can pay attention to global features, thereby improving matching efficiency. Furthermore, a first loss function and a second loss function are constructed using the first and second keypoint location feature maps and keypoint annotation information, respectively, to reduce the difference between the keypoint location features extracted by the keypoint matching model and the initial keypoint annotation information. A third loss function is also constructed using the first and second keypoint matching feature maps to reduce the keypoint matching difference between the original training image and the transformed image. Finally, the first, second, and third loss functions are used together to train the initial keypoint matching model, resulting in the trained keypoint matching model. This application improves the robustness of the keypoint matching model to image scene transformations and further enhances the accuracy of subsequent keypoint matching.

[0070] See Figure 2 As shown, this application discloses a specific training method for a key point matching model. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0071] Specifically, it includes:

[0072] Step S21: Obtain the original training image and perform image transformation on the original training image to obtain the transformed image.

[0073] Step S22: Input the original training image and the transformed image into the backbone network of the initial keypoint matching model, so as to extract the first image features of the original training image and the second image features of the transformed image through the backbone network.

[0074] In this embodiment, the initial keypoint matching model specifically includes a backbone network, a keypoint detection head, and a self-attention matching detection head. The model structure diagram can be found in [reference needed]. Figure 3 As shown. Figure 3 As shown, the original training image and the transformed image are first input into the backbone network of the initial keypoint matching model to extract the first image features of the original training image and the second image features of the transformed image through the backbone network. The backbone network can be the backbone structure of ResNet-50 (Residual Network) or other mainstream detection networks. This embodiment does not limit the specific structure of the backbone network.

[0075] Step S23: Input the first image features into the key point detection head and the self-attention matching detection head in the initial key point matching model, respectively, so as to output the first key point position feature map through the key point detection head and the first key point matching feature map through the self-attention matching detection head.

[0076] In this embodiment, the first image features are further input to the key point detection head and the self-attention matching detection head, respectively, so as to output the first key point position feature map through the key point detection head and the first key point matching feature map through the self-attention matching detection head.

[0077] The structure of the key point detection head is as follows: Figure 4 As shown, the image features are processed by three 3×3 convolution kernels with a stride of 1 to obtain the corresponding key point location feature map.

[0078] The structure of the self-attention matching detection head is as follows: Figure 5As shown, it should be noted that the self-attention matching detection head processes the input feature map based on a self-attention mechanism to obtain an output feature map. The processing flow of the self-attention matching detection head is as follows: Based on the self-attention mechanism, the input feature map is transformed into a query vector, a key vector, and a value vector; the query vector and the key vector are weighted using a preset weight matrix to obtain a weighted feature vector; a preset activation function is used to calculate the weighted feature vector to obtain an image weighting matrix; the similarity score between the image weighting matrix and the value vector is calculated, and the similarity score is added to the input feature map to obtain the output feature map. Figure 5 As shown, the input feature map is transformed into a query vector (Q), a key vector (K), and a value vector (V) by three 1×1 convolutional kernels. Then, a pre-defined weight matrix constructed using the 1×1 convolutional kernels is used to weight the query vector (Q) and the key vector (K) to obtain a weighted feature vector. The softmax activation function is then used to calculate the image weighting matrix. Further, the similarity score between the image weighting matrix and the value vector is calculated, and this similarity score is added to the input feature map to obtain the output feature map. To prevent gradient loss during the calculation process, the original input feature map is used to sum the calculated similarity scores to obtain the output feature map. The input feature map corresponds to the first image feature map in this embodiment, and the output feature map corresponds to the first keypoint matching feature map in this embodiment. In addition, it should be noted that by introducing a cross-self-attention module into the self-attention matching detection head, the embodiments of this application can reduce the computational load of the original self-attention structure while also modeling the features of the entire image, thereby strengthening the association between feature points in the image. This allows the neural network to learn a constraint similar to a geometric relationship, so that the matching results can meet the continuity and consistency of space in some cases, which is convenient for subsequent matching of key point information.

[0079] Step S24: Input the second image features into the key point detection head and the self-attention matching detection head respectively, so as to output the second key point position feature map through the key point detection head and the second key point matching feature map through the self-attention matching detection head.

[0080] In this embodiment, the second image feature corresponds to Figure 5 The input feature map, the second keypoint matching feature map corresponds to Figure 5 The output feature map is described in detail in step S23, and will not be repeated here.

[0081] Step S25: Construct a first loss function and a second loss function using the first key point location feature map and the second key point location feature map, respectively, and construct a third loss function using the first key point matching feature map and the second key point matching feature map.

[0082] Step S26: Train the initial keypoint matching model using the first loss function, the second loss function, and the third loss function to obtain the trained keypoint matching model.

[0083] For more detailed processing procedures of steps S21, S25 and S26, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0084] As can be seen, the initial keypoint matching model in this application embodiment specifically includes a backbone network, a keypoint detection head, and a self-attention matching detection head. The backbone network is used to extract image features from the original training image and the transformed image. The keypoint detection head is used to extract keypoint location feature maps corresponding to the image features. The self-attention matching detection head is used to extract keypoint matching feature maps corresponding to the image features. Through the above scheme, this application is insensitive to changes in the external environment, such as changes in lighting, angle, environment, weather, etc., and has good robustness. It also has consistency in keypoint locations for repetitive scenes and can be used for mapping in SLAM (Simultaneous Localization and Mapping) and memory parking functions in autonomous driving. Furthermore, this application embodiment introduces a cross-self-attention module in the self-attention matching detection head to strengthen the association between feature points in the same image, allowing the neural network to learn a constraint similar to a geometric relationship, thereby enabling the matching results to satisfy spatial continuity and consistency in certain situations.

[0085] See Figure 6 As shown in the figure, this application discloses a key point matching method, which includes:

[0086] Step S31: Obtain the current frame image, and use the key point matching model trained by the aforementioned disclosed key point matching model training method to detect the current frame image to obtain the current key point position feature map and the current key point matching feature map.

[0087] In this embodiment, the current frame image is acquired, and the keypoint matching model trained using the aforementioned disclosed keypoint matching model training method is used to detect the current frame image to obtain the current keypoint location feature map and the current keypoint matching feature map. That is, this application only needs to process the current frame image during forward inference, resulting in low computational load and facilitating deployment of edge computing devices.

[0088] Step S32: Calculate the corresponding feature association matrix using the current keypoint matching feature map and the keypoint matching feature map of the previous frame image, and decode the feature association matrix to obtain the keypoint matching result between the current frame image and the previous frame image.

[0089] In this embodiment, it should be noted that the network model also stores the keypoint matching feature map of the previous frame image. Therefore, by calculating the feature association matrix between the keypoint matching feature map of the previous frame image and the current keypoint matching feature map, and finally decoding the feature association matrix, the keypoint matching result between the current frame image and the previous frame image can be obtained. Furthermore, during the model training phase, this application does not rely on continuous frame information. Instead, it uses CycleGAN technology and homography transformation matrix to perform image transformation on the original training image to simulate the situation between consecutive frames. This enables the detection of keypoint association information between consecutive frames without relying on continuous frame information. Figure 7 This is a schematic diagram of a specific matching result disclosed in this application, where the curve after the point represents the trajectory of the key point's associated position in the first 1-5 frames.

[0090] As can be seen, this application can obtain the current key point location feature map and the current key point matching feature map by detecting the current frame image using the aforementioned disclosed key point matching model. Furthermore, the network model also stores the key point matching feature map of the previous frame image. Therefore, by calculating the feature correlation matrix between the key point matching feature map of the previous frame image and the current key point matching feature map, and finally decoding the feature correlation matrix, the key point matching result between the current frame image and the previous frame image can be obtained.

[0091] See Figure 7 As shown in the embodiment of this application, a training apparatus for a keypoint matching model is disclosed, the method comprising:

[0092] Image transformation module 11 is used to acquire the original training image and perform image transformation on the original training image to obtain a transformed image;

[0093] Feature output module 12 is used to input the original training image and the transformed image into an initial keypoint matching model, so as to output a first keypoint position feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint position feature map and a second keypoint matching feature map corresponding to the transformed image, respectively, through the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model.

[0094] The loss construction module 13 is used to construct a first loss function and a second loss function by using the first key point location feature map and the second key point location feature map with the key point annotation information, respectively, and to construct a third loss function by using the first key point matching feature map and the second key point matching feature map;

[0095] The model training module 14 is used to train the initial keypoint matching model using the first loss function, the second loss function and the third loss function to obtain the trained keypoint matching model.

[0096] As can be seen, this application acquires the original training image and performs image transformation on the original training image to obtain a transformed image; the original training image and the transformed image are input into an initial keypoint matching model, so that the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model respectively output a first keypoint position feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint position feature map and a second keypoint matching feature map corresponding to the transformed image; the first keypoint position feature map and the second keypoint position feature map are used with keypoint annotation information to construct a first loss function and a second loss function, respectively, and the first keypoint matching feature map and the second keypoint matching feature map are used to construct a third loss function; the initial keypoint matching model is trained using the first loss function, the second loss function and the third loss function to obtain a trained keypoint matching model. Therefore, this application requires image transformation processing of the original training image to obtain a transformed image, and inputs both the original training image and the transformed image into an initial keypoint matching model. The keypoint detection head and the self-attention matching detection head in the initial keypoint matching model output the first keypoint position feature map and the first keypoint matching feature map corresponding to the original training image, and output the second keypoint position feature map and the second keypoint matching feature map corresponding to the transformed image, respectively. In this way, by introducing the transformed image during the training stage, the model obtained after training can have stronger generalization ability, be more suitable for image transformation scenarios, and have better robustness. Furthermore, by introducing the self-attention detection head into the model, this application can pay attention to global features, thereby improving matching efficiency. Furthermore, a first loss function and a second loss function are constructed using the first and second keypoint location feature maps and keypoint annotation information, respectively, to reduce the difference between the keypoint location features extracted by the keypoint matching model and the initial keypoint annotation information. A third loss function is also constructed using the first and second keypoint matching feature maps to reduce the keypoint matching difference between the original training image and the transformed image. Finally, the first, second, and third loss functions are used together to train the initial keypoint matching model, resulting in the trained keypoint matching model. This application improves the robustness of the keypoint matching model to image scene transformations and further enhances the accuracy of subsequent keypoint matching.

[0097] Figure 8This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the training method for the keypoint matching model or related steps in the keypoint matching method disclosed in any of the foregoing embodiments.

[0098] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0099] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0100] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0101] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including computer programs capable of performing the training method or keypoint matching method of the keypoint matching model executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0102] Furthermore, this application also discloses a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the training method or key point matching method steps of the key point matching model disclosed in any of the foregoing embodiments.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0104] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0105] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0106] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0107] The model training method, key point matching method, apparatus, device, and medium provided by this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A training method for a keypoint matching model, characterized in that, include: Obtain the original training image and perform image transformation on the original training image to obtain the transformed image; The original training image and the transformed image are input into the initial keypoint matching model, so that the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model can output the first keypoint position feature map and the first keypoint matching feature map corresponding to the original training image, and the second keypoint position feature map and the second keypoint matching feature map corresponding to the transformed image, respectively. A first loss function and a second loss function are constructed using the first key point location feature map and the second key point location feature map, respectively, and the third loss function is constructed using the first key point matching feature map and the second key point matching feature map. The initial keypoint matching model is trained using the first loss function, the second loss function, and the third loss function to obtain the trained keypoint matching model.

2. The training method for the keypoint matching model according to claim 1, characterized in that, The step of performing image transformation on the original training image to obtain the transformed image includes: The original training image is transformed using a preset image transformation model and a homography transformation matrix to obtain a transformed image.

3. The training method for the keypoint matching model according to claim 1, characterized in that, The initial keypoint matching model includes a backbone network, the keypoint detection head, and the self-attention matching detection head; Accordingly, the step of inputting the original training image and the transformed image into the initial keypoint matching model to output a first keypoint location feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint location feature map and a second keypoint matching feature map corresponding to the transformed image, includes: The original training image and the transformed image are input into the backbone network to extract the first image features of the original training image and the second image features of the transformed image through the backbone network. The first image features are input to the key point detection head and the self-attention matching detection head, respectively, so as to output the first key point position feature map through the key point detection head and the first key point matching feature map through the self-attention matching detection head; The second image features are input to the key point detection head and the self-attention matching detection head, respectively, so that the key point detection head outputs a second key point position feature map, and the self-attention matching detection head outputs a second key point matching feature map.

4. The training method for the keypoint matching model according to claim 3, characterized in that, The self-attention matching detection head processes the input feature map based on a self-attention mechanism to obtain an output feature map. The processing flow of the self-attention matching detection head is as follows: The input feature map is transformed into a query vector, a key vector, and a value vector based on a self-attention mechanism. The query vector and the key vector are weighted using a preset weight matrix to obtain a weighted feature vector, and the weighted feature vector is then calculated using a preset activation function to obtain an image weighting matrix. Calculate the similarity score between the image weighting matrix and the value vector, and add the similarity score to the input feature map to obtain the output feature map.

5. The training method for the keypoint matching model according to claim 1, characterized in that, Also includes: Construct the first loss function and the second loss function based on the preset loss function; The third loss function is constructed based on a weighted bilateral contrastive loss function.

6. The training method for the keypoint matching model according to any one of claims 1 to 5, characterized in that, Also includes: The feature extraction network is trained using the generated virtual dataset; The trained feature extraction network is used to extract features from the original training image to obtain the key point coordinate information of the original training image, and the key point annotation information is generated based on the key point coordinate information.

7. A key point matching method, characterized in that, include: The current frame image is acquired, and the key point matching model trained by the key point matching model training method as described in any one of claims 1 to 6 is used to detect the current frame image to obtain the current key point position feature map and the current key point matching feature map. The corresponding feature association matrix is ​​calculated using the current keypoint matching feature map and the keypoint matching feature map of the previous frame image, and the feature association matrix is ​​decoded to obtain the keypoint matching result between the current frame image and the previous frame image.

8. A training device for a keypoint matching model, characterized in that, include: The image transformation module is used to acquire the original training image and perform image transformation on the original training image to obtain the transformed image; The feature output module is used to input the original training image and the transformed image into an initial keypoint matching model, so as to output a first keypoint position feature map and a first keypoint matching feature map corresponding to the original training image, and a second keypoint position feature map and a second keypoint matching feature map corresponding to the transformed image, respectively, through the keypoint detection head and the self-attention matching detection head in the initial keypoint matching model. The loss construction module is used to construct a first loss function and a second loss function by using the first key point location feature map and the second key point location feature map with the key point annotation information, respectively, and to construct a third loss function by using the first key point matching feature map and the second key point matching feature map; The model training module is used to train the initial keypoint matching model using the first loss function, the second loss function, and the third loss function to obtain the trained keypoint matching model.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when executed by a processor, the computer program implements the steps of the method as described in any one of claims 1 to 7.

Images