Key point matching method and device, and electronic device

Abstract

This disclosure proposes a keypoint matching method, apparatus, and electronic device. The method includes: performing keypoint detection on a first image and a second image to be matched to obtain keypoints in the first image and the second image, wherein the first image and the second image have overlapping areas; for any keypoint in the first image, determining a first sub-image of a set size based on a first position information of the keypoint in the first image as the center; in the second image, obtaining second position information that matches the first position information, and determining a second sub-image of a set size based on the second position as the center; and matching the keypoints in the first sub-image and the keypoints in the second image, thereby achieving efficient and accurate keypoint matching between the first image and the second image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a key point matching method, apparatus and electronic device. Background Technology

[0002] Keypoint matching is a crucial problem in computer vision. The task of keypoint matching is to accurately find identical points in two locally overlapping images (which can be color or black and white) and match them. The matches should have high recognizability, such as corner points or edge points, while minimizing the matching of textureless, repetitive regions. Keypoint matching has wide applications in Simultaneous Localization and Mapping (SLAM) and moving object tracking; therefore, accurate keypoint matching is extremely important. Summary of the Invention

[0003] This disclosure aims to at least partially address one of the technical problems in the related art.

[0004] To address this, this disclosure proposes a keypoint matching method, apparatus, and electronic device. This method first performs keypoint detection on a first image and a second image to be matched. Then, based on each keypoint detected in the first image, a first sub-image of a predetermined size is determined. At the same position in the second image, a second sub-image of the same size as the first sub-image is determined. Keypoints in the first sub-image and keypoints in the second sub-image are matched. Thus, by acquiring the keypoints in the first and second sub-images, and using each keypoint in the first image as the center of the predetermined size in the first sub-image, keypoint matching is performed on the second sub-image at the same position in the second image. This method can efficiently and accurately achieve keypoint matching between the first and second images.

[0005] According to a first aspect of this disclosure, a keypoint matching method is provided, comprising: performing keypoint detection on a first image and a second image to be matched to obtain keypoints in the first image and keypoints in the second image, wherein the first image and the second image have overlapping regions; for any keypoint in the first image, determining a first sub-image of a set size corresponding to any keypoint, based on a first position information of the keypoint in the first image as the center; in the second image, obtaining second position information that matches the first position information, and determining keypoints in a second sub-image of a set size centered on the second position information among the keypoints in the second image; and matching the keypoints in the first sub-image and the keypoints in the second sub-image.

[0006] According to a second aspect of the present disclosure, a keypoint matching apparatus is provided, comprising: a detection module, configured to perform keypoint detection on a first image and a second image to be matched, to obtain keypoints in the first image and keypoints in the second image, wherein the first image and the second image have overlapping regions; a first determination module, configured to, for any keypoint in the first image, determine a first sub-image of a set size corresponding to any keypoint, centered on a first position information of the keypoint in the first image; a second determination module, configured to, in the second image, obtain second position information matching the first position information, and determine keypoints in a second sub-image of a set size centered on the second position information; and a matching module, configured to match the keypoints in the first sub-image and the keypoints in the second sub-image.

[0007] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the keypoint matching method proposed in the first aspect of this disclosure.

[0008] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the keypoint matching method proposed in the first aspect of this disclosure.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor of an electronic device, enables the electronic device to perform the keypoint matching method proposed in the first aspect embodiment.

[0010] The technical solution disclosed herein involves performing keypoint detection on a first image and a second image to be matched, thereby obtaining keypoints in the first image and the second image, wherein the first image and the second image have overlapping areas. For any keypoint in the first image, a first sub-image of a set size is determined based on the first position information of the keypoint in the first image as the center. In the second image, second position information matching the first position information is obtained, and keypoints in the second sub-image of a set size centered on the second position information are determined. The keypoints in the first sub-image and the keypoints in the second sub-image are matched. Thus, by obtaining the keypoints in the first sub-image and the second sub-image, and matching the first sub-image of a set size centered on each keypoint in the first image with the second sub-image at the same position in the second image, keypoint matching of the first image and the second sub-image can be achieved efficiently and accurately.

[0011] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a flowchart illustrating a key point matching method according to an exemplary embodiment;

[0014] Figure 2 This is a flowchart illustrating a key point matching method according to an exemplary embodiment;

[0015] Figure 3 This is a flowchart illustrating a key point matching method according to an exemplary embodiment;

[0016] Figure 4 This is a flowchart illustrating a key point matching method according to an exemplary embodiment;

[0017] Figure 5 This is a flowchart illustrating a key point matching method according to an exemplary embodiment;

[0018] Figure 6 This is a schematic diagram of a key point matching device according to an exemplary embodiment;

[0019] Figure 7 This is a block diagram illustrating a keypoint matching electronic device according to an exemplary embodiment. Detailed Implementation

[0020] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0021] It's important to understand that keypoint matching methods can be divided into two categories: deep learning methods and traditional methods. Deep learning-based keypoint matching methods first use deep neural networks to detect keypoints in an image and predict feature descriptors for each keypoint. Then, they use deep learning or mathematical similarity calculations to match keypoints between two images. Traditional keypoint matching methods typically involve two steps: first, using traditional methods such as scale-invariant feature transformation to find corners, edges, etc., in a given image; then, using traditional point matching algorithms to find the location of that point in another image.

[0022] However, deep learning methods often require quantization to be deployed on dedicated deep learning chips. The current mainstream quantization process is Post-Training Quantization (PTQ), which can lead to irreversible loss of model accuracy. Furthermore, most current deep learning methods rely on feature descriptors for keypoint matching. Feature descriptors are high-dimensional feature vectors that are extremely sensitive to accuracy; quantization results in significant accuracy loss, and the performance of quantized high-dimensional feature descriptors is not ideal, leading to a decrease in keypoint matching accuracy. Traditional keypoint detection algorithms require sufficiently distinguishable keypoint features, such as large gradients and color variations. Secondly, traditional matching methods require extensive fitting and optimization processes, which demand high CPU computing power. For tasks like SLAM, this is unsuitable for deployment on low-cost dedicated deep learning chips, increasing hardware costs.

[0023] To address the aforementioned issues, this disclosure proposes a key point matching method, apparatus, and electronic device.

[0024] The following detailed description, with reference to the accompanying drawings, describes a key point matching method, apparatus, and electronic device provided in this disclosure.

[0025] Figure 1 This is a flowchart illustrating a keypoint matching method according to an exemplary embodiment. It should be noted that this keypoint matching method can be applied to a keypoint matching device. The keypoint matching can be configured in an electronic device. This electronic device can be a mobile terminal, such as a mobile phone, tablet computer, personal digital assistant, or other hardware device with various operating systems.

[0026] like Figure 1 As shown, the keypoint matching method includes the following steps:

[0027] Step 101: Perform keypoint detection on the first and second images to be matched to obtain keypoints in the first and second images.

[0028] The first and second images have overlapping areas.

[0029] In this embodiment of the disclosure, a keypoint detection algorithm can be used to detect keypoints in the first image and the second image to be matched, thereby obtaining keypoints in the first image and the second image. The keypoint detection algorithm may include: SIFT algorithm, FAST algorithm, ORB algorithm, etc. Keypoints may include: image corner points, edge points, etc.

[0030] In practical applications, to facilitate hardware deployment while improving the accuracy of keypoint detection, as an example, a lightweight neural network model can be used for keypoint detection. In this case, the lightweight neural network model for keypoint detection has already learned the correspondence between the image and the keypoints.

[0031] Step 102: For any key point in the first image, based on the first position information of any key point in the first image, determine the first sub-image of a set size corresponding to any key point.

[0032] In this embodiment of the disclosure, for any key point in the first image, the position information of the key point in the first image is used as the first position information. With the first position information as the center, a first sub-image of a set size is determined in the first image. For example, if the first position information is (x,y), a first sub-image of a set size of 60*60 is determined in the first image with (x,y) as the center.

[0033] Step 103: In the second image, obtain the second position information that matches the first position information, and determine a second sub-image of a set size with the second position as the center.

[0034] In other words, in the second image, second position information that matches the first position information of any key point is obtained. For example, if the first position information is (x,y), the second position information in the second image is determined to be (x′,y′). Then, in the second image, a second sub-image of a set size is determined with the second position information as the center, wherein the first position information (x,y) in the first image matches the second position information (x′,y′) in the second image.

[0035] Step 104: Match the key points in the first sub-image and the key points in the second sub-image.

[0036] Furthermore, a keypoint matching algorithm can be used to match keypoints in the first sub-image and keypoints in the second sub-image. Keypoint matching algorithms may include optical flow matching algorithms, stereo matching algorithms, etc.

[0037] In practical applications, to facilitate hardware deployment, as an example, a lightweight, trained keypoint matching model can be used to match keypoints in the first sub-image and keypoints in the second sub-image.

[0038] In summary, by performing keypoint detection on the first and second images to be matched, keypoints in the first and second images are obtained, wherein there is an overlapping area between the first and second images. For any keypoint in the first image, a first sub-image of a set size is determined based on the first position information of the keypoint in the first image as the center. In the second image, second position information matching the first position information is obtained, and keypoints in the second sub-image of a set size centered on the second position information are determined. The keypoints in the first sub-image and the keypoints in the second sub-image are matched. Thus, by obtaining the keypoints in the first and second sub-images, keypoint matching of the first and second images can be achieved efficiently and accurately by using the first sub-image of a set size centered on each keypoint in the first image and the second sub-image of the same position in the second image as keypoints in the second image.

[0039] In practical applications, to facilitate hardware deployment while improving keypoint matching accuracy, in this embodiment, a trained optical flow-based keypoint matching model can be used to perform keypoint matching on the first and second sub-images. The trained optical flow-based keypoint matching model can be a lightweight neural network model. The following describes... Figure 2 Please provide a detailed explanation.

[0040] Figure 2 This is a flowchart illustrating a key point matching method according to an exemplary embodiment.

[0041] like Figure 2 As shown, the keypoint matching method may include the following steps:

[0042] Step 201: Perform keypoint detection on the first and second images to be matched to obtain keypoints in the first and second images.

[0043] The first and second images have overlapping areas.

[0044] Step 202: For any key point in the first image, determine the first sub-image of a set size corresponding to any key point, based on the first position information of the key point in the first image.

[0045] Step 203: In the second image, obtain the second position information that matches the first position information, and determine a second sub-image of a set size with the second position as the center.

[0046] Step 204: Input the first sub-image and the second sub-image into the trained optical flow-based keypoint matching model, so that the trained optical flow-based keypoint matching model matches the keypoints in the first sub-image and the keypoints in the second sub-image, and outputs the optical flow value of the keypoints in the second sub-image relative to the keypoints in the first sub-image.

[0047] In this embodiment, a trained optical flow-based keypoint matching model can be used to match keypoints in a first sub-image and a second sub-image, and can output the optical flow value of a keypoint in the second sub-image that matches any keypoint relative to any keypoint in the first sub-image. The trained optical flow-based keypoint matching model can be a lightweight neural network model. That is, the input and output data of the trained optical flow-based keypoint matching model are both quantized low-bit data. To improve computational efficiency on dedicated deep learning chips, the trained optical flow-based keypoint matching model does not include other complex network layers besides convolutional layers and batch normalization (BN) layers.

[0048] It should be noted that the initial optical flow-based keypoint matching model can be trained before using the trained optical flow-based keypoint matching model to match keypoints in the first sub-image and keypoints in the second sub-image.

[0049] Optionally, an initial optical flow-based keypoint matching model is obtained; a first training dataset is obtained; wherein the first training dataset includes: a first sample image and a second sample image, wherein the first sample image and the second sample image contain keypoints; the optical flow values ​​of the keypoints in the second sample image and the matching keypoints in the first sample image are labeled to obtain labeled optical flow values; the first training dataset is input into the initial optical flow-based keypoint matching model to obtain the predicted optical flow values ​​output by the initial optical flow-based keypoint matching model; a first loss function value is generated based on the difference between the predicted optical flow value and the labeled optical flow value; and the initial optical flow-based keypoint matching model is trained based on the first loss function value.

[0050] Furthermore, it should be noted that the complexity of optical flow-based keypoint matching models is relatively lower than that of deep learning models. Therefore, the model parameters in the trained optical flow-based keypoint matching model can be converted into a specified hardware format using a set conversion tool, thereby enabling the deployment of the trained keypoint matching model on hardware and reducing hardware deployment costs.

[0051] In summary, by inputting the first and second sub-images into a trained optical flow-based keypoint matching model, the model matches keypoints in the first and second sub-images and outputs the optical flow value of keypoints in the second sub-image relative to keypoints in the first sub-image. Thus, using an optical flow-based keypoint matching model can improve the accuracy of keypoint matching results. Furthermore, the lightweight optical flow-based keypoint matching model is easy to deploy on hardware, reducing deployment costs.

[0052] In practical applications, to facilitate hardware deployment while improving keypoint detection accuracy, as an example, a lightweight, trained keypoint detection model can be used to detect keypoints in both the first and second sub-images. The following section combines... Figure 3 Please provide a detailed explanation.

[0053] Figure 3 This is a flowchart illustrating a key point matching method according to an exemplary embodiment.

[0054] like Figure 3 As shown, the keypoint matching method may include the following steps:

[0055] Step 301: Input the first image and the second image into the trained keypoint detection model, so that the trained keypoint detection model can detect keypoints in the first image and keypoints in the second image.

[0056] The first and second images have overlapping areas.

[0057] In this embodiment, a trained keypoint detection model can be used to detect keypoints in the first image and the second image. Specifically, the first image and the second image can be input into the trained keypoint detection model, allowing the model to detect keypoints in both images. The keypoint detection model can be a lightweight neural network model. Both the input and output data of the keypoint detection model are quantized low-bit data. To improve computational efficiency on dedicated deep learning chips, this keypoint detection does not include complex network layers other than convolutional layers and batch normalization (BN) layers.

[0058] Step 302: Obtain key points in the first image and key points in the second image output by the trained key point detection model.

[0059] Thus, key points in the first image and key points in the second image can be obtained from the output of the trained keypoint detection model. The trained keypoint detection model has learned the correspondence between images and key points.

[0060] Step 303: For any key point in the first image, based on the first position information of any key point in the first image, determine the first sub-image of a set size corresponding to any key point.

[0061] Step 304: In the second image, obtain the second position information that matches the first position information, and determine a second sub-image of a set size with the second position as the center.

[0062] Step 305: Match any key point in the first sub-image with a key point in the second sub-image.

[0063] In summary, the first image and the second image are respectively input into the trained keypoint detection model, so that the trained keypoint detection model can detect keypoints in the first image and keypoints in the second image; the keypoints in the first image and keypoints in the second image output by the trained keypoint detection model are obtained. Therefore, using the keypoint detection model for keypoint detection can improve the accuracy of keypoint detection. Furthermore, the keypoint detection model is a lightweight model, which is easy to deploy on hardware and reduces the hardware deployment cost of the keypoint detection model.

[0064] It should be noted that before using the trained keypoint detection model to detect keypoints in the first and second images, the initial keypoint detection model can be trained. The following section will combine... Figure 4 Please provide a detailed explanation.

[0065] Figure 4 This is a flowchart illustrating a key point matching method according to an exemplary embodiment.

[0066] like Figure 4 As shown, the keypoint matching method may include the following steps:

[0067] Step 401: Obtain the third sample image of the set shape.

[0068] The third sample image carries the first labeled key points.

[0069] In this embodiment of the disclosure, the third sample image may be a triangular image, a quadrilateral image, or other polygonal images. Key points in the third sample image may be annotated to obtain the first annotated key points in the third sample image.

[0070] Step 402: Using the third sample image and the first labeled key points carried by the third sample image, the initial key point detection model is trained for the first time to obtain the key point detection model after the first training.

[0071] As an example, a third sample image is input into an initial keypoint detection model, and the first predicted keypoints output by the initial keypoint detection model are obtained; a second loss function value is generated based on the difference between the first predicted keypoints and the corresponding first labeled keypoints; the second loss function value is used to perform the first training on the initial keypoint detection model to obtain the keypoint detection model after the first training.

[0072] Step 403: Obtain the fourth sample image from the set image database.

[0073] Step 404: Input the fourth sample image into the key point detection model trained in the first step to obtain the target key points of the fourth sample image.

[0074] Step 405: In response to the user's action, update the target key points.

[0075] It should be understood that since the keypoint detection model trained in the first training is trained with a small number of third sample images, the keypoint detection accuracy of the keypoint detection model trained in the first training is poor. Therefore, when the fourth sample image is input into the keypoint detection model trained in the first training, the target keypoints output by the keypoint detection model trained in the first training may be incorrect. Therefore, in this embodiment of the disclosure, in response to the user operation, the target keypoints can be updated to obtain the updated target keypoints.

[0076] Step 406: Using the fourth sample image and the updated target key points, perform a second training on the key point detection model that has been trained in the first training to obtain the trained key point detection model.

[0077] Then, the fourth sample image is input into the key point detection model trained in the first stage, and the second predicted key point output by the key point detection model trained in the first stage is obtained; the difference between the second predicted key point and the corresponding updated target key point is used to generate a third loss function value; the third loss function value is used to train the key point detection model trained in the first stage to obtain the trained key point detection model.

[0078] Step 407: Input the first image and the second image into the trained keypoint detection model, so that the trained keypoint detection model can detect keypoints in the first image and keypoints in the second image.

[0079] Step 408: Obtain the key points in the first image and the key points in the second image output by the trained key point detection model.

[0080] Step 409: For any key point in the first image, determine the first sub-image of a set size corresponding to any key point, based on the first position information of the key point in the first image.

[0081] Step 410: In the second image, obtain the second position information that matches the first position information, and determine a second sub-image of a set size with the second position as the center.

[0082] Step 411: Match the key points in the first sub-image and the key points in the second sub-image.

[0083] In summary, by acquiring a third sample image of a defined shape; using the third sample image and the first labeled keypoints carried by the third sample image to perform a first training on an initial keypoint detection model, a first-trained keypoint detection model is obtained; a fourth sample image is acquired from a defined image database; the fourth sample image is input into the first-trained keypoint detection model to obtain the target keypoints of the fourth sample image; in response to user operation, the target keypoints are updated; using the fourth sample image and the updated target keypoints, the first-trained keypoint detection model is trained a second time, resulting in a trained keypoint detection model. Thus, by first training the keypoint detection model with a small number of third sample images, using the first-trained keypoint detection model to determine the target keypoints on the fourth sample, and then using the fourth sample and the updated target keypoints to perform a second training on the first-trained keypoint detection model, the training of the keypoint detection model is enhanced, and the accuracy of the keypoint detection model is improved.

[0084] In any embodiment of this disclosure, such as Figure 5 As shown, the keypoint matching method can also be implemented based on the following steps:

[0085] 1. The first stage is the keypoint detection network (keypoint detection model), which takes a complete image as input and outputs all keypoints in that image. To facilitate the explanation of the subsequent process, the diagram above shows two images to be matched as input, and the keypoint detection results for each image as output.

[0086] 2. The second stage is the optical flow detection network (optical flow-based key point matching model). For two images I_a and I_b to be matched, all key points in I_a are traversed. For one of the key points kp_a, its coordinates are set as (x,y).

[0087] 3. Extract a 60x60 image patch_a centered at (x,y) of image I_a, and extract another 60x60 image patch_b centered at (x′,y′) of image I_b. It should be noted that the position information of (x,y) in image I_a matches the position information of (x′,y′) in image I_b.

[0088] 4. Input the two patches into the quantization neural network for optical flow detection, calculate the optical flow from the center point of image patch_a to image patch_b, and thus complete the matching of key points from the first image to the second image;

[0089] 5. For all detected key points, repeat steps 3 and 4 above;

[0090] 6. Train the low-bit model, and convert the trained model parameters using a chip quantization conversion toolchain to obtain a format that can be deployed on the chip, thus completing the hardware deployment.

[0091] The keypoint matching method of this disclosure involves performing keypoint detection on a first image and a second image to be matched, thereby obtaining keypoints in the first image and the second image, wherein there is an overlapping area between the first image and the second image. For any keypoint in the first image, a first sub-image of a set size is determined based on the first position information of the keypoint in the first image as the center. In the second image, second position information matching the first position information is obtained, and keypoints in the second sub-image of a set size centered on the second position information are determined. The keypoints in the first sub-image and the keypoints in the second image are matched. Thus, when keypoints in the first sub-image and the second image are obtained, keypoint matching is performed on the first sub-image of a set size centered on each keypoint in the first image and the second sub-image of the same position in the second image, which can efficiently and accurately achieve keypoint matching between the first image and the second image.

[0092] To implement the above embodiments, this disclosure proposes a key point matching device.

[0093] Figure 6 This is a schematic diagram of the structure of a key point matching device according to an exemplary embodiment.

[0094] like Figure 6 As shown, the key point matching device 600 includes: a detection module 610, a first determination module 620, a second determination module 630, and a matching module 640.

[0095] The detection module 610 is used to perform key point detection on the first image and the second image to be matched, so as to obtain key points in the first image and key points in the second image; the first determination module 620 is used to determine a first sub-image of a set size corresponding to any key point in the first image, based on the first position information of any key point in the first image as the center; the second determination module 630 is used to obtain second position information matching the first position information in the second image, and determine a second sub-image of a set size based on the second position as the center; the matching module 640 is used to match the key points in the first sub-image and the key points in the second sub-image.

[0096] As one possible implementation of this disclosure, the matching module 640 is used to input the first sub-image and the second sub-image into a trained optical flow-based keypoint matching model, so that the trained optical flow-based keypoint matching model matches any keypoint in the first sub-image and any keypoint in the second sub-image, and outputs the optical flow value of the keypoint in the second sub-image that matches any of the keypoints with respect to any of the keypoints in the first sub-image.

[0097] As one possible implementation of this disclosure, the key point matching device 600 further includes: a first acquisition module, a second acquisition module, an annotation module, a first input module, a first generation module, and a first training module.

[0098] The system comprises the following modules: a first acquisition module for acquiring an initial optical flow-based keypoint matching model; a second acquisition module for acquiring a first training dataset, wherein the first training dataset includes a first sample image and a second sample image, both containing keypoints; an annotation module for annotating the optical flow values ​​of the keypoints in the second sample image and the matching keypoints in the first sample image to obtain annotated optical flow values; a first input module for inputting the first training dataset into the initial optical flow-based keypoint matching network to obtain the predicted optical flow values ​​output by the initial optical flow-based keypoint matching; a first generation module for generating a first loss function value based on the difference between the predicted optical flow value and the annotated optical flow value; and a first training module for training the initial optical flow-based keypoint matching based on the first loss function value.

[0099] As one possible implementation of this disclosure, the key point matching device 600 further includes: a first conversion module.

[0100] The first conversion module is used to convert the model parameters in the trained optical flow-based keypoint matching model into a specified hardware format so as to deploy the trained keypoint matching model on the specified hardware.

[0101] As one possible implementation of this disclosure, the detection module 610 is configured to: input the first image and the second image into a trained keypoint detection model, respectively, so that the keypoint detection model detects keypoints in the first image and keypoints in the second image; and acquire the keypoints in the first image and keypoints in the second image output by the trained keypoint detection model.

[0102] As one possible implementation of this disclosure, the key point matching device 600 further includes: a third acquisition module, a second training module, a fourth acquisition module, a second input module, an update module, and a third training module.

[0103] The system includes a third acquisition module for acquiring a third sample image of a predetermined shape, wherein the third sample image carries first labeled key points; a second training module for performing a first training on an initial key point detection model using the third sample image and the first labeled key points carried by the third sample image, to obtain a key point detection model after the first training; a fourth acquisition module for acquiring a fourth sample image from a predetermined image database; a second input module for inputting the fourth sample image into the key point detection model after the first training, to obtain the target key points of the fourth sample image; an update module for updating the target key points in response to user operation; and a third training module for performing a second training on the key point detection model after the first training using the fourth sample image and the updated target key points, to obtain a trained key point detection model.

[0104] As one possible implementation of this disclosure, the second training module is configured to: input the third sample image into the initial keypoint detection model and obtain the first predicted keypoints output by the initial keypoint detection model; generate a second loss function value based on the difference between the first predicted keypoints and the corresponding first labeled keypoints; and use the second loss function value to perform a first training on the initial keypoint detection model to obtain a keypoint detection model that has undergone the first training.

[0105] As one possible implementation of this disclosure, the third training module is configured to: input a fourth sample image into a keypoint detection model that has been trained in the first stage, and obtain a second predicted keypoint output by the keypoint detection model that has been trained in the first stage; generate a third loss function value based on the difference between the second predicted keypoint and the corresponding updated target keypoint; and use the third loss function value to perform a second training on the keypoint detection model that has been trained in the first stage, so as to obtain a trained keypoint detection model.

[0106] The keypoint matching apparatus of this disclosure performs keypoint detection on a first image and a second image to be matched, thereby obtaining keypoints in the first image and the second image, wherein there is an overlapping area between the first image and the second image. For any keypoint in the first image, a first sub-image of a set size is determined based on the first position information of the keypoint in the first image as the center. In the second image, second position information matching the first position information is obtained, and keypoints in the second sub-image of a set size centered on the second position information are determined. The keypoints in the first sub-image and the keypoints in the second image are matched. Thus, when keypoints in the first sub-image and the second image are obtained, keypoint matching is performed on the first sub-image of a set size centered on each keypoint in the first image and the second sub-image of the same position in the second image, thereby achieving keypoint matching of the first image and the second image efficiently and accurately.

[0107] To implement the above embodiments, this disclosure also proposes an electronic device, such as... Figure 7 As shown, Figure 7 This is a block diagram illustrating a keypoint matching electronic device according to an exemplary embodiment. Figure 7 As shown, the above-mentioned electronic device 700 may include:

[0108] The memory 710 and processor 720 are connected by a bus 730, which connects different components (including the memory 710 and the processor 720). The memory 710 stores a computer program, which, when executed by the processor 720, implements the key point matching method described in the embodiments of this disclosure.

[0109] Bus 730 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0110] Electronic device 700 typically includes a variety of computer-readable media. These media can be any available media that can be accessed by electronic device 700, including volatile and non-volatile media, removable and non-removable media.

[0111] The memory 710 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 740 and / or cache memory 750. The electronic device 700 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 760 can be used to read and write non-removable, non-volatile magnetic media (…). Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 730 via one or more data media interfaces. Memory 710 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0112] A program / utility 780 having a set (at least one) of program modules 770 may be stored in, for example, memory 710. Such program modules 770 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 770 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0113] Electronic device 700 can also communicate with one or more external devices 790 (e.g., keyboard, pointing device, display 791, etc.), and with one or more devices that enable a user to interact with the electronic device 700, and / or with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 792. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 793. Figure 7 As shown, network adapter 793 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although... Figure 7 As not shown in the diagram, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0114] The processor 720 executes various functional applications and data processing by running programs stored in the memory 710.

[0115] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the key point matching of the embodiments of this disclosure, and will not be repeated here.

[0116] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium.

[0117] Specifically, when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the keypoint matching method as described above.

[0118] To implement the above embodiments, this disclosure also provides a computer program product that, when executed by the processor of an electronic device, enables the electronic device to perform the key point matching method as described above.

[0119] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0120] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A key point matching method, characterized in that, include: Keypoint detection is performed on the first and second images to be matched to obtain keypoints in the first image and the second image, wherein there is an overlapping area between the first and second images; For any key point in the first image, a first sub-image of a set size is determined based on the first position information of any key point in the first image as the center; In the second image, second position information matching the first position information is obtained, and a second sub-image of a set size is determined with the second position as the center; Match the key points in the first sub-image with the key points in the second sub-image; The matching of key points in the first sub-image and key points in the second sub-image includes: The first sub-image and the second sub-image are input into a trained optical flow-based key point matching model, so that the optical flow-based key point matching model matches any key point in the first sub-image with a key point in the second sub-image, and outputs the optical flow value of the key point in the second sub-image that matches any key point with ... in the first sub-image. This also includes: obtaining an initial optical flow-based keypoint matching model; Obtain a first training dataset; wherein the first training dataset includes: a first sample image and a second sample image, wherein the first sample image and the second sample image contain key points; The optical flow values ​​of key points in the second sample image and matching key points in the first sample image are labeled to obtain labeled optical flow values; The first training dataset is input into the initial optical flow-based keypoint matching model to obtain the predicted optical flow value output by the initial optical flow-based keypoint matching model. A first loss function value is generated based on the difference between the predicted optical flow value and the labeled optical flow value; The initial optical flow-based keypoint matching model is trained based on the first loss function value.

2. The method according to claim 1, characterized in that, The method further includes: The model parameters in the trained optical flow-based keypoint matching model are converted into a specified hardware format to deploy the trained keypoint matching model on the specified hardware.

3. The method according to claim 1, characterized in that, The keypoint detection process for the first and second images to be matched, to obtain keypoints in the first and second images, includes: The first image and the second image are respectively input into a trained keypoint detection model so that the trained keypoint detection model can detect keypoints in the first image and keypoints in the second image; Obtain the key points in the first image and the key points in the second image output by the trained key point detection model.

4. The method according to claim 3, characterized in that, The method further includes: Obtain a third sample image of a predetermined shape; wherein the third sample image carries first labeled key points; Using a third sample image and the first labeled key points carried by the third sample image, the initial key point detection model is trained for the first time to obtain a key point detection model trained for the first time. Obtain the fourth sample image from the established image database; The fourth sample image is input into the key point detection model trained in the first stage to obtain the target key points of the fourth sample image; In response to user actions, the target key points are updated; Using the fourth sample image and the updated target key points, the key point detection model that has been trained in the first training is trained in the second training to obtain the trained key point detection model.

5. The method according to claim 4, characterized in that, The step of using a third sample image and the first labeled key points carried by the third sample image to perform a first training on the initial key point detection model to obtain a key point detection model after the first training includes: The third sample image is input into the initial keypoint detection model, and the first predicted keypoint is obtained from the output of the initial keypoint detection model. A second loss function value is generated based on the difference between the first predicted key point and the corresponding first labeled key point; The second loss function value is used to perform the first training on the initial keypoint detection model to obtain the keypoint detection model after the first training.

6. The method according to claim 4, characterized in that, The step of using the fourth sample image and the updated target key points to perform a second training on the key point detection model trained in the first training phase, in order to obtain a trained key point detection model, includes: The fourth sample image is input into the key point detection model trained in the first stage, and the second predicted key point is obtained from the output of the key point detection model trained in the first stage. A third loss function value is generated based on the difference between the second predicted key point and the corresponding updated target key point; The third loss function value is used to perform a second training on the keypoint detection model that has been trained in the first training, so as to obtain a trained keypoint detection model.

7. The method according to any one of claims 4-6, characterized in that, The method further includes: The model parameters in the trained keypoint detection model are converted into a specified hardware format to deploy the trained keypoint detection model on the specified hardware.

8. A key point matching device, characterized in that, include: The detection module is used to perform key point detection on the first image and the second image to be matched, so as to obtain the key points in the first image and the key points in the second image; The first determining module is used to determine a first sub-image of a set size corresponding to any one of the key points in the first image, based on the first position information of any one of the key points in the first image as the center. The second determining module is used to obtain second position information that matches the first position information in the second image, and determine a second sub-image of a set size with the second position as the center; A matching module is used to match key points in the first sub-image and key points in the second sub-image; The matching module is used to input the first sub-image and the second sub-image into a trained optical flow-based key point matching model, so that the optical flow-based key point matching model matches any key point in the first sub-image with a key point in the second sub-image, and outputs the optical flow value of the key point in the second sub-image that matches any key point with any key point with any key point with any key point with any key point in the first sub-image. This also includes: obtaining an initial optical flow-based keypoint matching model; Obtain a first training dataset; wherein the first training dataset includes: a first sample image and a second sample image, wherein the first sample image and the second sample image contain key points; The optical flow values ​​of key points in the second sample image and matching key points in the first sample image are labeled to obtain labeled optical flow values; The first training dataset is input into the initial optical flow-based keypoint matching model to obtain the predicted optical flow value output by the initial optical flow-based keypoint matching model. A first loss function value is generated based on the difference between the predicted optical flow value and the labeled optical flow value; The initial optical flow-based keypoint matching model is trained based on the first loss function value.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the keypoint matching method according to any one of claims 1-7.

10. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the keypoint matching method as described in any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor of an electronic device, enables the electronic device to perform the keypoint matching method as described in any one of claims 1-7.

Images