Object feature point matching method and device, training method and device
Through the combination of feature extraction model and feature point matching network, automated feature point matching is achieved, solving the problems of low accuracy of feature point matching and high cost of manual labeling in image registration, adapting to various changes, and improving image matching Accurate accuracy and efficiency.
Patent Information
- Application Number
- CN202210217497.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-07
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-07
AI Technical Summary
The prior art has low accuracy in image registration, especially when the image offset is large, and the template matching method based on shape features is prone to failure when the light changes and deformation is required, and a large amount of manual labeling is required, which is cost-effective. high.
By obtaining the current point image and template feature information of the target object, the feature extraction model is used to extract feature points in the region of interest, and the matching processing is performed through the feature point matching network, and feature point optimization is carried out in combination with data enhancement and linear attention mechanism. The loss function is optimized for network and realizes automated feature matching.
It improves the accuracy of feature point matching, adapts to long-term changes, viewing angle changes and lens distortion, reduces the error kick rate and missed detection rate of quality inspection, reduces the cost of manual labeling, and can deal with problems such as lighting changes and deformation.
Smart Images

Figure CN114663686B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and specifically to a method for matching object feature points, a feature point matching device, a training method and training device for a feature matching network, a medium computer device, and a computer program product. Background Art
[0002] Feature point matching is the basis of many applications in the field of computer vision, and plays an important role in image registration, image stitching, 3D reconstruction, and target recognition. When stitching and registering images, it is necessary to solve the geometric transformation model of the image based on the matched feature points to achieve image registration. The accuracy of feature point matching determines the accuracy of image stitching and registration.
[0003] Currently, in the image registration methods based on template matching, the image registration methods require manual selection of easily recognizable feature areas for different images, and the cost of annotation is relatively high and there is a high tendency of human intervention. At the same time, this type of method is prone to failure when the image offset is large. In another type of template matching method based on shape features, obvious gradient changes are required, but in actual scenes there will be a lot of data without obvious extraction and it is also prone to failure when deformed. Summary of the invention
[0004] Embodiments of the present application provide an object feature point matching method, a feature point matching device, a feature matching network training method and training device, a medium, a computer device, and a computer program product.
[0005] On the one hand, a method for matching object feature points is provided, comprising:
[0006] Acquire a current point image and template feature information of a target object, wherein the template feature information includes feature points of a region of interest (ROI) area image of the target object extracted in advance by a feature extraction model in a plurality of point images;
[0007] Performing image segmentation on the current point image to obtain a current ROI area image, and performing feature extraction processing on the current ROI area image using the feature extraction model to obtain target feature information;
[0008] The target feature information and the template feature information are matched using a feature point matching network so that the feature points of the target object match the template feature information.
[0009] On the other hand, a method for training a feature point matching network is provided, comprising:
[0010] Obtain historical point images and perform data augmentation processing to generate training image pairs;
[0011] Using a feature extraction model to extract features from the training image pairs to obtain training feature information;
[0012] Sending the feature point position coordinates of the training feature information to an encoder composed of a multi-layer perceptron for processing to obtain local features;
[0013] Iterate the local features multiple times using a linear attention mechanism, and process the results of the multiple iterations using an iterative algorithm to obtain matching feature points;
[0014] A loss function is established according to the matched feature points to optimize the encoder and the linear attention mechanism. When training conditions are met, the optimized encoder and linear attention mechanism are determined as the feature point matching network.
[0015] On the other hand, a feature point matching device is provided, comprising:
[0016] An acquisition unit, used for acquiring a current point image and template feature information of a target object, wherein the template feature information includes feature points of a region of interest (ROI) area image of the target object extracted in advance by a feature extraction model in a plurality of point images;
[0017] A feature extraction unit, used for performing image segmentation on the current point image to obtain a current ROI area image, and performing feature extraction processing on the current ROI area image using the feature extraction model to obtain target feature information;
[0018] The feature matching unit is used to perform matching processing on the target feature information and the template feature information by using a feature point matching network, so that the feature points of the target object match the template feature information.
[0019] On the other hand, a training device for a feature point matching network is provided, comprising:
[0020] A data enhancement unit, used to obtain historical point images and perform data enhancement processing to generate training image pairs;
[0021] A feature extraction unit, used to extract features from the training image pair using a feature extraction model to obtain training feature information;
[0022] A first training unit, configured to send the feature point position coordinates of the training feature information to an encoder composed of a multi-layer perceptron for processing to obtain local features;
[0023] A second training unit is used to iterate the local features multiple times using a linear attention mechanism, and process the results of the multiple iterations using an iterative algorithm to obtain matching feature points;
[0024] An optimization unit is used to establish a loss function according to the matched feature points to optimize the encoder and the linear attention mechanism, and when training conditions are met, the optimized encoder and linear attention mechanism are determined as the feature point matching network.
[0025] On the other hand, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the method of any of the above embodiments.
[0026] On the other hand, a computer device is provided. The computer device includes a processor and a memory. A computer program is stored in the memory. The processor executes the steps in the method of any of the above embodiments by calling the computer program stored in the memory.
[0027] On the other hand, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, the steps in the method of any of the above embodiments are implemented.
[0028] The present application obtains the current point image and template feature information of the target object, and the template feature information includes the feature points of the target object's region of interest ROI area image in multiple point images extracted in advance by a feature extraction model; the current point image is segmented to obtain the current ROI area image, and the feature extraction model is used to perform feature extraction processing on the current ROI area image to obtain the target feature information; the target feature information and the template feature information are matched using a feature point matching network, so that the feature points of the target object match the template feature information. The present application proposes a complete set of feature matching method processes, which automatically extracts the target object ROI area image through a model, and inputs the target object and template features after feature extraction into a trained feature matching network. The learned matching features can better adapt to long-term changes, perspective changes, lens distortion, etc. compared to traditional manually labeled features, so that they can be used in image registration in practical applications, such as estimating the position and posture of the target object in robot arm grasping, and more accurate calculations can be performed on sensitive areas or shielded areas when performing quality inspection on objects to reduce the false kick rate and missed detection rate of quality inspection.
[0029] At the same time, compared to the image registration method based on template extraction and matching, which requires manual selection of easily recognizable feature areas for different images, making the annotation cost high, this application does not require a large amount of manual annotation, and automatically extracts a rough ROI through the model for subsequent feature matching. Compared with the template matching based on shape features, which requires obvious gradient changes, there will be a lot of data without obvious extraction in the actual scene and it is easy to fail when deformed. This application does not require template matching of shape features, and can better solve problems such as illumination changes, offsets, large background proportions, and no obvious gradients.
[0030] This application obtains historical point images and performs data enhancement processing to generate training image pairs; uses a feature extraction model to extract features from the training image pairs to obtain training feature information; sends the feature point position coordinates of the training feature information to an encoder composed of a multi-layer sensor for processing to obtain local features; uses a linear attention mechanism to iterate the local features multiple times, and uses an iterative algorithm to process the results of multiple iterations to obtain matching feature points; establishes a loss function based on the matched feature points to optimize the encoder and the linear attention mechanism, and when the training conditions are met, the optimized encoder and linear attention mechanism are determined as feature point matching networks. During the entire training process, this application fixes the model of the feature point extraction part, and only optimizes and trains the model of the feature point matching part, and constructs a self-supervised training data set through data enhancement. After image segmentation, after extracting features from ROI using the Unet model, the problem of feature loss of small objects under deep learning downsampling can be better solved, and more and denser feature points can be extracted.
[0031] In addition, the feature matching method of the present application does not require a large amount of manual labeling and can adapt to changes in lighting and posture. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical methods in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0033] Figure 1 This is a flowchart of the image registration method based on template matching;
[0034] Figure 2 It is a flowchart of the template matching method based on shape features;
[0035] Figure 3 A schematic diagram of a flow chart of a feature point matching method provided in an embodiment of the present application;
[0036] Figure 4 An example diagram of a feature point matching method provided in an embodiment of the present application;
[0037] Figure 5 An example diagram of a feature point matching method provided in an embodiment of the present application;
[0038] Figure 6 Another schematic diagram of the process of the feature point matching method provided in the embodiment of the present application;
[0039] Figure 7 Another example diagram of the feature point matching method provided in an embodiment of the present application;
[0040] Figure 8 A flowchart of a feature point matching method provided in an embodiment of the present application;
[0041] Fig. 9 A schematic diagram of a flow chart of a method for training a feature matching network provided in an embodiment of the present application;
[0042] Fig.10 An example diagram of a method for training a feature matching network provided in an embodiment of the present application;
[0043] Fig.11 A system framework diagram of a feature matching network provided in an embodiment of the present application;
[0044] Fig.12 A schematic diagram of the structure of a feature point matching device provided in an embodiment of the present application;
[0045] Fig.13 A schematic diagram of the structure of a training device provided in an embodiment of the present application;
[0046] Fig.14 A schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical methods in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0048] First, some nouns or terms that appear in the description of the embodiments of the present application are explained as follows:
[0049] Blockchain system: It can be a distributed system formed by connecting clients and multiple nodes (any form of computing devices connected to the network, such as servers and user terminals) through network communication. The nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In a distributed system, any machine such as a server or terminal can join and become a node. The node includes the hardware layer, the middle layer, the operating system layer and the application layer.
[0050] ROI (Region of Interest): Region of interest. In machine vision and image processing, the area to be processed is outlined in the image in the form of a box, circle, ellipse, irregular polygon, etc., which is called the region of interest, ROI. In machine vision software such as Halcon, OpenCV, and Matlab, various operators and functions are often used to obtain the region of interest ROI and perform the next step of image processing.
[0051] POSE: refers to the position and posture changes of an object relative to the template.
[0052] SuperPoint: SuperPoint is an end-to-end feature point and descriptor extraction network. The network structure is an encoder-decoder structure similar to the semantic segmentation network. A complete picture is input, and the deep features of the image are extracted through a shared encoder. Then, the image is passed through two decoders, one for feature points and the other for descriptors, to output feature points and descriptors respectively. Unlike the completely manually designed algorithm that first detects feature points and then calculates descriptors, feature points and descriptors are generated in parallel.
[0053] The RANSAC (Random Sample Consensus) algorithm is an iterative algorithm that correctly estimates the parameters of a mathematical model from a set of data containing "outliers". "Outliers" generally refer to the noise in the data, such as mismatches in matching and outliers in the estimated curve, so RANSAC is also an "outlier" detection algorithm. The RANSAC algorithm is an uncertain algorithm that can only produce results under a certain probability, and this probability will increase with the number of iterations.
[0054] Currently, in the image registration method based on template matching, please refer to Figure 1, step 01 acquires the image, and in step 02, the external expansion template completes the extraction of the current feature area and completes the template matching. The center point of the external expansion template is used as the feature point, and the N groups of feature points are matched with reference to the Random Sample Consensus (RANSAC) algorithm. In step 03, when the number of matching areas is greater than or equal to 2, step 04 randomly selects 2 groups to calculate the transfer matrix M, and N-2 groups obtain the transformed points through the matrix, and calculate the error with the feature points on the standard map, and count the number of points within the error range; traverse all combinations to obtain the best transfer matrix M. When the number of matching areas is less than 2, the calibration failure information is output. In step 05, the scaling factor and the rotation angle are calculated based on the transfer matrix M to see if they are less than the predetermined threshold. If they are less than, the transfer matrix is output to determine that the calibration is successful (step 06), otherwise the calibration failure information is output (step 07). However, template matching based on shape features requires obvious gradient changes, and there will be a lot of data without obvious extraction in actual scenes and it is easy to fail when deformed.
[0055] For another type of template matching method based on shape features, see Figure 2 When implementing image shape feature extraction and generation, the specific steps of generating the corresponding template feature map and the to-be-matched feature map from the template map and the to-be-matched map are inconsistent. In particular, the template map will perform feature point sampling and mask filtering steps, while the to-be-matched map will perform a spread operation. However, the image registration method based on template extraction and matching requires manual selection of easily recognizable feature areas for different images, and has a high artificial tendency due to the high cost of relative annotation. At the same time, this type of method is prone to failure when the image offset is large.
[0056] The embodiment of the present application provides an object feature point matching method and a feature point matching device, medium and equipment. The image point under the current viewing angle is associated with the feature point image in the template library to obtain the posture change of the part at the current position relative to the part at the template position, and the calculation of the area where the object is located is completed by combining the traditional threshold segmentation, and after the area is extracted, the feature point extraction and matching are completed by combining the deep learning algorithm, and finally the relative posture change matrix calculation is completed by the RANSAC algorithm. Compared with traditional manual features, the matching features learned after deep learning can better adapt to long-term changes, changes in viewing angles, lens distortion and other situations, and after extraction, it can better solve the problem of feature loss of small objects under deep learning downsampling, and can extract more and denser feature points. The algorithm does not require annotation and can adapt to changes in lighting and posture. The embodiment of the present application can be applied to image registration, such as estimating the posture of the target object in robot arm grasping. When performing quality inspection on the object, more accurate calculations can be performed on sensitive areas or shielded areas, thereby improving the recognition accuracy of the target object to reduce the false kick rate and missed detection rate of quality inspection.
[0057] Specifically, the method of the embodiment of the present application can be executed by a computer device, wherein the computer device can be a terminal or a server. User terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, intelligent robots, robotic arms, etc.
[0058] In order to better understand the technical method provided in the embodiment of the present application, the following briefly introduces the application scenarios applicable to the technical method provided in the embodiment of the present application. It should be noted that the application scenarios introduced below are only used to illustrate the embodiment of the present application and are not limited. Take the feature point matching method executed by a computer device as an example, where the computer device can be a terminal or a server.
[0059] The embodiments of the present application can be implemented in combination with cloud technology or blockchain network technology. For example, the feature point matching method disclosed in the embodiments of the present application, wherein these data can be stored on the blockchain, for example: feature extraction model, multiple template point images, current point image, template feature information, current ROI area image, target feature information, trained feature point matching network, historical point image, training image pair, training feature information, encoder, local features, matched feature points and loss function, can all be stored on the blockchain.
[0060] In order to facilitate the storage and query of the feature extraction model, multiple template point images, the current point image, the template feature information, the current ROI area image, the target feature information, the trained feature point matching network, the historical point image, the training image pair, the training feature information, the encoder, the local features, the matching feature points and the loss function, optionally, the feature point matching method also includes: sending the feature extraction model, multiple template point images, the current point image, the template feature information, the current ROI area image, the target feature information, the trained feature point matching network, the historical point image, the training image pair, the training feature information, the encoder, the local features, the matching feature points and the loss function to the blockchain network, so that the nodes of the blockchain network fill the feature extraction model, multiple template point images, the current point image, the template feature information, the current ROI area image, the target feature information, the trained feature point matching network, the historical point image, the training image pair, the training feature information, the encoder, the local features, the matching feature points and the loss function into the new block, and when consensus is reached on the new block, the new block is appended to the end of the blockchain. The embodiments of the present application can store the feature extraction model, multiple template point images, the current point image, template feature information, the current ROI area image, the target feature information, the trained feature point matching network, the historical point images, the training image pairs, the training feature information, the encoder, the local features, the matched feature points and the loss function on the chain to achieve the backup of the records. When it is necessary to obtain the matched feature points, the corresponding feature point information can be directly and quickly obtained from the blockchain, thereby improving the processing efficiency of the feature point matching.
[0061] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.
[0062] Each embodiment of the present application provides a method for matching feature points of an object. The embodiments of the present application use a computer device as an example to illustrate the method for matching feature points.
[0063] See also Figure 3 , FIG. 1 is a flow chart of a feature point matching method provided in an embodiment of the present application, the method comprising:
[0064] Step 310: obtaining the current point image and template feature information of the target object, wherein the template feature information includes feature points of the region of interest (ROI) area image of the target object in multiple point images extracted in advance by a feature extraction model.
[0065] Specifically, the current point image and template feature information of the target object are obtained, wherein the current point image is a point image of the current position of the target object, and the current point image of the target object is collected according to a preset point collection method.
[0066] It can be understood that the point refers to the image captured by the camera at different positions of the object, please refer to Figure 4 , the figure shows an example of a target object. You can select 32 points on the long straight edge of the side of the middle frame, 12 points on the short straight edge, and 48 points on the arc edge, for a total of 92 points. The RT surface angle of the arc area can be divided into 4 orders, among which the lower arc BGRT surface is 4 orders, with 42 points in each order, for a total of 168 points; the upper arc CGRT surface is 4 orders, with 42 points in each order, for a total of 168 points. At the same time, there are 30 points at the junction of BGRT and the back BG surface. For the target object in the figure, a total of 458 points can be collected at different positions.
[0067] The template feature information is a pre-stored feature point image of a template corresponding to the target object, and the feature points of the template ROI region image of multiple template point images of the target object are obtained through a template feature extraction model.
[0068] Optionally, the template feature information obtains a region of interest (ROI) feature point map of the target object through a template feature extraction model, including:
[0069] Perform M point sampling on the target object to obtain M template point images of the target object, where M is a natural number;
[0070] Perform image cropping on the M template point images respectively to obtain M template ROI area images;
[0071] Enlarging the side lengths of the M template ROI area images to a preset value, and extracting feature points and descriptors of the enlarged M template ROI area images;
[0072] The feature points and descriptors are stored to obtain the template feature information.
[0073] In advance, M point samples are taken for the target object, where M is a natural number and can be greater than 1. M template point images are obtained. The M template point images are cropped respectively, such as by using the traditional threshold segmentation method to obtain M template ROI area images. The side lengths of the M template ROI area images are enlarged to a preset value for the long side, and then feature points and descriptors are extracted from the enlarged ROI image area, and the extracted feature points and descriptors are stored to obtain template feature information, which includes feature points and their descriptors. For example, algorithms such as Harris / FAST / SIFT / SURF can be used to determine feature points, and BRIEF can be used to describe the detected feature points to obtain a descriptor, which is a binary-coded descriptor.
[0074] The template extraction method can utilize the template feature extraction process based on local feature points.
[0075] In some examples, the template image may not be labeled with ROI, but a rough region of interest ROI may be extracted using ROI, such as using a traditional threshold segmentation algorithm to complete ROI extraction, which can reduce the impact of the background area and reduce the labeling cost. For example, the image can be grayed, and then the maximum between-class variance method (OTSU) can be used to divide the image into background and target ROI according to the gray threshold T.
[0076] In this way, feature extraction and storage of the ROI area where the object is located can reduce the influence of the background area to a certain extent and ensure the effectiveness of the information.
[0077] Step 320: performing image segmentation on the current point image to obtain the current ROI region image, and performing feature extraction processing on the current ROI region image using a feature extraction model to obtain target feature information.
[0078] Specifically, the current point image is segmented, for example, by using a traditional threshold segmentation method or a network segmentation method. After the image segmentation is completed, a corresponding ROI area image is obtained.
[0079] The feature extraction model is used to extract features from the current ROI area image to obtain target feature information. For a certain part, template sampling at different points will be performed in advance. For example, if there are 458 points, there will be 458 template images. For each template image, feature extraction is performed on the ROI image area. Feature extraction methods such as SuperPoint and DISK can be used to extract the features of the template to obtain template feature information.
[0080] Optionally, the current point image is segmented to obtain the current ROI area image, and the current ROI area image is subjected to feature extraction processing using a feature extraction model to obtain target feature information including:
[0081] Performing image segmentation on the current point image to obtain a current ROI region image, and performing similarity calculation between the current ROI region image and the template ROI region image to obtain a first similarity;
[0082] When the first similarity is lower than a preset threshold, a feature extraction model is used to perform feature extraction processing on the current ROI region image to obtain target feature information.
[0083] Specifically, after obtaining the current ROI region image, the current ROI region image and the template ROI region image may be similarly calculated to obtain a first similarity, so as to perform a preliminary consistency test on the two. After the first similarity is calculated, the first similarity is compared with a preset threshold, and when the first similarity is lower than the preset threshold, the feature extraction model is used to perform feature extraction processing on the current ROI region image to obtain target feature information.
[0084] Specifically, a preset threshold value of the first similarity is set, such as 80%. When the first similarity is lower than the preset threshold value, or the first similarity is lower than or equal to the preset threshold value, the feature extraction model is continued to be used to perform feature extraction processing on the current ROI region image. When the first similarity is higher than or equal to the preset threshold value, it indicates to a certain extent that the current ROI region image and the template ROI region image are highly consistent, and there is no need to enter the subsequent feature extraction and feature matching processing.
[0085] In this way, a preliminary detection is performed on the similarity or consistency between the current ROI area image and the template ROI area image before feature point matching, so that when a lower similarity is detected, further deep learning feature point matching is performed, which improves the efficiency of feature point matching to a certain extent and improves the computational efficiency of sensitive areas and shielded areas in applications such as robotic arm grasping and quality inspection modules.
[0086] Optionally, the similarity calculation method includes:
[0087] Get the template pixel coordinates of the template ROI area image;
[0088] Get the current pixel coordinates of the current ROI area image;
[0089] Calculate the first overlap ratio between the current pixel coordinates and the template pixel coordinates;
[0090] A first similarity is determined according to the first overlap ratio.
[0091] Specifically, obtain the template pixel coordinates of the template ROI area image and the current pixel coordinates of the current ROI area image. Then, calculate the first overlap rate between the current pixel coordinates and the template pixel coordinates, and perform coloring calculation on the two, that is, project the 2D point sets of the current pixel coordinates and the template pixel coordinates to the 2D plane to obtain the A point set corresponding to the current pixel coordinates and the B point set corresponding to the template pixel coordinates, then merge the A point set and the B point set to the same projection plane, color the points that exist in both at the same pixel position, and obtain the overlapping coloring result, thereby completing the overlap rate calculation. Please refer to Figure 5In the figure, there are point sets A and B, where the coloring value of point set A is 12, the coloring value of point set B is 9, and the value of the overlapping coloring result of A and B is 8. Then the overlapping rate of point set A and point set B is R = 8 / max(12,9) = 67%.
[0092] After the first overlap rate between the current pixel coordinates and the template pixel coordinates is calculated, the first similarity between the current ROI area image and the template ROI area image is determined according to the first overlap rate. The higher the first overlap rate, the higher the first similarity, or in other words, the higher the consistency between the current ROI area image and the template ROI area image.
[0093] Step 330: Use a feature point matching network to match the target feature information with the template feature information so that the feature points of the target object match the template feature information.
[0094] The target feature information and the template feature information are matched by the feature point matching network so that the feature points of the target object match the template feature information. In the trained feature point matching network, the feature points of the current ROI area image are extracted, and then the extracted feature points are matched.
[0095] The trained feature point matching network includes a feature matching algorithm based on a graph neural network. A feature matching algorithm based on a graph neural network, such as SuperGlue, treats the feature points in the image as nodes of the graph, aggregates feature information through an attention mechanism, and obtains a feature vector for matching. The matching problem is then treated as an optimal transportation problem and solved using an iterative algorithm such as the Sinkhorn algorithm. The training method of the feature point matching network will be described in detail below.
[0096] In this way, the present application obtains the current point image and template feature information of the target object, and the template feature information includes the feature points of the region of interest ROI area image of the target object in multiple point images extracted in advance by the feature extraction model; the current point image is segmented to obtain the current ROI area image, and the feature extraction model is used to perform feature extraction processing on the current ROI area image to obtain the target feature information; the target feature information and the template feature information are matched using the feature point matching network to match the feature points of the target object with the template feature information. The image under the current viewing angle and the data in the template library can be associated and calculated to obtain the pose change of the target object at the current position relative to the template position. In addition, the matching features learned by the trained feature matching network of the present application can better adapt to long-term changes, viewing angle changes, lens distortion and other situations than the traditional manually labeled features, so that in practical applications, it can assist in the calculation of sensitive areas and shielded areas in robot arm grasping and quality inspection modules to reduce the false kick rate and missed detection rate of quality inspection.
[0097] At the same time, compared to the image registration method based on template extraction and matching, which requires manual selection of easily recognizable feature areas for different images, making the annotation cost high, this application does not require a large amount of manual annotation, and automatically extracts a rough ROI through the model for subsequent feature matching. Compared with the template matching based on shape features, which requires obvious gradient changes, there will be a lot of data without obvious extraction in the actual scene and it is easy to fail when deformed. This application does not require template matching of shape features, and can better solve problems such as illumination changes, offsets, large background proportions, and no obvious gradients.
[0098] Optional, see Figure 6 , FIG. 1 is another flow chart of the feature point matching method provided in an embodiment of the present application, including:
[0099] Step 610: obtaining a current point image and template feature information of the target object, wherein the template feature information includes feature points of a template ROI region image of a plurality of template point images of the target object extracted in advance by a feature extraction model;
[0100] Step 620: performing image segmentation on the current point image to obtain a current ROI region image, and performing feature extraction processing on the current ROI region image using a feature extraction model to obtain target feature information;
[0101] Step 630: using a feature point matching network to match the target feature information with the template feature information, so that the feature points of the target object match the template feature information;
[0102] Step 640: according to the result of feature point matching, perform pose estimation on the current point image to obtain the pose of the target object;
[0103] Step 650: performing coordinate transformation on the target posture to obtain a calibrated target posture;
[0104] Specifically, after obtaining the matched feature points, the pose can be estimated based on the feature points to obtain the target pose. Since the feature point coordinates are the coordinates of the current ROI image area, after the scale change and the xy offset in the horizontal and vertical directions, the obtained POSE is not the original image. The calculated target pose needs to be inversely transformed to be projected into the original coordinate system.
[0105] Optionally, the step of performing coordinate transformation on the target posture to obtain a calibrated target posture includes:
[0106] Obtaining a first scale and a first offset corresponding to the template feature information;
[0107] Obtain the second scale and second offset corresponding to the current ROI area image;
[0108] A target pose is generated according to the first scale, the first offset, the second scale, and the second offset.
[0109] For example, the corresponding scale change and horizontal and vertical xy offset in the extracted template ROI area image are Extract the scale change and horizontal and vertical xy offset of the image in the current ROI area: The estimated pose, i.e., the change matrix, is M, which corresponds to the calibrated pose M origin have:
[0110]
[0111] See also Figure 7 ,The figure includes four small pictures a), b), c), and d), where the left part of each small picture is the visualized target object before calibration, and the right part is the target object after calibration. It can be seen that the calibrated image can basically overlap with the template image.
[0112] Step 660: Calculate the similarity between the current ROI region image after coordinate transformation and the template ROI region image to obtain a second similarity;
[0113] Step 670: Generate output information according to the second similarity.
[0114] Optionally, the step of calculating the similarity between the current ROI region image after coordinate transformation and the template ROI region image includes:
[0115] Obtain the template pixel coordinates of the template ROI area image and the transformed pixel coordinates of the current ROI area image after coordinate transformation;
[0116] Calculating a second overlap ratio between the transformed pixel coordinates and the template pixel coordinates;
[0117] A second similarity is determined according to the second overlap ratio.
[0118] Specifically, the template pixel coordinates of the template ROI region image and the transformed pixel coordinates of the current ROI region image after coordinate transformation are obtained. Then, the second overlap ratio between the transformed pixel coordinates and the template pixel coordinates is calculated, and the two can be colorized. The colorization calculation method is the same as above and will not be elaborated here.
[0119] After the second overlap rate between the transformed pixel coordinates and the template pixel coordinates is calculated, the second similarity between the current ROI area image after the coordinate transformation and the template ROI area image is determined according to the second overlap rate. The higher the second overlap rate, the higher the second similarity, or in other words, the higher the consistency between the current ROI area image after the coordinate transformation and the template ROI area image.
[0120] After the second similarity is calculated, calibration information is generated according to the second similarity, and the calibration result includes the target pose, calibration result and calibration information. For example, the calibration information includes: a 3x3 pose matrix, a Boolean flag used to determine whether the calibration is successful, and a string used to describe the reason for the failure, such as feature points cannot be extracted, the number of feature points extracted is small (less than 4), the number of feature point matches is small (less than 4 pairs), and the corrected second similarity score is low, such as less than 0.9.
[0121] See also Figure 8 , Figure is a flow chart of this embodiment, which is divided into two parts: the template feature extraction part that is completed and stored offline before actual application, and the online feature matching part. For template feature extraction, the template image is input in advance, and then the ROI area is extracted, the feature points and descriptors are extracted through the feature extraction model, and the template features are stored. Feature extraction and storage of the ROI area where the object is located ensures the validity of the information to a certain extent.
[0122] In the online feature matching part of the figure, the process can be mainly divided into consistency detection, feature extraction and matching, and pose estimation. The current image and the current point image are obtained, and the consistency detection of the current point image and the template is performed, that is, similarity calculation. Then the ROI area image is extracted from the current point image, and the feature points and descriptors are extracted through the feature extraction model, and the target feature information containing the feature points and descriptors, as well as the template feature information obtained in the template feature extraction part, are input into the trained feature matching network for feature matching processing, so that the feature points of the target object match the template feature information, thereby completing the feature matching. Further, the pose estimation is completed according to the matched feature points, and the second similarity between the current ROI area image and the template ROI area image after the coordinate transformation is calculated, and the second similarity is compared with the preset similarity threshold. When the similarity threshold is reached, it can be judged that the current ROI area image and the template ROI area image are matching objects, and the calibration is successful, and then the information is output, and the output information may include the target pose, calibration results and calibration information.
[0123] In this way, the entire prior registration process provided by this embodiment can perform online image registration on the target object. By extracting and matching features in the ROI area, the image under the current viewing angle and the data in the template library can be correlated and calculated to obtain the pose change of the target object at the current position relative to the template position. Compared with traditional manually labeled features, it can better adapt to long-term changes, viewing angle changes, lens distortion, etc., so that in practical applications, it can assist in robot arm grasping, and the calculation of sensitive areas and shielded areas in the quality inspection module to reduce the false kick rate and missed detection rate of quality inspection.
[0124] In the application of industrial quality inspection projects, the calibration success rate of this application can reach 94%, which can be well promoted to different projects to assist in quality inspection post-processing and fusion. In addition, this solution does not require a large amount of labeling, saves a lot of manpower, and has good reproducibility.
[0125] See also Fig. 9 , the present application also provides a method for training a feature point matching network, including:
[0126] Step 910: Acquire historical point images and perform data enhancement processing to generate training image pairs;
[0127] Step 920: extracting features from the training image pairs using a feature extraction model to obtain training feature information;
[0128] Step 930: sending the feature point position coordinates of the training feature information to an encoder composed of a multi-layer perceptron for processing to obtain local features;
[0129] Step 940: using a linear attention mechanism to iterate the local features multiple times, and using an iterative algorithm to process the results of the multiple iterations to obtain matching feature points;
[0130] Step 950: Establish a loss function based on the matched feature points to optimize the encoder and the linear attention mechanism. When the training conditions are met, the optimized encoder and the linear attention mechanism are determined as the feature point matching network.
[0131] Specifically, historical point images are obtained. The historical point images can be a large number of original point images of any object used for training. The historical point images are enhanced by performing data enhancement processing, such as random image cropping, pixel content transformation, random perturbation, etc., and training image pairs are generated with the historical point images.
[0132] Optionally, the step of obtaining historical point images and performing data enhancement processing to generate training image pairs includes:
[0133] Performing pixel content transformation on the historical point image to obtain a first image and a second image;
[0134] Randomly selecting a region of the first image to obtain a first sub-image;
[0135] Perform random perturbation processing on the coordinate pixels corresponding to the rectangular area selected from the random area to obtain the coordinates of the intersection point of the area;
[0136] Performing image affine transformation processing on the second image and the coordinates of the region intersection point to obtain a transformed image;
[0137] Selecting the transformed image according to the rectangular area to obtain a second sub-image;
[0138] A training image pair is determined based on the first sub-image and the second sub-image.
[0139] For details, please refer to Fig.10 , the historical point image is transformed in pixel content to obtain the first image and the second image, wherein the pixel content transformation includes brightness, contrast and other transformations. Further, a random region is selected from the first image to obtain a first sub-image. At the same time, the coordinate pixels corresponding to the rectangular region selected from the random region are Perform random perturbation within a certain range to obtain new regional intersection coordinates The corresponding transformation matrix is calculated by the four vertices of the rectangular area, and the image affine transformation warp processing is performed on the second image and the coordinates of the intersection of the area to obtain a transformed image, and the transformed image is selected according to the rectangular area to obtain a second sub-image. The first sub-image and the second sub-image are then determined as a training image pair.
[0140] Thus, compared with conventional image detection and segmentation which require a large amount of data annotation, this embodiment completes the construction of image pairs through relevant data enhancement, and performs relevant transformations to obtain image pairs and transformation matrices that are closer to real data. At the same time, after the data enhancement, the obtained image pairs are more robust when the illumination changes and pose changes.
[0141] Furthermore, after the construction of the training data set of the training image pair is completed, the image information in the training image pair, as well as the pose change value of the first image and the second image, etc. can be obtained according to the corresponding data enhancement processing.
[0142] The feature extraction model is used to extract features from the training image pairs to obtain training feature information. The parameters of the feature extraction model are fixed parameters and do not participate in network optimization training. The feature extraction model can be composed of a well-known image semantic segmentation Unet network. The feature extraction model outputs multi-dimensional features such as 128-dimensional features with the same size as the original image, and a feature point score map. The score map is used to describe whether the feature point is a real and valid feature point. By comparing the score map with a preset threshold, a feature point position set, i.e., training feature information, is obtained.
[0143] After extracting the feature information, the feature point position coordinates of the training feature information are sent to the encoder composed of a multi-layer perceptron for processing to obtain local features. The local features are iterated multiple times using the linear attention mechanism, and the results of multiple iterations are processed using the Sinkhorn algorithm to obtain matching feature points.
[0144] See also Fig.11 The figure includes a feature extraction model in the form of Unet and a feature matching network of SuperGlue, which includes a keypoints encoder module composed of a multi-layer perceptron and a linear self-cross attention mechanism. The feature extraction model outputs the coordinates of the feature point position, including the feature point position and confidence, and sends them to the encoder composed of a multi-layer perceptron for processing, and is fused with the descriptor of the feature point to complete the construction of the matching feature information. The fused local features (0) X i =d i +MLP enc (p i ), where d i is the descriptor, p i is the feature point position coordinate, MLP enc (p i ) is the keypoints encoder module.
[0145] Furthermore, the linear self-cross attention mechanism is used to iterate the local features multiple times, and the iterative algorithm is used to process the results of the multiple iterations to obtain the matching feature points.
[0146] Specifically, Linear Self-Cross Attention iterates to obtain two feature description vectors for matching:
[0147] f i A =[delf(f1,f2),cross(f1,f2)]×N
[0148] f i B =[self(f1,f2),cross(f2,f1)]×N;
[0149] Wherein, f1 is the first sub-image in the training image pair, f2 is the second sub-image in the training image pair, and N is the number of iterations, which is a natural number.
[0150] After obtaining the feature description vectors used for matching, the matching score matrix S can be obtained by performing inner product calculations between the two vectors. i,j :
[0151] S i,j = <f i A ,f i B >.
[0152] The higher the corresponding score, the greater the probability that the two feature points match. Finally, the Sinkhorn Algorithm is used to solve the matching feature points that meet the optimal matching result.
[0153] For the above feature matching process, a loss function is established using a training data set composed of a large number of training images to optimize the encoder and the linear attention mechanism. When the training conditions are met, the optimized encoder and the linear attention mechanism are determined as the feature point matching network. Among them, the loss function can use the loss function in the well-known SuperGlue algorithm, which will not be elaborated here.
[0154] In this way, the present application obtains historical point images and performs data enhancement processing to generate training image pairs; uses a feature extraction model to extract features from the training image pairs to obtain training feature information; sends the feature point position coordinates of the training feature information to an encoder composed of a multi-layer sensor for processing to obtain local features; uses a linear attention mechanism to iterate the local features multiple times, and uses an iterative algorithm to process the results of multiple iterations to obtain matching feature points; establishes a loss function based on the matched feature points to optimize the encoder and the linear attention mechanism, and when the training conditions are met, the optimized encoder and the linear attention mechanism are determined as feature point matching networks. During the entire training process, the present application fixes the model of the feature point extraction part, and only optimizes and trains the model of the feature point matching part, and constructs a self-supervised training data set through data enhancement. After image segmentation, after extracting features from the ROI using the Unet model, the problem of feature loss of small objects under deep learning downsampling can be better solved, and more and denser feature points can be extracted. In addition, the feature matching method of the present application does not require a large amount of manual annotation, and can adapt to changes in illumination and posture.
[0155] All of the above technical methods can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.
[0156] In order to better implement the feature point matching method of the embodiment of the present application, the embodiment of the present application also provides a feature point matching device. Fig.12 , FIG. 1 is a schematic diagram of the structure of a feature point matching device provided in an embodiment of the present application. The feature point matching device 1000 may include:
[0157] An acquisition unit 1100 is used to acquire a current point image and template feature information of a target object, wherein the template feature information includes feature points of a region of interest (ROI) area image of the target object extracted in advance by a feature extraction model from a plurality of point images;
[0158] The feature extraction unit 1200 is used to perform image segmentation on the current point image to obtain the current ROI area image, and perform feature extraction processing on the current ROI area image using a feature extraction model to obtain target feature information;
[0159] The feature matching unit 1300 is used to perform matching processing on the target feature information and the template feature information by using a feature point matching network, so that the feature points of the target object match the template feature information.
[0160] Optionally, the feature extraction unit 1200 can also be used to perform image segmentation on the current point image to obtain the current ROI area image, and calculate the similarity between the current ROI area image and the template ROI area image to obtain a first similarity; when the first similarity is lower than a preset threshold, the feature extraction model is used to perform feature extraction processing on the current ROI area image to obtain target feature information.
[0161] Optionally, the feature point matching device 1000 further includes:
[0162] The pose estimation unit 1400 is used to estimate the pose of the current point image according to the result of feature point matching to obtain the pose of the target object;
[0163] A coordinate transformation unit 1500 is used to transform the position and posture to obtain a calibrated target position and posture;
[0164] A calculation unit 1600 is used to calculate the similarity between the current ROI area image and the template ROI area image according to the calibrated target posture to obtain a second similarity;
[0165] The generating unit 1700 is configured to generate output information according to the second similarity.
[0166] Optionally, the coordinate transformation unit 1500 can also be used to obtain the first scale and first offset corresponding to the template feature information; obtain the second scale and second offset corresponding to the current ROI area image; and perform coordinate transformation on the posture according to the first scale, the first offset, the second scale and the second offset to obtain the calibrated target posture.
[0167] Optionally, the calculation unit 1600 can also be used to obtain the template pixel coordinates of the template ROI area image; obtain the current pixel coordinates of the current ROI area image, or the transformed pixel coordinates of the current ROI area image after coordinate transformation; calculate the first overlap rate between the current pixel coordinates and the template pixel coordinates, or the second overlap rate between the transformed pixel coordinates and the template pixel coordinates; determine the first similarity based on the first overlap rate, or determine the second similarity based on the second overlap rate.
[0168] Optionally, the acquisition unit 1100 can also be used to sample M points of the target object to obtain M template point images of the target object, where M is a natural number; perform image cropping on the M template point images respectively to obtain M template ROI area images; enlarge the side lengths of the M template ROI area images to a preset value, and extract feature points and feature descriptors of the enlarged M template ROI area images; store the feature points and feature descriptors to obtain template feature information.
[0169] It should be noted that the functions of each module in the feature point matching device 1000 in the embodiment of the present application can correspond to the specific implementation methods of any embodiment in the above-mentioned method embodiments, and will not be repeated here.
[0170] Each unit in the feature point matching device 1000 can be implemented in whole or in part by software, hardware, or a combination thereof. Each unit can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each unit.
[0171] The feature point matching device 1000 can be integrated in a terminal or server with a storage device and a processor installed and having computing power, or the feature point matching device 1000 is the terminal or server. The terminal can be a smart phone, a tablet computer, a laptop computer, a smart TV, a smart speaker, a wearable smart device, a personal computer (PC), etc. The terminal can also include a client, which can be a video client, a browser client, or an instant messaging client. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0172] In order to better implement the method for training a feature point matching network in the embodiment of the present application, the embodiment of the present application also provides a training device 2000 for a feature matching network. Fig.13 , FIG. 1 is a schematic diagram of the structure of a training device 2000 provided in an embodiment of the present application. The training device 2000 may include:
[0173] The data enhancement unit 2100 is used to obtain historical point images and perform data enhancement processing to generate training image pairs;
[0174] A feature extraction unit 2200 is used to extract features from the training image pairs using a feature extraction model to obtain training feature information;
[0175] The first training unit 2300 is used to send the feature point position coordinates of the training feature information to an encoder composed of a multi-layer perceptron for processing to obtain local features;
[0176] The second training unit 2400 is used to perform multiple iterations on the local features using a linear attention mechanism, and process the results of the multiple iterations using an iterative algorithm to obtain matching feature points;
[0177] The optimization unit 2500 is used to establish a loss function according to the matched feature points to optimize the encoder and the linear attention mechanism. When the training conditions are met, the optimized encoder and the linear attention mechanism are determined as the feature point matching network.
[0178] Optionally, the data enhancement unit 2100 can also be used to perform pixel content transformation on the historical point image to obtain a first image and a second image; select the first image according to a preset border to obtain a first sub-image; perform random perturbation processing on the coordinate pixels of the preset border to obtain a random border; perform image affine transformation processing on the second image and the random border to obtain a transformed image; select the transformed image according to the preset border to obtain a second sub-image; and determine a training image pair based on the first sub-image and the second sub-image.
[0179] It should be noted that the functions of each module in the training device 2000 in the embodiment of the present application can correspond to the specific implementation methods of any embodiment in the above-mentioned method embodiments, and will not be repeated here.
[0180] Each unit in the training device 2000 can be implemented in whole or in part by software, hardware, or a combination thereof. Each unit can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each unit.
[0181] The training device 2000 can be integrated in a terminal or server with a storage device and a processor installed and having computing power, or the feature point matching device 1000 is the terminal or server. The terminal can be a smart phone, a tablet computer, a laptop computer, a smart TV, a smart speaker, a wearable smart device, a personal computer (PC), etc. The terminal can also include a client, which can be a video client, a browser client, or an instant messaging client. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0182] Fig.14 A schematic structural diagram of a computer device 3000 provided in an embodiment of the present application is shown in the figure. The computer device 3000 may include: a communication interface 3001, a memory 3002, a processor 3003 and a communication bus 3004. The communication interface 3001, the memory 3002, and the processor 3003 communicate with each other through the communication bus 3004. The communication interface 3001 is used for the computer device 3000 to communicate data with an external device. The memory 3002 can be used to store software programs and modules, and the processor 3003 runs the software programs and modules stored in the memory 3002, such as the software programs of the corresponding operations in the aforementioned method embodiments.
[0183] Optionally, the processor 3003 may call the software program and module stored in the memory 3002 to perform the following operations:
[0184] Acquire the current point image and template feature information of the target object, wherein the template feature information includes feature points of a region of interest (ROI) area image of the target object extracted in advance by a feature extraction model in a plurality of point images;
[0185] Perform image segmentation on the current point image to obtain the current ROI area image, and use the feature extraction model to perform feature extraction processing on the current ROI area image to obtain target feature information;
[0186] The target feature information and the template feature information are matched using a feature point matching network so that the feature points of the target object match the template feature information.
[0187] Optionally, the processor 3003 may also call software programs and modules stored in the memory 3002 to perform the following operations:
[0188] Obtain historical point images and perform data augmentation processing to generate training image pairs;
[0189] Using the feature extraction model to extract features from the training image pairs, obtaining training feature information;
[0190] The feature point position coordinates of the training feature information are sent to an encoder composed of a multi-layer perceptron for processing to obtain local features;
[0191] The local features are iterated multiple times using the linear attention mechanism, and the results of multiple iterations are processed using the iterative algorithm to obtain matching feature points;
[0192] A loss function is established based on the matched feature points to optimize the encoder and the linear attention mechanism. When the training conditions are met, the optimized encoder and the linear attention mechanism are determined as the feature point matching network.
[0193] Optionally, the computer device 3000 may be integrated in a terminal or server having a storage device and a processor installed therein and having computing capabilities, or the computer device 3000 is the terminal or server. The terminal may be a smart phone, a tablet computer, a laptop computer, a smart TV, a smart speaker, a wearable smart device, a personal computer, or other devices. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0194] The present application also provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to a computer device, and the computer program enables the computer device to execute the corresponding process in the feature point matching method or the feature matching network training method in the embodiment of the present application, which will not be described in detail for the sake of brevity.
[0195] The present application also provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the object feature point matching method in the embodiment of the present application, or the corresponding process in the feature matching network training method, which will not be described here for the sake of brevity.
[0196] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical method of the present application, or the part that contributes to the prior art or the part of the technical method, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.
[0197] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for matching object feature points, characterized in that: include: Acquire a current point image and template feature information of a target object, wherein the template feature information includes feature points of a region of interest (ROI) area image of the target object extracted in advance by a feature extraction model in a plurality of point images; Performing image segmentation on the current point image to obtain a current ROI area image, and performing feature extraction processing on the current ROI area image using the feature extraction model to obtain target feature information; Using a feature point matching network to match the target feature information with the template feature information, so that the feature points of the target object match the template feature information; The performing image segmentation on the current point image to obtain the current ROI area image, and performing feature extraction processing on the current ROI area image using the feature extraction model to obtain target feature information includes: Performing image segmentation on the current point image to obtain the current ROI area image, and performing similarity calculation between the current ROI area image and the template ROI area image to obtain a first similarity; When the first similarity is lower than a preset threshold, the feature extraction model is used to perform feature extraction processing on the current ROI area image to obtain the target feature information.
2. The method according to claim 1, characterized in that The performing image segmentation on the current point image to obtain the current ROI area image, and performing similarity calculation on the current ROI area image and the template ROI area image to obtain a first similarity includes: Acquire the template pixel coordinates of the template ROI area image and the current pixel coordinates of the current ROI area image; Calculating a first overlap ratio between the current pixel coordinates and the template pixel coordinates; The first similarity is determined according to the first overlap ratio.
3. The method according to claim 1, characterized in that After the method uses a feature point matching network to match the target feature information with the template feature information so that the feature points of the target object match the template feature information, it also includes: According to the result of the feature point matching, the pose of the current point image is estimated to obtain the pose of the target object; Performing coordinate transformation on the posture to obtain a calibrated target posture; Performing the similarity calculation on the current ROI area image and the template ROI area image according to the calibrated target posture to obtain a second similarity; Output information is generated according to the second similarity.
4. The method according to claim 3, characterized in that The performing the similarity calculation on the current ROI area image and the template ROI area image according to the calibrated target posture to obtain the second similarity comprises: Acquire the template pixel coordinates of the template ROI area image and the transformed pixel coordinates of the current ROI area image after the coordinate transformation; Calculating a second overlap ratio between the transformed pixel coordinates and the template pixel coordinates; The second similarity is determined according to the second overlap ratio.
5. The method according to claim 3, characterized in that: The coordinate transformation of the posture to obtain a calibrated target posture includes: Obtaining a first scale and a first offset corresponding to the template feature information; Obtain the second scale and second offset corresponding to the current ROI area image; The pose is coordinate-transformed according to the first scale, the first offset, the second scale, and the second offset to obtain the calibrated target pose.
6. The method according to claim 1, characterized in that The template feature information includes feature points of a region of interest (ROI) area image of a target object extracted in advance by a feature extraction model in a plurality of point images, including: Performing M point sampling on the target object to obtain M template point images of the target object, where M is a natural number; Performing image cropping on the M template point images respectively to obtain M template ROI area images; Enlarging the side lengths of the M template ROI area images to a preset value, and extracting feature points and feature descriptors of the enlarged M template ROI area images; The feature points and feature descriptors are stored to obtain the template feature information.
7. A method for training a feature point matching network, characterized in that: include: Obtain historical point images and perform data augmentation processing to generate training image pairs; Using a feature extraction model to extract features from the training image pairs to obtain training feature information; Sending the feature point position coordinates of the training feature information to an encoder composed of a multi-layer perceptron for processing to obtain local features; Iterate the local features multiple times using a linear attention mechanism, and process the results of the multiple iterations using an iterative algorithm to obtain matching feature points; Establishing a loss function according to the matched feature points to optimize the encoder and the linear attention mechanism, and determining the optimized encoder and linear attention mechanism as the feature point matching network when training conditions are met; The acquiring of historical point images and performing data enhancement processing to generate training image pairs includes: Performing pixel content transformation on the historical point image to obtain a first image and a second image; Selecting the first image according to a preset frame to obtain a first sub-image; Performing random perturbation processing on the coordinate pixels of the preset border to obtain a random border; Performing image affine transformation processing on the second image and the random border to obtain a transformed image; Selecting the transformed image according to the preset border to obtain a second sub-image; The training image pair is determined according to the first sub-image and the second sub-image.
8. A feature point matching device, characterized in that: include: An acquisition unit, used for acquiring a current point image and template feature information of a target object, wherein the template feature information includes feature points of a region of interest (ROI) area image of the target object extracted in advance by a feature extraction model in a plurality of point images; A feature extraction unit, used for performing image segmentation on the current point image to obtain a current ROI area image, and performing feature extraction processing on the current ROI area image using the feature extraction model to obtain target feature information; A feature matching unit, configured to perform matching processing on the target feature information and the template feature information using a feature point matching network, so that the feature points of the target object match the template feature information; The feature extraction unit is also used to perform image segmentation on the current point image to obtain the current ROI area image, and calculate the similarity between the current ROI area image and the template ROI area image to obtain a first similarity; when the first similarity is lower than a preset threshold, the feature extraction model is used to perform feature extraction processing on the current ROI area image to obtain the target feature information.
9. A training device for a feature point matching network, characterized in that: include: A data enhancement unit, used to obtain historical point images and perform data enhancement processing to generate training image pairs; A feature extraction unit, used to extract features from the training image pair using a feature extraction model to obtain training feature information; A first training unit, configured to send the feature point position coordinates of the training feature information to an encoder composed of a multi-layer perceptron for processing to obtain local features; A second training unit is used to iterate the local features multiple times using a linear attention mechanism, and process the results of the multiple iterations using an iterative algorithm to obtain matching feature points; An optimization unit, configured to establish a loss function according to the matched feature points to optimize the encoder and the linear attention mechanism, and determine the optimized encoder and linear attention mechanism as the feature point matching network when a training condition is met; The data enhancement unit is further used to perform pixel content transformation on the historical point image to obtain a first image and a second image; select the first image according to a preset border to obtain a first sub-image; Performing random perturbation processing on the coordinate pixels of the preset border to obtain a random border; Performing image affine transformation processing on the second image and the random border to obtain a transformed image; The transformed image is selected according to the preset border to obtain a second sub-image; and the training image pair is determined according to the first sub-image and the second sub-image.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the method according to any one of claims 1 to 7.
11. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the steps in the method according to any one of claims 1 to 7 by calling the computer program stored in the memory.
12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Fingerprint feature matching method and device
CN106060779A
Object pose detection tracking method and device, electronic equipment and storage medium
CN112800806A
Mechanical arm grabbing system and method based on feature matching
CN112883984A
Panoramic image feature point matching model training method and device and server
CN113240031A
Facial recognition for masked individuals
US20210374468A1
