A method for obtaining the projection coordinates of feature points in a two-dimensional image

Abstract

The present invention relates to a method for obtaining the projection coordinates of feature points in a two-dimensional image, comprising the following steps: S1. An image acquisition device is used to acquire a two-dimensional image of a target object and to acquire a three-dimensional model of the target object; S2. Based on the pose of the image acquisition device, the target pose of the target object in the coordinate system of the image acquisition device is obtained; S3. The attitude of the three-dimensional model is adjusted and fitted to the target pose; S4. After the fitting is completed, the projection coordinates of the feature points selected in the two-dimensional image are obtained based on the three-dimensional model. By adopting the method of obtaining the feature point pairs in the two-dimensional image and the three-dimensional model to fit the target object and the three-dimensional model in the two-dimensional image in sequence, the difficulty of fitting the three-dimensional model to the two-dimensional image is effectively solved. By sequentially adjusting the three-dimensional model based on the fitted feature point pairs, the number of times of adjusting the three-dimensional model is reduced, and the fitting efficiency is ensured.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular, to a method for obtaining the projection coordinates of feature points in a two-dimensional image. Background Art

[0002] Pose estimation is applied in many fields such as robot vision, motion tracking, and single-camera calibration. The sensors used for pose estimation are different in different fields. Here, the focus is on vision-based pose estimation. Vision-based pose estimation can be further divided into monocular vision pose estimation and multiocular vision pose estimation according to the number of cameras used.

[0003] Currently, many research solutions are basically based on laser point clouds to estimate the target pose. The hardware devices for obtaining data are extremely costly and have a short service life. At the same time, the data volume is large, the processing process is complex, and the efficiency is slow. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for obtaining the projection coordinates of feature points in a two-dimensional image, so as to solve the problem of slow efficiency in the process of obtaining the projection coordinates of feature points.

[0005] To achieve the above object of the invention, the present invention provides a method for obtaining the projection coordinates of feature points in a two-dimensional image, including the following steps:

[0006] S1. Use an image acquisition device to obtain a two-dimensional image of a target object and obtain a three-dimensional model of the target object;

[0007] S2. Based on the pose of the image acquisition device, obtain the target pose of the target object in the coordinate system of the image acquisition device;

[0008] S3. Adjust the pose of the three-dimensional model and fit it to the target pose;

[0009] S4. After the fitting is completed, based on the three-dimensional model, obtain the projection coordinates of the selected feature points in the two-dimensional image.

[0010] According to one aspect of the present invention, in step S3, in the step of adjusting the pose of the three-dimensional model and fitting it to the target pose, it includes;

[0011] S31. Import the two-dimensional image into the three-dimensional model;

[0012] S32. Obtain pairs of corresponding feature points in the two-dimensional image and the three-dimensional model, and sequentially adjust the pose of the three-dimensional model based on the pairs of feature points.

[0013] According to one aspect of the present invention, step S32 includes:

[0014] S321. Obtain the corresponding first pair of feature points in the two-dimensional image and the three-dimensional model, move the three-dimensional model, and align the feature points included in the first pair of feature points;

[0015] S322. Obtain the second pair of feature points, and taking the first pair of feature points as a reference, adjust the pose of the three-dimensional model to align the feature points included in the second pair of feature points;

[0016] S323. Obtain the third pair of feature points, and taking both the first pair of feature points and the second pair of feature points as references, adjust the pose of the three-dimensional model to align the feature points included in the third pair of feature points;

[0017] S324. Sequentially obtain other pairs of feature points, and taking the already aligned pairs of feature points as references. Adjust the pose of the three-dimensional model to align the feature points included in all the pairs of feature points respectively in sequence.

[0018] According to one aspect of the present invention, in step S32, for each pair of feature points aligned, calculate the matching error of the pair of feature points, and compare the obtained error result with a preset threshold.

[0019] According to one aspect of the present invention, in step S32, if all the pairs of feature points are aligned, select the boundaries of the two-dimensional image and the three-dimensional model for boundary error calculation, and compare the obtained error result with a preset threshold.

[0020] According to one aspect of the present invention, in step S2, after obtaining the target pose of the target object, based on the target pose, construct the first minimum three-dimensional envelope of the target object in the two-dimensional image, and construct the second minimum three-dimensional envelope of the three-dimensional model based on the three-dimensional model.

[0021] According to one aspect of the present invention, both the first minimum three-dimensional envelope and the second minimum three-dimensional envelope are rectangular parallelepiped envelopes.

[0022] According to one aspect of the present invention, in the step of obtaining the corresponding pairs of feature points in the two-dimensional image and the three-dimensional model in step S32, the pairs of feature points are the vertex feature point pairs and the center point feature point pairs of the first minimum three-dimensional envelope and the second minimum three-dimensional envelope.

[0023] According to one aspect of the present invention, the number of pairs of feature points is 9.

[0024] According to one solution of the present invention, by adopting the method of obtaining the feature point pairs in the two-dimensional image and the three-dimensional model to fit the target object in the two-dimensional image and the three-dimensional model in sequence, the difficulty of fitting the three-dimensional model with the two-dimensional image is effectively solved. At the same time, by sequentially adjusting the three-dimensional model based on the fitted feature point pairs during the process of matching the remaining feature point pairs, the number of times of adjusting the three-dimensional model is effectively reduced, ensuring the fitting efficiency of the present invention.

[0025] According to one solution of the present invention, by calculating the matching error of the aligned feature point pairs, the matching accuracy of the subsequent feature point pairs is effectively guaranteed, and further the accuracy of finally obtaining the projection coordinates of all selected feature points can be effectively guaranteed.

[0026] According to one solution of the present invention, by calculating the boundary error of the boundaries between the two-dimensional image and the three-dimensional model after the feature point alignment is completed, the effectiveness of fitting the two-dimensional image with the three-dimensional model is further effectively determined, and further the accuracy of finally obtaining the projection coordinates of all selected feature points can be more effectively guaranteed.

[0027] According to one solution of the present invention, the minimum three-dimensional envelopes are respectively constructed for the target object in the two-dimensional image and the three-dimensional model, which effectively simplifies the external shape structure of the target object, especially for irregular objects with complex external shapes. When constructing the three-dimensional envelopes and matching the feature point pairs on the three-dimensional envelopes, not only the computational amount of the matching process is effectively simplified, but also the original matching accuracy is guaranteed.

[0028] According to one solution of the present invention, by selecting the obvious feature point pairs on the two three-dimensional envelopes, not only the fitting accuracy is guaranteed, but also the computational amount is simplified and the fitting efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart schematically showing the steps of a method for obtaining the projection coordinates of feature points in a two-dimensional image according to an embodiment of the present invention;

[0030] Figure 2 A flowchart schematically showing the model matching process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] When describing the embodiments of the present invention, the orientation or positional relationships expressed by the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" are based on the orientation or positional relationships shown in the relevant drawings. These are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting the present invention.

[0033] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments cannot be enumerated one by one here, but the embodiments of the present invention are not limited to the following embodiments.

[0034] As Figure 1 shown, according to an embodiment of the present invention, a method for obtaining the projection coordinates of feature points in a two-dimensional image includes the following steps:

[0035] S1. Use an image acquisition device to obtain a two-dimensional image of the target object and obtain a three-dimensional model of the target object;

[0036] S2. Based on the pose of the image acquisition device, obtain the target pose of the target object in the coordinate system of the image acquisition device;

[0037] S3. Adjust the pose of the three-dimensional model and fit it to the target pose;

[0038] S4. After the fitting is completed, based on the three-dimensional model, obtain the projection coordinates of the feature points selected in the two-dimensional image.

[0039] According to an embodiment of the present invention, in step S3, in the step of adjusting the pose of the three-dimensional model and fitting it to the target pose, it includes;

[0040] S31. Import the two-dimensional image into the three-dimensional model;

[0041] S32. Obtain the corresponding feature point pairs in the two-dimensional image and the three-dimensional model, and sequentially adjust the pose of the three-dimensional model based on the feature point pairs.

[0042] According to an embodiment of the present invention, step S32 includes:

[0043] S321. Obtain the corresponding first feature point pair in the two-dimensional image and the three-dimensional model, move the three-dimensional model, and align the feature points included in the first feature point pair;

[0044] S322. Obtain the second feature point pair, and based on the first feature point pair, adjust the pose of the three-dimensional model to align the feature points included in the second feature point pair;

[0045] S323. Obtain the third pair of feature points, and at the same time, taking the first pair of feature points and the second pair of feature points as a reference, adjust the pose of the three-dimensional model to align the feature points included in the third pair of feature points;

[0046] S324. Sequentially obtain other pairs of feature points, and at the same time, taking the pairs of feature points that have been aligned as a reference. Adjust the pose of the three-dimensional model to align the feature points included in all pairs of feature points respectively in sequence.

[0047] According to the present invention, by adopting the method of obtaining pairs of feature points in the two-dimensional image and the three-dimensional model to sequentially fit the target object in the two-dimensional image and the three-dimensional model, the difficulty of fitting the three-dimensional model with the two-dimensional image is effectively solved. At the same time, by sequentially adjusting the three-dimensional model with the pairs of feature points that have been fitted as a reference during the process of matching the remaining pairs of feature points, the number of times of adjusting the three-dimensional model is effectively reduced, ensuring the fitting efficiency of the present invention.

[0048] According to an embodiment of the present invention, in step S32, every time a pair of feature points is aligned, calculate the matching error of the pair of feature points, and compare the obtained error result with a preset threshold.

[0049] According to the present invention, by calculating the matching error of the pairs of feature points that have been aligned, the matching accuracy of the subsequent pairs of feature points is effectively guaranteed, and thus the accuracy of the projection coordinates of all the selected feature points finally obtained can be effectively guaranteed.

[0050] According to an embodiment of the present invention, in step S32, if all pairs of feature points are aligned, select the boundaries of the two-dimensional image and the three-dimensional model to calculate the boundary error, and compare the obtained error result with a preset threshold.

[0051] According to the present invention, by calculating the boundary error of the boundaries of the two-dimensional image and the three-dimensional model after the alignment of the feature points, the effectiveness of the fitting of the two-dimensional image and the three-dimensional model is further effectively determined, and thus the accuracy of the projection coordinates of all the selected feature points finally obtained can be more effectively guaranteed.

[0052] According to an embodiment of the present invention, in step S2, after obtaining the target pose of the target object, construct the first minimum three-dimensional envelope of the target object in the two-dimensional image based on the target pose, and construct the second minimum three-dimensional envelope of the three-dimensional model based on the three-dimensional model.

[0053] According to an embodiment of the present invention, both the first minimum three-dimensional envelope and the second minimum three-dimensional envelope are cuboid envelopes.

[0054] Through the above settings, the minimum three-dimensional envelopes are constructed for the target object in the two-dimensional image and the three-dimensional model respectively, which effectively simplifies the external structure of the target object, especially for irregular objects with complex shapes. When constructing the three-dimensional envelope and matching the feature point pairs on the three-dimensional envelope, it not only effectively simplifies the computational complexity of the matching process, but also ensures the original matching accuracy.

[0055] According to an embodiment of the present invention, in step S32, in the step of obtaining the corresponding feature point pairs in the two-dimensional image and the three-dimensional model, the feature point pairs are the vertex feature point pairs and the center point feature point pairs of the first minimum three-dimensional envelope and the second minimum three-dimensional envelope. In this embodiment, the number of feature point pairs is 9.

[0056] Combined Figure 2 , the model matching process of the present invention will be further described.

[0057] In this embodiment, the three-dimensional coordinates of the target workpiece in the two-dimensional image are obtained. Here, the representation form of the three-dimensional coordinates is the 9 feature points described above. The position of the target object is random. Even if the world coordinate system is known, its coordinates in the world coordinate system cannot be directly read. However, for the three-dimensional model of the target object generated by the auxiliary software, its pose is artificially defined, and the 3D coordinates of its bounding box can be easily obtained. In the same coordinate system, the three-dimensional model is aligned with the part point cloud. This alignment process is a rigid transformation. By the coordinates of the three-dimensional model before alignment and the pose transformation after alignment, the coordinates of the three-dimensional model after alignment, that is, the coordinates of the target object in the world coordinate system, can be obtained.

[0058] In this embodiment, the Cloudcompare software can provide functions such as model scale transformation, point cloud segmentation, manual registration, automatic registration, etc., and can save the displacement and rotation matrix of the three-dimensional model after registration. Model matching refers to fitting two models with slightly different scales. In order to make the three-dimensional model of the target object and the point cloud consistent in scale, special points need to be selected for size measurement, and the size of the three-dimensional model is adjusted according to the ratio. After the size adjustment is completed, the coordinates of the bounding box of the three-dimensional model are read, and the scale transformation of the three-dimensional model is performed according to the coordinate values.

[0059] In this embodiment, in order to reduce errors and increase the matching accuracy, rough matching is first performed manually to adjust the position and pose of the three-dimensional model to make the three-dimensional model and the point cloud roughly consistent in direction, and the transformation matrix after matching is saved. The transformation matrix is the rotation matrix R and the displacement matrix T.

[0060] In this embodiment, after the rough matching is completed, automatic registration is performed. Through the point cloud automatic matching function based on the ICP algorithm in the Cloudcompare software, the point cloud data with different coordinates is merged into the same coordinate system. Essentially, the ICP algorithm is an optimal registration method based on the least squares method. This algorithm repeatedly selects corresponding point pairs and calculates the optimal rigid body transformation until the convergence accuracy requirement of the preset registration is met. In this embodiment, the purpose of the ICP algorithm is to find the rotation matrix R and displacement matrix T between the point cloud data to be registered and the reference point cloud data, so that the two point data are optimal under the set metric criteria.

[0061] In this embodiment, let the coordinates of the boundary box after scale adjustment be p i , and the coordinates after two transformations be p j . Let the two transformations be R1, T1, R2, and T2 respectively. Then the following relationship is satisfied: p j = R2(R1p i + T1) + T2, and the 3D coordinates of the boundary box of the target object in the world coordinate system can be calculated.

[0062] Through the above settings, by selecting obvious feature point pairs on two three-dimensional envelopes, not only the fitting accuracy is guaranteed, but also the calculation amount is simplified and the fitting efficiency is improved.

[0063] The above content is only an example of the specific solution of the present invention. For the devices and structures not described in detail therein, it should be understood that the existing general devices and general methods in the art are adopted for implementation.

[0064] The above is only one solution of the present invention and is not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for obtaining the projection coordinates of feature points in a two-dimensional image for target pose estimation, comprising the following steps: S1. Obtain a two-dimensional image of a target object by using an image acquisition device, and obtain a three-dimensional model of the target object; S2. Obtain the target pose of the target object in the coordinate system of the image acquisition device based on the pose of the image acquisition device; S3. Adjust the pose of the three-dimensional model and fit it to the target pose; S4. After the fitting is completed, obtain the projection coordinates of the selected feature points in the two-dimensional image based on the three-dimensional model; In step S2, after obtaining the target pose of the target object, construct a first minimum three-dimensional envelope of the target object in the two-dimensional image based on the target pose, and construct a second minimum three-dimensional envelope of the three-dimensional model based on the three-dimensional model; Both the first minimum three-dimensional envelope and the second minimum three-dimensional envelope are rectangular parallelepiped envelopes; In step S3, the steps of adjusting the pose of the three-dimensional model and fitting it to the target pose include; S31. Import the two-dimensional image into the three-dimensional model; S32. Obtain corresponding feature point pairs in the two-dimensional image and the three-dimensional model, and sequentially adjust the pose of the three-dimensional model based on the feature point pairs; in step S32, in the step of obtaining corresponding feature point pairs in the two-dimensional image and the three-dimensional model, the feature point pairs are vertex feature point pairs and center point feature point pairs of the first minimum three-dimensional envelope and the second minimum three-dimensional envelope; In step S32, it includes: S321. Obtain a first corresponding feature point pair in the two-dimensional image and the three-dimensional model, move the three-dimensional model, and align the feature points included in the first feature point pair; S322. Obtain a second feature point pair, and adjust the pose of the three-dimensional model with the first feature point pair as a reference to align the feature points included in the second feature point pair; S323. Obtain a third feature point pair, and simultaneously adjust the pose of the three-dimensional model with the first feature point pair and the second feature point pair as references to align the feature points included in the third feature point pair; S324. Sequentially obtain other feature point pairs, and simultaneously use the already aligned feature point pairs as references; adjust the pose of the three-dimensional model to align the feature points included in all the feature point pairs respectively in sequence.

2. The method according to claim 1, wherein In step S32, for each alignment of a feature point pair, calculate the matching error of the feature point pair, and compare the obtained error result with a preset threshold.

3. The method according to claim 2, characterized in that, In step S32, if all the feature point pairs are aligned, select the boundaries of the two-dimensional image and the three-dimensional model to calculate the boundary error, and compare the obtained error result with a preset threshold.

4. The method according to claim 1, characterized in that The number of the feature point pairs is 9.

Images