Feature descriptor extraction method and three-dimensional tracking registration method

Abstract

The invention provides a feature descriptor extraction method and a three-dimensional tracking registration method, and the method comprises the steps: extracting angular points in a to-be-processed image through employing a KLT algorithm, and obtaining an angular point set; extracting straight line segments in the to-be-processed image based on an EDLines straight line detection algorithm to obtain an initial line segment set; screening line segments passing through at least one angular point in the angular point set from the initial line segment set to obtain a screened line segment set; combining the screened line segments in the line segment set according to a preset combination rule to obtain a combined line segment set; and calculating a feature descriptor of each line segment in the combined line segment set, and forming a feature descriptor set of the to-be-processed image based on the feature descriptor of each line segment. The method effectively reduces the calculation amount on the basis of guaranteeing the feature integrity, improves the tracking registration efficiency, is also associated with the key image feature points, guarantees the accuracy of the extracted feature descriptors, and guarantees the accuracy of tracking registration.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and more specifically, to a method for extracting feature descriptors and a three-dimensional tracking and registration method. Background Art

[0002] With the rapid development of intelligent manufacturing technology, augmented reality has become an effective technology for assisting manual assembly operations. It uses computer three-dimensional rendering and tracking and registration technologies to superimpose virtual objects onto the real physical space to guide and prompt operators to complete the assembly tasks of products, thereby improving assembly efficiency and saving production costs.

[0003] In three-dimensional tracking and registration technology, a key step is to extract feature descriptors from the collected images. In the prior art, some methods form a directed chain by connecting the head and tail of the directed line segments in the two-dimensional image of mechanical parts, and construct feature descriptors according to the shape of the chain.

[0004] The process of forming a directed chain from directed lines in this method is relatively complex, and the process of forming a chain-like feature vector by arbitrarily arranging and combining the feature lines in both forward and reverse directions according to the adjacency relationship has a large amount of calculation. There are a large number of redundant features in this process, and it is difficult to achieve real-time three-dimensional tracking and registration of the target object. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method for extracting feature descriptors and a three-dimensional tracking and registration method to reduce the amount of calculation in the tracking and registration process, improve the registration efficiency, and achieve real-time tracking and registration.

[0006] In a first aspect, a method for extracting feature descriptors is provided, which is applied to a three-dimensional tracking and registration scenario in the assembly process. The method includes:

[0007] Based on the KLT algorithm, corner points in the image to be processed are extracted to obtain a set of corner points; at the same time, based on the EDLines straight line detection algorithm, straight line segments in the image to be processed are extracted to obtain an initial set of line segments; the image to be processed is a virtual model image or a real-time image of the part to be assembled;

[0008] In the initial set of line segments, line segments passing through at least one corner point in the set of corner points are screened to obtain a screened set of line segments;

[0009] Each line segment in the screened set of line segments is merged according to a preset merging rule to obtain a merged set of line segments;

[0010] The feature descriptor of each line segment in the merged set of line segments is calculated, and a set of feature descriptors of the image to be processed is formed based on the feature descriptors of each line segment; wherein, the feature descriptor of each line segment at least includes the included angle and the shortest distance between each line segment and other line segments.

[0011] Optionally, the preset merging rule is:

[0012] Determine whether the angle between any two line segments to be merged is less than a preset angle threshold and whether the shortest distance between any two line segments to be merged is less than a preset distance threshold;

[0013] If both are satisfied, then merge.

[0014] In a second aspect, a three-dimensional tracking and registration method is provided. The method includes:

[0015] Obtain a real-time captured image of the part to be assembled;

[0016] Extract a first set of feature descriptors of the real-time captured image based on the method described in the first aspect;

[0017] Perform similarity matching between the first set of feature descriptors and a second set of feature descriptors of all template images in a template information library pre-constructed for the virtual model of the part to be assembled, and obtain the camera pose of the template image with the maximum similarity;

[0018] Register the virtual model of the part to be assembled into the real physical device based on the obtained camera pose;

[0019] When the acquisition view of the part to be assembled changes, perform corner tracking on the real-time image of the part to be assembled based on the KLT algorithm, and update the camera pose based on the tracking result for real-time tracking and registration.

[0020] Optionally, the construction process of the template information library includes:

[0021] Determine a plurality of acquisition viewpoints;

[0022] Obtain images of the virtual model of the part to be assembled acquired at each acquisition viewpoint;

[0023] Extract a set of feature descriptors of the image of each virtual model based on the method described in the first aspect;

[0024] Store the set of feature descriptors of the image of each virtual model and the camera pose at the acquisition viewpoint corresponding to the image as a set of data in the template information library, where the camera pose is the camera pose in the coordinate system with the virtual model as the origin.

[0025] Optionally, determining a plurality of acquisition viewpoints includes:

[0026] Determine the acquisition surface according to the type of the part to be assembled; the acquisition surface is a hemispherical surface or a global surface;

[0027] Randomly generate a number of initial acquisition viewpoints on the acquisition surface;

[0028] The position of the initial acquisition viewpoint is corrected to obtain acquisition viewpoints with a uniform distribution.

[0029] Optionally, correcting the position of the initial acquisition viewpoint to obtain acquisition viewpoints with a uniform distribution includes:

[0030] Regarding the initial acquisition viewpoints as particles with forces acting on each other, and presetting an initial velocity for each initial acquisition viewpoint to obtain an initial velocity matrix;

[0031] Calculating the forces exerted on each acquisition viewpoint by other acquisition viewpoints in three-dimensional space based on the initial positions of the initial acquisition viewpoints;

[0032] Updating the initial velocity matrix based on the forces;

[0033] Updating the positions of each acquisition viewpoint based on the updated velocity matrix; until the modulus value of the velocity matrix meets a preset threshold.

[0034] Optionally, when the acquisition perspective of the part to be assembled changes, performing corner tracking on the real-time image of the part to be assembled based on the KLT algorithm, and updating the camera pose based on the tracking result includes:

[0035] When the acquisition perspective of the part to be assembled changes, matching the corners of the images acquired before and after the change in the acquisition perspective;

[0036] If the number of matched corners is greater than a preset number threshold, calculating the transformation matrix of the corners based on the corner features before and after the change;

[0037] Updating the camera pose based on the transformation matrix of the corners;

[0038] If the number of matched corners is less than the preset number threshold, re-acquiring the camera pose under the current acquisition perspective based on the similarity matching method with the template image.

[0039] In a third aspect, a three-dimensional tracking and registration device is provided. The device includes:

[0040] An acquisition unit for acquiring a real-time acquisition image of the part to be assembled;

[0041] An extraction unit for extracting a first set of feature descriptors of the real-time acquisition image based on the method of the first aspect;

[0042] A matching unit for performing similarity matching between the first set of feature descriptors and a second set of feature descriptors of all template images in a template information library pre-constructed for the virtual model of the part to be assembled, and obtaining the camera pose of the template image with the maximum similarity;

[0043] A registration unit for registering the virtual model of the part to be assembled into a real physical device based on the obtained camera pose;

[0044] A tracking unit, configured to, when the acquisition perspective of the part to be assembled changes, perform corner point tracking on the real-time image of the part to be assembled based on the KLT algorithm, and update the camera pose based on the tracking result for real-time tracking registration.

[0045] In a fourth aspect, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0046] The memory is used to store a computer program;

[0047] The processor is configured to, when executing the program stored on the memory, implement the method steps described in any one of the first aspect.

[0048] In a fifth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first aspect are implemented.

[0049] A feature descriptor extraction method and a three-dimensional tracking registration method provided by the present invention. This method uses the KLT algorithm to extract corner points in the image to be processed, obtaining a set of corner points; at the same time, based on the EDLines line detection algorithm, straight line segments in the image to be processed are extracted, obtaining an initial set of line segments; in the initial set of line segments, line segments passing through at least one corner point in the set of corner points are screened, obtaining a screened set of line segments; each line segment in the screened set of line segments is merged according to a preset merging rule, obtaining a merged set of line segments; the feature descriptor of each line segment in the merged set of line segments is calculated, and a set of feature descriptors of the image to be processed is formed based on the feature descriptors of each line segment. Compared with the existing method using a directed chain, only the straight lines passing through the corner points are saved and merged, effectively reducing the amount of calculation while ensuring feature integrity, improving the efficiency of tracking registration, and also being associated with key image feature points, ensuring the accuracy of the extracted feature descriptors, and thus ensuring the accuracy of tracking registration.

[0050] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given in conjunction with the accompanying drawings and are described in detail as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0052] Figure 1 shows a flowchart of a feature descriptor extraction method provided by an embodiment of the present invention;

[0053] Figure 2 shows a flowchart of a three-dimensional tracking and registration method provided by an embodiment of the present invention;

[0054] Figure 3 shows a schematic structural diagram of a three-dimensional tracking and registration device provided by an embodiment of the present invention;

[0055] Figure 4 shows a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Usually, the components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0057] Considering that the process of forming a directed chain from directed lines in the existing feature descriptor extraction method is relatively complex, and the process of forming a chain-like feature vector by arbitrarily arranging and combining feature lines in both positive and negative directions according to the adjacency relationship has a large amount of calculation, there are a large number of redundant features in this process, and it is difficult to achieve real-time three-dimensional tracking and registration of the target object.

[0058] Based on this, the embodiments of the present invention provide a feature descriptor extraction method and a three-dimensional tracking and registration method based on the feature descriptor extraction method, which will be described below through embodiments.

[0059] The embodiments of the present invention provide a feature descriptor extraction method, which is applied to the three-dimensional tracking and registration scenario in the assembly process, such as Figure 1 as shown, and the method includes the following steps:

[0060] Step S101: Extract corner points in the image to be processed based on the KLT (Kanade-Lucas-Tomasi) algorithm to obtain a set of corner points; at the same time, extract straight line segments in the image to be processed based on the EDLines straight line detection algorithm to obtain an initial set of line segments.

[0061] Among them, the image to be processed is a virtual model image or a real-time image of the part to be assembled.

[0062] In this step, the KLT algorithm is a feature point tracking algorithm based on the optical flow principle. In multiple continuously acquired frames of images, first, the first image is selected as the initial frame, a set of corner points is identified in the initial frame, and then in subsequent frames, these corner points are tracked through the KTL algorithm to obtain the positions of the corner points in each frame of the image. The corner points extracted from all frames of images form a corner point set P.

[0063] Corner points are usually defined as pixel points with significant changes in their small neighborhoods. Specifically, corner points are places where the brightness or color in the image changes drastically. They are often one of the most stable features in the image and can remain unchanged under different perspectives, scales, and lighting conditions. Using corner points as key feature points of the image helps improve the accuracy of image processing.

[0064] The EDLines algorithm is an efficient line detection method, especially suitable for images rich in edge information. The main steps of this algorithm are as follows:

[0065] The first step, edge detection: First, perform edge detection (such as the Canny algorithm) on the input image to highlight the edge pixels that may form line segments.

[0066] The second step, line segment extraction: Based on these edge pixels, the EDLines algorithm determines potential line segments by analyzing the direction and continuity of the edge pixels.

[0067] The third step, set generation: All detected line segments form an initial line segment set L def .

[0068] Step S102: Screen the line segments in the initial line segment set that pass through at least one corner point in the corner point set to obtain a screened line segment set.

[0069] In this step, to determine whether a corner point is located on a line segment, the following method can be used. First, each line segment can be represented by the general expression of a straight line: y = ax + b, and the corner point can be represented by pixel coordinates (u, v). Substitute the coordinates of each corner point in the corner point set into the expression of the line segment. If the straight line expression is satisfied, it means that the corner point is located on the line segment, indicating that the line segment passes through the corner point, and then it is retained. If all corner points are not located on this line segment, it is deleted to obtain a screened line segment set L Pro .

[0070] In the embodiment of the present invention, by combining the detected corner points and line segments to screen the initial line segments, it helps to ensure that the set of screened line segments not only accurately reflects the straight line structure in the image, but also is associated with key image features (corner points), providing a solid foundation for subsequent processing such as line segment merging.

[0071] Step S103: Merge each line segment in the set of screened line segments according to a preset merging rule to obtain a set of merged line segments.

[0072] In this step, the preset merging rule is:

[0073] Judge whether the included angle between any two line segments to be merged is less than a preset included angle threshold and whether the shortest distance between any two line segments to be merged is less than a preset distance threshold.

[0074] In this step, calculate the included angle between two line segments to be merged: First, obtain the included angle of each line segment with the positive direction of the X-axis in the image coordinate system, and use the difference between the included angles of the two line segments with the positive direction of the X-axis to calculate the included angle between the two line segments to be merged.

[0075] The shortest distance between two line segments to be merged can be determined by the distances between the endpoints at both ends of the two line segments respectively. For example, the two endpoints of line segment A are A1 and A2, and the two endpoints of line segment B are B1 and B2. Then calculate the distances between A1 - B1, A1 - B2, A2 - B1, and A2 - B2 respectively. After comparison, it is known that A1 and B1 are the closest, so the distance between A1 and B1 can be used as the shortest distance between the two line segments.

[0076] If both are satisfied, then merge.

[0077] In a specific example, the preset merging rule can be represented by the following formula 1:

[0078]

[0079] Among them, α and β respectively represent the included angles of the two line segments with the positive direction of the X-axis, and t ang represents the preset included angle threshold. A and B respectively represent the nearest neighbor endpoints of the two line segments, and t dis represents the preset distance threshold. Merge the two line segments that meet the above conditions into a long line segment, and the set of line segments after merging all line segments is denoted as L fin .

[0080] Step S104: Calculate the feature descriptors of each line segment in the set of merged line segments, and form a set of feature descriptors of the image to be processed based on the feature descriptors of each line segment.

[0081] Among them, the feature descriptor of each line segment includes at least the angle between each line segment and other line segments and the shortest distance.

[0082] In a specific example, calculate the angles and distances between each line segment in the line segment set L fin The calculation method refers to the method in the above steps and will not be elaborated here. Based on the angle and distance, establish the feature descriptor of the line segment. The feature descriptor set in each image to be processed can be composed of the feature descriptors of each line segment.

[0083] It can be understood from the above embodiments that the present invention extracts the feature descriptor set of the image by combining corner points and line segments, only preserves the straight lines passing through the corner points and merges the short line segments, effectively reducing the calculation amount on the basis of ensuring feature integrity. It can not only accurately reflect the straight line structure in the image, but also be associated with the key image feature points, and does not need to consider the directionality of the line segments. Compared with the calculation methods of the prior art, it not only reduces the calculation amount, improves the efficiency, but also ensures that the key feature points of the image can be extracted at the same time, thereby ensuring the accuracy of the extracted feature descriptors.

[0084] Based on the same inventive concept, the embodiment of the present invention provides a three-dimensional tracking and registration method, which is applied to a mechanical assembly scenario, such as the assembly of a generator, as Figure 2 shown, and the method includes the following steps:

[0085] Step S201: Obtain a real-time captured image of the part to be assembled.

[0086] In an example, the part to be assembled is a spare part of a generator.

[0087] Step S202: Extract the first feature descriptor set of the real-time captured image based on the feature descriptor extraction method.

[0088] The feature descriptor extraction method is the feature descriptor method in the above embodiments and will not be elaborated here.

[0089] In an example, the first feature descriptor set is denoted as D real .

[0090] Step S203: Perform similarity matching between the first feature descriptor and the second feature descriptor sets of all template images in the template information library pre-constructed for the virtual model of the part to be assembled, and obtain the camera pose of the template image with the largest similarity.

[0091] Among them, the construction process of the template information library includes:

[0092] Determine multiple acquisition viewpoints;

[0093] Obtain the images of the virtual models of the parts to be assembled collected at each acquisition viewpoint;

[0094] Extract the set of feature descriptors of the image of each virtual model based on the feature descriptor extraction method of the above embodiment;

[0095] Store the set of feature descriptors of the image of each virtual model and the camera pose at the acquisition viewpoint corresponding to this image as a set of data in the template information library, where the camera pose is the camera pose in the coordinate system with the virtual model as the origin.

[0096] In this step, the second set of feature descriptors is denoted as D vir .

[0097] The construction process of this template information library can be carried out in an offline state.

[0098] In one example, the Cosine similarity algorithm is used for similarity calculation. Assume the first set of feature descriptors where the second set of feature descriptors Then the similarity calculation formula is as follows:

[0099]

[0100] where, T is the similarity value; x i represents the i-th feature descriptor in the first set of feature descriptors; y i represents the i-th feature descriptor in the second set of feature descriptors; k represents the number of feature descriptors in the set of feature descriptors.

[0101] The embodiment of the present invention uses the Cosine similarity algorithm to calculate the similarity of feature descriptors, which can improve the accuracy of feature matching and ensure the accuracy of the final 3D registration result.

[0102] Step S204: Register the virtual model of the part to be assembled into the real physical device based on the obtained camera pose.

[0103] If the similarity between the real-time acquired image and one of the template images is the highest, then use the camera pose corresponding to this template image as the pose of the current camera. Since this camera pose is the pose data constructed in the coordinate system with the virtual model as the origin, when the two match, it means that the camera pose of this virtual model is consistent with the coordinate system with the real physical device as the coordinate origin, and the coordinate system of this virtual model is aligned with the coordinate system of the real physical device.

[0104] In one example, for instance, the real physical device is the output shaft of a generator, and the assembly part is a bearing. Pre-registering the virtual model of the bearing onto the output shaft can assist the staff in better installing the bearing. Specifically, the virtual model can be constructed through CAD.

[0105] When the staff observes the augmented reality scene where the virtual model is combined with the real physical device, the essence of the final presentation of the virtual model is still through two-dimensional interface images. Therefore, the internal parameter matrix of the camera can be combined and the points in the camera coordinate system can be transformed to the imaging coordinate system and pixel coordinate system through coordinate transformation, so that the virtual model can be displayed at the correct position to achieve the three-dimensional registration of the model.

[0106] Since the position of the staff and the camera view angle are dynamically changing, in order to better present the effect of the combination of virtual and real, when the camera view angle or the staff's observation view angle changes, it is necessary to readjust the position and pose of the virtual model to increase the sense of reality. Therefore, the following steps are adopted for real-time tracking registration.

[0107] Step S205: When the acquisition view angle of the part to be assembled changes, perform corner tracking on the real-time image of the part to be assembled based on the KLT algorithm, and update the camera pose based on the tracking result for real-time tracking registration.

[0108] The process of this tracking registration will be described in detail in the following embodiments and will not be elaborated here.

[0109] The present invention extracts the feature descriptors of the real-time acquired image and the template image by adopting an improved feature descriptor extraction method, improving the efficiency of tracking registration, and improving the accuracy of the three-dimensional registration result by performing similarity matching with the feature descriptors of the template image.

[0110] In the offline stage, in order to make the distribution of acquisition viewpoints relatively uniform, the acquisition viewpoints are distributed on the hemisphere or the sphere. When the acquisition viewpoints are distributed according to longitude and latitude on the hemisphere or the global sphere, there will be a situation where the acquisition points near the poles are too dense and the acquisition points near the equator are too sparse. Therefore, based on the above embodiments, determining multiple acquisition viewpoints includes:

[0111] Step A: Determine the acquisition surface according to the type of the part to be assembled. The acquisition surface is a hemisphere or a global sphere.

[0112] In one example, for instance, if the part to be assembled is a motor base and the motor base is located on the ground, then the lower half of it does not need to be imaged. Therefore, a hemisphere can be used as the acquisition surface, which can reduce the acquisition data and the calculation amount. If the part to be assembled is small, such as a bearing, then 360-degree image acquisition is required, and then a global sphere acquisition method can be adopted.

[0113] Step B: Randomly generate a number of initial acquisition viewpoints on the acquisition surface.

[0114] In one example, a number of initial acquisition viewpoints can be randomly generated by the following formula:

[0115] a = rand(n,1)*2*π (3);

[0116] b = arcsin(rand(n,1)*2 - 1) (4);

[0117] where rand(n,1) represents generating a random number matrix of size n*1, and each element value in the matrix is between 0 and 1. Multiplying by 2π is to expand the range to 0 - 2π to correspond to the azimuth angle a;

[0118] Similarly, for the elevation angle b, multiplying by 2 and then subtracting 1 changes the range to -1 to 1; the elevation angle b is obtained through the arcsine transformation.

[0119] Based on the randomly generated azimuth angle a and elevation angle b, through the following position conversion formula, calculate the coordinates of all acquisition viewpoints in the three dimensions of XYZ to form an initial position matrix. The position conversion formula is:

[0120] P = [cos(a).*cos(b), sin(a).*cos(b), sin(b)] (5).

[0121] Step C: Correct the positions of the initial acquisition viewpoints to obtain acquisition viewpoints with a uniform distribution.

[0122] In a feasible implementation, correcting the positions of the initial acquisition viewpoints to obtain acquisition viewpoints with a uniform distribution includes:

[0123] Step C1: Treat the initial acquisition viewpoints as particles with forces acting on each other, and preset an initial velocity for each initial acquisition viewpoint to obtain an initial velocity matrix.

[0124] In this step, different initial velocities are assigned to each particle, and the initial velocity matrix is denoted as V.

[0125] Step C2: Calculate the forces exerted on each acquisition viewpoint by other acquisition viewpoints in three-dimensional space based on the initial positions of the initial acquisition viewpoints.

[0126] In one example, Coulomb's law is used to calculate the resultant force exerted on each particle by other particles in the three dimensions of XYZ. For example, the resultant force F j (1 ≤ j ≤ b) received by particle j is calculated as follows:

[0127]

[0128] where D jk =(P j -P k ) T ; D kj =-D jk (7);

[0129]

[0130] where D jk (1≤j,k≤n) and D kj represents the vector difference between particle j and particle k. L jk represents the distance between particle j and particle k; m represents one of the three dimensions of x, y, and z; n is the number of particles.

[0131] Step C3: Update the initial velocity matrix based on the acting force.

[0132] In this step, the resultant force is split into solving the radial component and the tangential component

[0133] The tangential component is used to update the velocity matrix, and the update formula is:

[0134]

[0135] where V is the initial velocity matrix; V ′ is the updated velocity matrix; G is the repulsive force constant.

[0136] In an example, the radial component and the tangential component

[0137] are solved through the following steps

[0138] First step, set the coordinates of the center of the sphere of the acquisition surface to (0, 0, 0); Second step, given the position coordinates of one of the acquisition viewpoints P and the coordinates of the center of the sphere, the line connecting the center of the sphere and the acquisition viewpoint P is the normal vector, that is, the radial direction of the resultant force

[0139] Third step, calculate the angle between the resultant force and ;

[0140]

[0141] Fourth step, solve the radial vector tangential vector

[0142] Step C4: Update the positions of each acquisition view point based on the updated velocity matrix; until the modulus value of the velocity matrix meets the preset threshold.

[0143] Repeat the above position correction process until the modulus ||V|| of the velocity matrix meets the requirements of the preset threshold. At this time, the particles reach a stable equilibrium state on the spherical surface, that is, the acquisition view points evenly distributed on the spherical surface can be obtained. Regard the particle positions as the acquisition view points of the virtual model of the part, store the acquired images as template images, and record the pose information of the camera under each acquisition view angle respectively.

[0144] The present invention generates acquisition view points on the hemispherical surface or the global spherical surface through the balanced force method, which can make the distribution of the acquisition points relatively uniform, improve the data quality of the template information library, and further improve the accuracy of the 3D registration result.

[0145] Based on the above embodiments, when the acquisition view angle of the part to be assembled changes, corner tracking is performed on the real-time image of the part to be assembled based on the KLT algorithm. Updating the camera pose based on the tracking result includes:

[0146] Step S205A: When the acquisition view angle of the part to be assembled changes, match the corners of the images acquired before and after the change of the acquisition view angle.

[0147] In this step, the KLT algorithm is used to track the corners between the images acquired before and after the view angle change;

[0148] For example, image A is the image acquired before the view angle change, and image B is the image acquired after the view angle change. The corners on these two images are tracked through the KLT algorithm, and it is found that the number of identical corners on image A and image B is 20.

[0149] Step S205B: If the number of matched corners is greater than the preset number threshold, calculate the transformation matrix of the corners based on the corner features before and after the change.

[0150] If the number of matched corners is greater than the preset number threshold, it indicates successful tracking. If the tracking is successful, the transformation matrix of the identical corners on image A and image B can be calculated.

[0151] In a feasible implementation manner, before calculating the transformation matrix, the wrong corners can be removed from the matched corners to improve the accuracy of the transformation matrix calculation.

[0152] Step S205C: Update the camera pose based on the transformation matrix of the corners.

[0153] Update the camera pose based on the corner-based transformation matrix. Specifically, multiply the original camera pose by this transformation matrix to obtain the updated camera pose. This method can calculate the new camera pose very quickly, perform re-registration, and improve the efficiency of tracking and registration.

[0154] Step S205D: If the number of matched corners is less than the preset number threshold, re-obtain the camera pose under the current acquisition view based on the similarity matching method with the template image.

[0155] If it is less than the preset number threshold, it indicates that the tracking fails. If the tracking fails, the process of repeating the above steps S201 - S203 is required to recalculate the camera pose.

[0156] Based on the same inventive concept, a three-dimensional tracking and registration device is provided, as Figure 3 shown. The device includes:

[0157] An acquisition unit 301, configured to acquire a real-time acquisition image of the part to be assembled;

[0158] An extraction unit 302, configured to extract a first set of feature descriptors of the real-time acquisition image based on the feature descriptor extraction method;

[0159] A matching unit 303, configured to perform similarity matching between the first set of feature descriptors and a second set of feature descriptors of all template images in the template information library pre-constructed for the virtual model of the part to be assembled, and obtain the camera pose of the template image with the maximum similarity;

[0160] A registration unit 304, configured to register the virtual model of the part to be assembled into the real physical device based on the obtained camera pose;

[0161] A tracking unit 305, configured to, when the acquisition view of the part to be assembled changes, perform corner tracking on the real-time image of the part to be assembled based on the KLT algorithm, and update the camera pose based on the tracking result for real-time tracking and registration.

[0162] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, as Figure 4 shown, including a processor 401, a communication interface 402, a memory 403, and a communication bus 404. Among them, the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0163] The memory 403 is used to store a computer program;

[0164] The processor 401, when executing the program stored on the memory 403, implements the steps of the feature descriptor extraction method or the three-dimensional tracking and registration method.

[0165] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0166] The communication interface is used for communication between the above electronic device and other devices.

[0167] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0168] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0169] The computer program product for the method of feature descriptor extraction or the method of three-dimensional tracking and registration provided by the embodiments of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, reference can be made to the method embodiments, which will not be elaborated here.

[0170] The device for the three-dimensional tracking and registration method provided by the embodiments of the present invention may be specific hardware on a device, or software or firmware installed on the device, etc. The device provided by the embodiments of the present invention has the same implementation principle and technical effects as those of the foregoing method embodiments. For the sake of brief description, for the parts not mentioned in the device embodiments, reference may be made to the corresponding content in the foregoing method embodiments. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the foregoing described systems, devices, and units can all refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0171] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or units can be electrical, mechanical, or other forms.

[0172] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0174] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0175] It should be noted that like reference numerals and letters denote like items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used for descriptive distinction and should not be construed as indicating or implying relative importance.

[0176] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, which are used to illustrate the technical solutions of the present invention rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments or can easily conceive of changes, or make equivalent replacements for some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A feature descriptor extraction method, characterized in that: Applied to a three-dimensional tracking registration scenario in an assembly process, the method comprises: Extracting corner points in the image to be processed based on the KLT algorithm to obtain a set of corner points; extracting straight line segments in the image to be processed based on the EDLines straight line detection algorithm to obtain an initial line segment set; the image to be processed is a virtual model image or a real-time image of the part to be assembled; Selecting line segments passing through at least one corner point in the corner point set from the initial line segment set to obtain a selected line segment set; Merging each line segment in the filtered line segment set according to a preset merging rule to obtain a merged line segment set; The feature descriptor of each line segment in the merged line segment set is calculated, and the feature descriptor set of the image to be processed is formed based on the feature descriptor of each line segment; wherein the feature descriptor of each line segment at least includes the angle and the shortest distance between each line segment and other line segments.

2. The method according to claim 1, characterized in that The preset merging rules are: Determine whether the angle between any two line segments to be merged is less than a preset angle threshold and whether the shortest distance between any two line segments to be merged is less than a preset distance threshold; If both conditions are met, they are merged.

3. A three-dimensional tracking registration method, characterized in that: The method comprises: Obtain real-time captured images of parts to be assembled; Extracting a first feature descriptor set of the real-time acquired image based on the feature descriptor extraction method according to claim 1 or 2; Performing similarity matching between the first feature descriptor and a second feature descriptor set of all template images in a template information library pre-constructed for the virtual model of the part to be assembled, and obtaining a camera pose of the template image with the greatest similarity; Registering the virtual model of the part to be assembled into the real physical device based on the acquired camera pose; When the acquisition viewing angle of the part to be assembled changes, the corner points of the real-time image of the part to be assembled are tracked based on the KLT algorithm, and the camera pose is updated based on the tracking result to perform real-time tracking registration.

4. The method according to claim 3, characterized in that The construction process of the template information library includes: Determine multiple acquisition viewpoints; Acquire an image of the virtual model of the to-be-assembled part collected at each of the collection viewpoints; Extracting a set of feature descriptors of the image of each virtual model based on the feature descriptor extraction method; The feature descriptor set of each virtual model's image and the camera pose under the acquisition viewpoint corresponding to the image are stored in the template information library as a set of data, wherein the camera pose is the camera pose in the coordinate system with the virtual model as the origin.

5. The method according to claim 3, characterized in that: Determining a plurality of acquisition viewpoints comprises: Determine the collection surface according to the type of parts to be assembled; the collection surface is a hemispherical surface or a full-spherical surface; Randomly generate a number of initial acquisition viewpoints on the acquisition plane; The positions of the initial acquisition viewpoints are corrected to obtain evenly distributed acquisition viewpoints.

6. The method according to claim 5, characterized in that The step of correcting the positions of the initial acquisition viewpoints to obtain evenly distributed acquisition viewpoints comprises: The initial acquisition viewpoints are regarded as particles having mutual forces, and an initial velocity is preset for each of the initial acquisition viewpoints to obtain an initial velocity matrix; Calculate the force exerted on each acquisition viewpoint by other acquisition viewpoints in three-dimensional space based on the initial position of the initial acquisition viewpoint; updating the initial velocity matrix based on the force; The position of each acquisition viewpoint is updated based on the updated velocity matrix until the modulus value of the velocity matrix meets a preset threshold.

7. The method according to claim 3, characterized in that When the acquisition viewing angle of the part to be assembled changes, tracking corner points of the real-time image of the part to be assembled based on the KLT algorithm, and updating the camera pose based on the tracking result include: When the acquisition angle of the parts to be assembled changes, the corner points of the images acquired before and after the acquisition angle changes are matched; If the number of matched corner points is greater than a preset threshold, the transformation matrix of the corner points is calculated based on the features of the corner points before and after the change; Update the camera pose based on the transformation matrix of the corner points; If the number of matched corner points is less than the preset threshold, the camera pose under the current acquisition perspective is re-acquired based on the similarity matching method with the template image.

8. A three-dimensional tracking and registration device, characterized in that: The device comprises: An acquisition unit, used for acquiring real-time acquisition images of the parts to be assembled; An extraction unit, configured to extract a first feature descriptor set of the real-time acquired image based on the feature descriptor extraction method according to claim 1 or 2; A matching unit, configured to perform similarity matching between the first feature descriptor and a second feature descriptor set of all template images in a template information library pre-constructed for the virtual model of the part to be assembled, and to obtain a camera pose of the template image with the greatest similarity; A registration unit, used for registering the virtual model of the part to be assembled into the real physical device based on the acquired camera pose; The tracking unit is used to track the corner points of the real-time image of the to-be-assembled part based on the KLT algorithm when the acquisition viewing angle of the to-be-assembled part changes, and to update the camera posture based on the tracking result so as to perform real-time tracking registration.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images