Target reconstruction method, target reconstruction system and computer readable storage medium
By projecting identification in large workpiece areas and combining 2D images and point cloud features, the positioning pose transformation relationship is determined, and the problem of point cloud loss caused by viewing angle limitation and occlusion in large workpiece reconstruction is solved, achieving high-precision target reconstruction.
Patent Information
- Application Number
- CN202510287516.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-11
- Publication Date
- 2025-07-22
AI Technical Summary
It is difficult for the prior art to obtain high-precision 3D point cloud information in the industrial processing of large workpieces, especially due to the lack of point clouds caused by camera viewing angle limitations, workpiece geometric structure occlusion and material reflection. Traditional splicing technology cannot build a complete workpiece point cloud.
After the target area is projected, 2D images and 3D images at different locations are obtained, the positioning transformation relationship is determined through feature extraction and matching, and the target reconstruction is carried out in combination with 2D images and point clouds, and the identification features are used to improve the accuracy of the positioning transformation relationship.
Improve the accuracy of target reconstruction and enable the construction of high-precision 3D point clouds on large workpieces lacking geometric structures, enhancing the robustness of reconstruction.
Smart Images
Figure CN120355567A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of object reconstruction, and particularly to an object reconstruction method, an object reconstruction system, and a computer-readable storage medium. Background Art
[0002] In the industrial processing fields of large workpieces such as gluing and welding, obtaining the complete 3D information of large workpieces is crucial for further improving the automation and intelligence of related tasks. The industry usually uses 3D cameras to obtain the 3D point cloud information of workpieces. In order to obtain a point cloud with high spatial resolution, the camera is usually close to the workpiece. At this time, due to the limitation of the camera's viewing angle, the complete workpiece cannot be photographed. At the same time, due to the geometric structure of the workpiece, occlusion is likely to occur in a single view, resulting in the missing of some parts in the point cloud. In addition, factors such as reflection and light absorption caused by the workpiece material will also interfere with the point cloud imaging quality in some views. Therefore, it is necessary to photograph the workpiece from different angles, and then stitch the point clouds at different photographing positions to obtain the 3D point cloud of the complete large workpiece.
[0003] In reality, some large workpieces such as flat plates and cylinders have problems such as simple geometric information or lack of geometric structure. Traditional point cloud-based stitching techniques cannot accurately construct the complete point cloud of large workpieces. Due to the existence of calibration errors, some prior pose-based stitching techniques cannot obtain high-precision reconstruction results. Summary of the Invention
[0004] The object reconstruction method, object reconstruction system, and computer-readable storage medium provided by this application can improve the accuracy of object reconstruction.
[0005] In a first aspect, this application provides an object reconstruction method, which includes: after projecting identifiers in the area where the object is located, obtaining 2D images and 3D images corresponding to the object collected at different positions; converting the 3D images into point clouds; determining the pose transformation relationship between every two positions based on the 2D images and the point clouds; and performing object reconstruction based on the pose transformation relationship and the point clouds at different positions.
[0006] Among them, determining the pose transformation relationship between every two positions based on the 2D images and the point clouds includes: extracting the identifiers in each 2D image; for each identifier, determining the target point cloud corresponding to the identifier from the point cloud corresponding to the 2D image; performing feature extraction on the target point cloud to obtain the feature vector corresponding to the target point cloud; where the feature vector includes the distances between the target point cloud and K nearest neighbor point clouds and the angles between adjacent nearest neighbor point clouds; and determining the pose transformation relationship between every two positions by using the feature vectors corresponding to the 2D images.
[0007] Among them, the pose transformation relationship between every two positions is determined by using the feature vectors corresponding to the 2D images, including: comparing the distance relationship between the feature vectors corresponding to the 2D images of the two positions; in response to the distance relationship satisfying a preset condition, the feature vectors corresponding to the 2D images of the two positions are matched to obtain a first matching relationship; and the pose transformation relationship between every two positions is determined according to the first matching relationship.
[0008] Among them, determining the pose transformation relationship between every two positions according to the first matching relationship includes: performing a first registration on the first matching relationship to obtain an initial transformation relationship between every two positions; extracting corresponding point cloud clusters according to the first matching relationship, and performing a second registration according to the point cloud clusters and the initial transformation relationship to obtain the pose transformation relationship between every two positions.
[0009] Among them, feature extraction is performed on the target point cloud to obtain the feature vector corresponding to the target point cloud, including: obtaining the point cloud center of the target point cloud; performing feature extraction on the point cloud center to obtain the feature vector corresponding to the point cloud center; where the feature vector includes the distances between the point cloud center and the centers of the K nearest neighbor point clouds and the angles between adjacent nearest neighbor point cloud centers.
[0010] Among them, determining the pose transformation relationship between every two positions according to the 2D images and the point cloud includes: performing feature point matching on the 2D images of every two positions; determining corresponding key point pairs from the point clouds of the 2D images of the two positions according to the matching relationship; and determining the pose transformation relationship between every two positions according to the key point pairs.
[0011] Among them, determining the pose transformation relationship between every two positions according to the key point pairs includes: determining the first neighborhood point cloud corresponding to the first key point in the key point pair, and determining the second neighborhood point cloud corresponding to the second key point in the key point pair; performing a first registration on the key points to obtain an initial transformation relationship between every two positions; and performing a second registration according to the initial transformation relationship, the first neighborhood point cloud, and the second neighborhood point cloud to obtain the pose transformation relationship between every two positions.
[0012] Among them, target reconstruction is performed according to the pose transformation relationship and the point clouds at different positions, including: determining a target position from different positions as the pose starting node; obtaining the best estimated pose of each position relative to the pose starting node according to the pose starting node and the pose transformation relationship; transforming the point cloud corresponding to each position according to its corresponding best estimated pose, and splicing all the pose-transformed point clouds to complete the target reconstruction.
[0013] In a second aspect, the present application provides a target reconstruction system, which includes: an identification projection component for projecting an identification onto the area where the target is located; an image acquisition component for acquiring the 2D image and 3D image corresponding to the target at different positions after projecting the identification onto the area where the target is located; a processing component connected to the image acquisition component for receiving the 2D image and 3D image, converting the 3D image into a point cloud, determining the pose transformation relationship between every two positions based on the 2D image and the point cloud, and performing target reconstruction based on the pose transformation relationship and the point cloud at different positions.
[0014] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program, which when executed by a processor is used to implement the target reconstruction method provided in the first aspect.
[0015] The beneficial effects of the present application are as follows: Different from the prior art, the target reconstruction method, target reconstruction system, and computer-readable storage medium provided by the present application acquire the 2D image and 3D image corresponding to the target collected at different positions after projecting the identification onto the area where the target is located, convert the 3D image into a point cloud, then determine the pose transformation relationship between every two positions based on the 2D image with the identification and the point cloud, and perform target reconstruction based on the pose transformation relationship and the point cloud at different positions. That is, based on the point cloud, the present application combines the identification features in the 2D image to construct the pose transformation relationship between different positions, thereby improving the accuracy of the pose transformation relationship and further improving the accuracy of target reconstruction. Description of the Drawings
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. Among them:
[0017] Figure 1 is a schematic flowchart of an embodiment of the target reconstruction method provided by the present application;
[0018] Figure 2 is a schematic flowchart of another embodiment of the target reconstruction method provided by the present application;
[0019] Figure 3 is Figure 2 a schematic flowchart of an embodiment of step 26 in
[0020] Figure 4 is Figure 3 a schematic flowchart of an embodiment of step 263 in
[0021] Figure 5 It is a schematic diagram showing the relationship between the point cloud center and the neighborhood point cloud center provided by this application;
[0022] Figure 6 It is a schematic flowchart of another embodiment of the target reconstruction method provided by this application;
[0023] Figure 7 It is Figure 6 a schematic flowchart of an embodiment of step 65 in
[0024] Figure 8 It is a schematic structural diagram of an embodiment of the target reconstruction system provided by this application;
[0025] Figure 9 It is a schematic structural diagram of an embodiment of the computer-readable storage medium provided by this application. Detailed implementation manners
[0026] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. It can be understood that the specific embodiments described herein are only used to explain this application, rather than limiting this application. Additionally, it should be noted that for the sake of description, only parts related to this application rather than all structures are shown in the accompanying drawings. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.
[0027] Referring to "embodiment" herein means that the specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment of this application. The phrase appears at various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.
[0028] In the industrial processing fields of large workpieces such as gluing and welding, obtaining the complete 3D information of large workpieces is crucial for further improving the automation and intelligence of related tasks. The industry usually uses 3D cameras to obtain the 3D point cloud information of workpieces. In order to obtain a point cloud with high spatial resolution, the camera is usually placed relatively close to the workpiece. At this time, due to the limitation of the camera's viewing angle, the complete workpiece cannot be photographed. At the same time, due to the geometric structure of the workpiece, occlusion is likely to occur from a single perspective, resulting in the absence of some parts in the point cloud. In addition, factors such as reflection and light absorption caused by the workpiece material will also interfere with the point cloud imaging quality in certain perspectives. Therefore, it is necessary to photograph the workpiece from different angles and then stitch the point clouds at different photographing positions to obtain the complete 3D point cloud of the large workpiece.
[0029] In reality, some large workpieces such as flat plates and cylinders have problems such as simple geometric information or lack of geometric structure. Traditional point cloud-based stitching techniques cannot accurately construct the complete point cloud of large workpieces. Due to the existence of calibration errors, some stitching techniques based on prior poses also cannot obtain high-precision reconstruction results.
[0030] Based on this, this application proposes to project an identifier in the area where the target is located, and then obtain the 2D images and 3D images of the target collected at different positions; convert the 3D images into point clouds; then determine the pose transformation relationship between every two positions based on the 2D images with identifiers and the point clouds; perform target reconstruction based on the pose transformation relationship and the point clouds at different positions. That is, based on the point cloud, this application combines the identifier features in the 2D image to construct the pose transformation relationship between different positions, thereby improving the accuracy of the pose transformation relationship and further improving the accuracy of target reconstruction. For the specific technical solutions, please refer to any of the following embodiments.
[0031] Refer to Figure 1 , Figure 1 FIG.
[0032] Step 11: Project an identifier in the area where the target is located, and then obtain the 2D images and 3D images of the target collected at different positions.
[0033] In some embodiments, the target can be an item such as a workpiece to be three-dimensionally reconstructed. The material of the workpiece can be metal, plastic, etc.
[0034] In some embodiments, the identifier can be some recognizable features. For example, speckles, etc. A speckle projection component can be used to project speckles onto the area where the target is located, so that there are speckles on the surface of the target. Then, when collecting the 2D image of the target, the pixel features corresponding to the speckles will also exist in the image. That is, after projecting the identifier in the area where the target is located, the pixel features corresponding to the identifier exist in the 2D images of the target collected at different positions.
[0035] In some embodiments, the 2D image and the 3D image can be collected by the same image acquisition component. For example, an image acquisition component with depth acquisition function is used to collect the 2D image and the 3D image of the target. That is, the 2D image and the 3D image of the target are collected at different positions by the same image acquisition component.
[0036] In some embodiments, the image acquisition component may be disposed on a moving mechanism. By adjusting the position of the moving mechanism, the image acquisition component can be moved to different positions, and then 2D images and 3D images corresponding to the target can be acquired at different positions. That is, the number of positions where the image acquisition component stays corresponds to the corresponding number of 2D images and 3D images. For example, if the image acquisition component acquires images at 5 positions, there are 5 2D images and 5 3D images at different positions.
[0037] In some embodiments, the 2D images and 3D images acquired at different positions often have different acquisition postures, such as different acquisition angles.
[0038] Step 12: Convert the 3D image into a point cloud.
[0039] In some embodiments, since the 3D image has depth information, the 3D image can be converted into a point cloud at this position according to the internal parameters of the image acquisition component. That is, the 3D image at each position can be converted into a point cloud at the corresponding position.
[0040] Step 13: Determine the pose transformation relationship between every two positions based on the 2D image and the point cloud.
[0041] In some embodiments, since each position corresponds to a corresponding 2D image and 3D image, there is an association relationship between the 2D image and the 3D image. Based on this, there is also an association relationship between the 2D image and the point cloud of the 3D image at the same position, and the pose transformation relationship between every two positions can be determined based on the 2D image and the point cloud.
[0042] In some embodiments, in order to reduce the calculation, it is not necessary to determine the pose transformation relationship between every two positions. Instead, it is necessary to determine the pose transformation relationship between adjacent positions in a closed-loop form according to the acquisition order of the 2D images. For example, the 2D image A1 and the point cloud A2 corresponding to position 1, the 2D image B1 and the point cloud B2 corresponding to position 2, and the 2D image C1 and the point cloud C2 corresponding to position 3. Then, the pose transformation relationship between position 1 and position 2 can be determined according to the 2D image A1 and the point cloud A2 corresponding to position 1, and the 2D image B1 and the point cloud B2 corresponding to position 2. The pose transformation relationship between position 2 and position 3 can be determined according to the 2D image B1 and the point cloud B2 corresponding to position 2, and the 2D image C1 and the point cloud C2 corresponding to position 3. The pose transformation relationship between position 1 and position 3 can be determined according to the 2D image A1 and the point cloud A2 corresponding to position 1, and the 2D image C1 and the point cloud C2 corresponding to position 3.
[0043] Step 14: Perform target reconstruction based on the pose transformation relationship and the point clouds at different positions.
[0044] In some embodiments, after obtaining the pose transformation relationship, corresponding pose optimization is performed on it, and then target reconstruction is carried out.
[0045] Pose graph optimization can be achieved by minimizing the cumulative error existing in continuous observations. For example, the pose transformation relationship is optimized to minimize the transformation relationship error between different observations.
[0046] First, taking the photographing positions P1, P2, and P3 as examples, the photographing position P1 is used as the starting node of the pose graph, and its pose is the identity matrix Then the pose of the node of the photographing position P2 relative to the photographing position P1 is The pose of the node of the photographing position P3 relative to the photographing position P1 is After adding the nodes, the transfer matrix corresponding to the photographing position is added as an edge. Then, by performing optimization, the best estimated pose of each photographing position relative to the first node can be obtained.
[0047] Then, the point cloud corresponding to each photographing position is transformed according to the best estimated pose of each photographing position, and all the point clouds corresponding to the photographing positions can be uniformly transformed into the coordinate system of the photographing position P1, completing the stitching of the point clouds and realizing target reconstruction.
[0048] In this embodiment, after projecting the identifier in the area where the target is located, 2D images and 3D images corresponding to the target collected at different positions are obtained; the 3D images are converted into point clouds; then, the pose transformation relationship between every two positions is determined based on the 2D images with identifiers and the point clouds; target reconstruction is carried out based on the pose transformation relationship and the point clouds at different positions, that is, in this application, based on the point clouds, combined with the identifier features in the 2D images, the pose transformation relationship between different positions is constructed, so as to improve the accuracy of the pose transformation relationship and further improve the accuracy of target reconstruction.
[0049] Refer to Figure 2 , Figure 2 which is a schematic flowchart of another embodiment of the target reconstruction method provided by this application. The method includes:
[0050] Step 21: After projecting the identifier in the area where the target is located, obtain 2D images and 3D images corresponding to the target collected at different positions.
[0051] Step 22: Convert the 3D images into point clouds.
[0052] In some embodiments, steps 21 to 22 have the same or similar technical solutions as any embodiment of this application, and will not be elaborated here.
[0053] Step 23: Extract the identifiers in each 2D image.
[0054] In some embodiments, the number of projected identifiers can be determined according to the actual situation, such as 10, 20, 30, etc.
[0055] In some embodiments, since the identifiers are usually preset, the identifier regions in the 2D image can be collected, and the features of the identifier regions are compared with the preset identifiers for similarity. The identifier regions with similarity greater than the threshold are used as the identified identifier regions. Thus, several identifiers can be extracted from each 2D image. Among them, each identifier has corresponding image coordinates.
[0056] Step 24: For each identifier, determine the target point cloud corresponding to the identifier from the point cloud corresponding to the 2D image.
[0057] In some embodiments, after several identifiers are extracted from the 2D image, since the 2D image corresponds to a corresponding point cloud, the target point cloud corresponding to the identifier is determined from the corresponding point cloud. It can be understood that an identifier usually occupies several pixels, so it is a region, and thus the identifier corresponds to a block of point cloud rather than a single point.
[0058] Since several identifiers can be extracted from each 2D image, there are corresponding target point clouds for the corresponding several identifiers in their corresponding point clouds. For example, if 10 identifiers are extracted from each 2D image, there are also corresponding 10 clusters of target point clouds in their corresponding point clouds.
[0059] Step 25: Extract features from the target point cloud to obtain the feature vector corresponding to the target point cloud; wherein, the feature vector includes the distances between the target point cloud and the K nearest neighbor point clouds and the angles between adjacent nearest neighbor point clouds.
[0060] In some embodiments, considering the complexity of the target point cloud, the center of the point cloud of the target point cloud can be obtained, and the target point cloud is characterized by the center of the point cloud. Then, features are extracted from the center of the point cloud to obtain the feature vector corresponding to the center of the point cloud. The feature vector includes the distances between the center of the point cloud and the centers of the K nearest neighbor point clouds and the angles between adjacent nearest neighbor point cloud centers.
[0061] In some embodiments, since there are several target point clouds, there will be several centers of the point cloud. Therefore, using the K nearest neighbor method, K nearest neighbor point cloud centers are determined for each center of the point cloud from several centers of the point cloud. After determining the K nearest neighbor point cloud centers, the distances between the current center of the point cloud and the K nearest neighbor point cloud centers and the angles between adjacent nearest neighbor point cloud centers are retained.
[0062] Step 26: Determine the pose transformation relationship between each two positions using the feature vector corresponding to the 2D image.
[0063] In some embodiments, refer toFigure 3 , Step 26 may be the following process:
[0064] Step 261: Compare the distance relationship between the feature vectors corresponding to the 2D images at two positions.
[0065] In some embodiments, each feature vector corresponding to the 2D image at a position includes K nearest neighbor point cloud centers, and the K nearest neighbor point cloud centers can be arranged in ascending or descending order of distance in the feature vector. Then compare them in sequence.
[0066] In an application scenario, the feature vector corresponding to the 2D image at position 1 is [a1, b1, c1], where a1, b1, c1 represent 3 nearest neighbor point cloud centers. The feature vector corresponding to the 2D image at position 2 is [a2, b2, c2], where a2, b2, c2 represent 3 nearest neighbor point cloud centers. Then the distance relationship between a1 and a2, the distance relationship between b1 and b2, and the distance relationship between c1 and c2 can be compared. For example, the distance relationship can be the distance between two nearest neighbor point cloud centers.
[0067] Step 262: In response to the distance relationship satisfying a preset condition, the feature vectors corresponding to the 2D images at two positions are matched to obtain a first matching relationship.
[0068] In some embodiments, the preset condition may be that a continuous plurality of distance relationships are less than a preset distance value. For example, for K nearest neighbor point cloud centers, it may be that K continuous distance relationships are less than the preset distance value. In some embodiments, a plurality may be a proportional number of K. This proportion is greater than one-half and less than or equal to 1. For example, if K is 10, the preset condition may be that 6, 7, 8, 9, 10 continuous distance relationships are less than the preset distance value.
[0069] In some embodiments, since the feature vector also includes angle information, in addition to limiting that a continuous plurality of distance relationships are less than a preset distance value in the preset condition, it can also limit that a continuous plurality of angle errors are less than a preset error.
[0070] In an application scenario, the feature vector corresponding to the 2D image at position 1 is [a1, b1, c1, d1, e1, f1], where a1, b1, c1 represent the centers of 3 nearest neighbor point clouds, and d1, e1, f1 represent the angles between adjacent nearest neighbor point cloud centers, that is, d1 represents the angle between a1 and b1, e1 represents the angle between b1 and c1, and f1 represents the angle between a1 and c1. The feature vector corresponding to the 2D image at position 2 is [a2, b2, c2, d2, e2, f2], where a2, b2, c2 represent the centers of 3 nearest neighbor point clouds, and d2, e2, f2 represent the angles between adjacent nearest neighbor point cloud centers, that is, d2 represents the angle between a2 and b2, e2 represents the angle between b2 and c2, and f2 represents the angle between a2 and c2. Then, the distance relationship between a1 and a2, the distance relationship between b1 and b2, and the distance relationship between c1 and c2 can be compared. For example, the distance relationship can be the distance between two nearest neighbor point cloud centers. And the distance relationship between d1 and d2, the distance relationship between e1 and e2, and the angular error between f1 and f2 can be compared. In response to a continuous plurality of distance relationships being less than a preset distance value and a continuous plurality of angular errors being less than a preset error, the feature vectors corresponding to the 2D images at the two positions match, obtaining a first matching relationship.
[0071] That is, in this way, a plurality of first matching relationships can be obtained from the 2D images between several positions, and the plurality of first matching relationships can form a matching set.
[0072] Step 263: Determine the pose transformation relationship between each two positions according to the first matching relationship.
[0073] In some embodiments, referring to Figure 4 , step 263 may be the following process:
[0074] Step 2631: Perform the first registration on the first matching relationship to obtain the initial transformation relationship between each two positions.
[0075] In some embodiments, the MAC algorithm can be used to perform rough registration on the matching set to obtain the initial transformation relationship between the images or objects or point clouds of the two positions after rough registration. That is, after the point cloud corresponding to the first position is transformed by the initial transformation relationship, it will be in the same coordinate system as the point cloud corresponding to the second position.
[0076] The MAC algorithm (Maximal Clique Algorithm) is a 3D registration method based on maximal clique. Its core idea is to construct a compatibility graph to represent the affinity relationship between initial correspondences, then search for maximal cliques in the graph, and each clique represents a consensus set. Pose transformation hypotheses are calculated through node-guided clique selection and the Singular Value Decomposition (SVD) algorithm, and finally registration is performed using the best hypothesis.
[0077] The specific steps of the MAC algorithm are as follows:
[0078] Construct a compatibility graph: Represent the affinity relationship between initial correspondences.
[0079] Search for maximal cliques: Mine more local consensus information in the graph, and each maximal clique represents a consensus set.
[0080] Node-guided clique selection: Select the maximal clique with the maximum graph weight.
[0081] Calculate pose transformation hypotheses: Use the SVD algorithm to calculate pose transformation hypotheses for the selected clique.
[0082] Registration: Perform registration using the best hypothesis.
[0083] Step 2632: Extract the corresponding point cloud clusters according to the first matching relationship, and perform a second registration based on the point cloud clusters and the initial transformation relationship to obtain the pose transformation relationship between every two positions.
[0084] In some embodiments, according to the first matching relationship, extract the corresponding point cloud clusters, and then use the initial transformation relationship as the initial pose of ICP to perform a fine-tuning of pose estimation (second registration) to obtain the pose transformation relationship. Thus, an accurate pose transformation relationship between two positions is obtained.
[0085] That is, the point clouds at different positions can be processed in the above manner to obtain an accurate pose transformation relationship between two positions.
[0086] It can be understood that in practical applications, when the target size is large, there may be no common viewing area between two scans, and it is not possible to obtain a pose transformation relationship between any two frames.
[0087] Step 27: Perform target reconstruction based on the pose transformation relationship and the point clouds at different positions.
[0088] Step 27 has the same or similar technical solutions as any embodiment of this application, and will not be elaborated here.
[0089] In this embodiment, after projecting the marker in the target area, the 2D image and 3D image corresponding to the target collected at different positions are obtained; the 3D image is converted into a point cloud; then, the point cloud of the 3D image corresponding to the marker in each 2D image is corresponded to obtain several point clouds corresponding to the markers, and then the features of these point clouds are extracted to obtain the corresponding feature vectors, and then the feature vectors corresponding to the images at different positions are matched, and the posture transformation relationship between each two positions is determined by using the relationship between the matched feature vectors; the target is reconstructed according to the posture transformation relationship and the point clouds at different positions, that is, the present application, based on the point cloud, combines the marker features in the 2D image to construct the posture transformation relationship between different positions, so as to improve the accuracy of the posture transformation relationship, and then improve the accuracy of the target reconstruction.
[0090] In one application scenario, a laser projector is used to project speckles (markers) in the working area (the area where the target is located), and then a 3D camera is used to take pictures at different positions P1, .., P n Take pictures, and at each shooting position, obtain the RGB image (2D image) of the speckle at the current position, denoted as C n , and the 3D depth map (3D image) of the position, and convert the 3D depth map into the point cloud of the position according to the intrinsic parameters of the camera, recorded as D n The point cloud D and the RGB image C correspond to each other pixel by pixel. That is, if the coordinates of a pixel in an RGB image C1 are known to be i rows and j columns, then the pixel The 3D information of the corresponding point is
[0091] For each RGB image, first extract all the marker points (such as scattered spots) in the image. In order to improve the success rate of marker extraction, perform image processing operations such as filtering on the image. Assume that for image C i After extracting the markers, all markers are recorded as The superscript N indicates the Nth marker extracted. Indicates the second marker extracted from image C1. Each marker is a small image block of a certain size.
[0092] For each marker, extract the corresponding point cloud from the corresponding point cloud according to its coordinates. Then extract the center of these point clouds as the center of the marker and record it as In this way, the point cloud clusters corresponding to a series of markers in the image will be simplified and represented as the center of the marker.
[0093] Then, feature extraction is performed on the center of each marker. First, for the center of each marker, find the K other marker centers closest to it. As described in Figure 5 the first marker center corresponding to the image at the photographing position P1 is and its four nearest neighbor markers are found to be Then calculate the distances to each of the nearest neighbors, and calculate the angles between adjacent pairs of nearest neighbors and
[0094] Then the marker center can be represented by a set of feature vectors as where dist1 represents the distance between and dist2 represents the distance between and dist3 represents the distance between and dist4 represents the distance between and ang1 represents the angle between and ang2 represents the angle between and ang3 represents the angle between and ang4 represents the angle between and Such feature extraction is performed on the center of each marker. Then, all N marker centers in the image corresponding to the photographing position P1 can be represented as a two-dimensional array of where each row is the feature vector corresponding to a marker.
[0095] The above-described feature extraction operation is performed on the image corresponding to each photographing position. Taking the three photographing positions P1, P2, and P3 as an example, three two-dimensional arrays can be obtained, which respectively store the features of all detected markers in the images of the three photographing positions. Then, based on pairwise matching of the two-dimensional arrays, check whether there are co-visible markers in any two pairs of images. For example, perform feature matching on the images of the photographing position P1 and the photographing position P2, the images of the photographing position P1 and the photographing position P3, and the images of the photographing position P2 and the photographing position P3 respectively. Taking the matching between the images of the photographing position P1 and the photographing position P2 as an example, the matching strategy is described in detail.
[0096] The features extracted from all marker centers in the image of the photographing position P1 are The features extracted from the centers of all markers in the image at the photographing position P2 are For the i markers in the image at the photographing position P1, calculate their similarity relationships with each marker in the image at the photographing position P2. When the distance relationship between the jth marker and the ith marker satisfies a preset condition, it is considered that the ith marker in the image at the photographing position P1 and the jth marker in the image at the photographing position P2 are matched. Among them, for the distance part of the feature vectors, there are the following rules:
[0097] When the difference in the distances between the ith marker in the image at the photographing position P1 and the jth marker in the image at the photographing position P2 has k consecutive values less than a preset error, and at the same time, z consecutive angular errors are less than the preset error, then it is considered that They are matched. Performing such an operation on each marker in the image at the photographing position P1 can obtain a set of matching relationships, denoted as the match set
[0098] Then use the MAC algorithm to perform a rough registration on the matching set MS 1-2 to obtain the transformation relationship between the roughly registered images at the photographing positions P1 and P2 where the subscript 12 indicates that the transformation is in the order from the photographing position P1 to the scanning photographing position P2, and the superscript r represents rough alignment. That is, after the point cloud of the image at the photographing position P1 undergoes the transformation, it will be in the same coordinate system as the point cloud of the image at the photographing position P2.
[0099] Furthermore, according to the matching relationship of MS 1-2 extract the corresponding point cloud clusters, and then use as the initial pose of ICP to perform a fine adjustment of the pose estimation to obtain where the meaning of the superscript f is fine tuning. Thus, the precise pose transformation relationship between the images at the photographing positions P1 and P2 is obtained.
[0100] Repeating the above process for all pairs of photographing position images can obtain the pose transformation relationships between pairs. For example, the set of pose transformations corresponding to the images of the three photographing positions P1, P2, and P3 is denoted as In practical applications, when the size of the workpiece (target) is large, there may be no common viewing area between two frames of images, and it is not possible to obtain a pose transformation relationship between any two frames.
[0101] After obtaining the pose transformation relationship, perform corresponding pose optimization, and then perform target reconstruction.
[0102] The pose graph optimization can be achieved by minimizing the cumulative error existing in continuous observations. For example, optimize the pose transformation relationship to minimize the error of the transformation relationship between different observations.
[0103] First, take the photographing position P1 as the starting node of the pose graph, and its pose is the identity matrix. Then the pose of the node of the photographing position P2 relative to the photographing position P1 is The pose of the node of the photographing position P3 relative to the photographing position P1 is After adding the nodes, the transfer matrix corresponding to the photographing position is added as an edge. Then, by performing optimization, the optimal estimated pose of each photographing position relative to the first node can be obtained.
[0104] Then, the point cloud corresponding to each photographing position is transformed according to the optimal estimated pose of each photographing position, so that the point clouds corresponding to all photographing positions can be uniformly transformed into the coordinate system of the photographing position P1, and the stitching of the point clouds is completed.
[0105] Refer to Figure 6 , Figure 6 which is a schematic flowchart of another embodiment of the target reconstruction method provided by this application. The method includes:
[0106] Step 61: After projecting the identifier in the area where the target is located, obtain the 2D image and 3D image corresponding to the target collected at different positions.
[0107] Step 62: Convert the 3D image into a point cloud.
[0108] In some embodiments, steps 61 to 62 have the same or similar technical solutions as any embodiment of this application, and will not be elaborated here.
[0109] Step 63: Perform feature point matching on the 2D images at every two positions.
[0110] In some embodiments, the feature points of the 2D images at two positions can be extracted respectively to obtain the first feature point set of the 2D image at the first position and the second feature point set of the 2D image at the second position. Then, perform feature point matching between the first feature point set and the second feature point set. That is, let the feature points in the first feature point set be matched with the feature points in the second feature point set.
[0111] In some embodiments, the SuperPoint+LightGlue algorithm can be used to perform feature point matching on the 2D images at every two positions.
[0112] SuperPoint is a deep learning-based method for feature point detection and descriptor extraction. Through self-supervised learning training, it can stably extract key points in various complex environments and generate feature descriptors with strong robustness for accurate matching between images. SuperPoint is particularly suitable for alignment in image stitching, ensuring a high matching rate under different lighting and perspective changes.
[0113] LightGlue is a flexible image registration library mainly used for optimizing image geometric transformation and stitching. It can calculate the best image fusion strategy based on the feature point information provided by SuperPoint, thus achieving smooth transition and eliminating discontinuities in the overlapping area.
[0114] Step 64: Determine the corresponding key point pairs from the point clouds of the 2D images at two positions according to the matching relationship.
[0115] In some embodiments, since the 2D image corresponds to a corresponding point cloud, the feature points also substantially correspond to the points in the point cloud. Based on this, the corresponding key point pairs can be determined from the point clouds corresponding to the 2D images at two positions according to the matching relationship. Among them, the first key point in the key point pair is a point in the point cloud at one position, and the second key point in the key point pair is a point in the point cloud at another position.
[0116] Step 65: Determine the pose transformation relationship between each two positions according to the key point pairs.
[0117] In some embodiments, referring to Figure 7 , Step 65 can be the following process:
[0118] Step 651: Determine the corresponding first neighborhood point cloud according to the first key point in the key point pair, and determine the corresponding second neighborhood point cloud according to the second key point in the key point pair.
[0119] In some embodiments, after determining the first key point, the corresponding first neighborhood point cloud can be determined according to the first key point. For example, first find the neighborhood area of the feature point corresponding to the first key point, and then determine the first neighborhood point cloud corresponding to the first key point from the corresponding point cloud according to the neighborhood area.
[0120] In some embodiments, after determining the second key point, the corresponding second neighborhood point cloud can be determined according to the second key point. For example, first find the neighborhood area of the feature point corresponding to the second key point, and then determine the second neighborhood point cloud corresponding to the second key point from the corresponding point cloud according to the neighborhood area.
[0121] Step 652: Perform the first registration on the key points to obtain the initial transformation relationship between each two positions.
[0122] In some embodiments, the MAC algorithm can be used to register these key points (the first registration) to obtain the initial transformation relationship between each two positions.
[0123] Step 653: Perform the second registration based on the initial transformation relationship, the first neighborhood point cloud, and the second neighborhood point cloud to obtain the pose transformation relationship between each two positions.
[0124] In some embodiments, the ICP algorithm is used. Taking the initial transformation relationship as the initial value of ICP, the first neighborhood point cloud and the second neighborhood point cloud are finely adjusted (the second registration) to obtain the finely adjusted transformation relationship (the pose transformation relationship).
[0125] Step 66: Perform target reconstruction based on the pose transformation relationship and the point clouds at different positions.
[0126] Step 66 has the same or similar technical solutions as any embodiment of the present application, and will not be elaborated here.
[0127] In this embodiment, after projecting the identifier in the area where the target is located, 2D images and 3D images corresponding to the target collected at different positions are obtained; the 3D images are converted into point clouds; then the feature points between two 2D images are matched, and according to the feature point matching relationship, the corresponding key point pairs are determined from the point clouds of the 2D images at two positions, and the pose transformation relationship between each two positions is determined by using the neighborhood point clouds corresponding to the key point pairs, so as to improve the accuracy of the pose transformation relationship and further improve the accuracy of target reconstruction.
[0128] In an application scenario, a laser projector is used to project speckles (identifiers) in the working area (the area where the target is located). Secondly, a 3D camera is used to take pictures at different photographing positions P1,..,P n For each photographing position, the RGB image (2D image) of the speckles at the current position is obtained and denoted as C n , and the 3D depth map (3D image) at this position, and the 3D depth map is converted into the point cloud at this position according to the internal parameters of the camera and denoted as D n . The point cloud D and the RGB image C correspond pixel by pixel. That is, if the coordinates of a certain pixel point in a RGB image C1 are known to be in the i-th row and j-th column, then the 3D information of the corresponding point of this pixel is
[0129] Taking the photographing positions P1 and P2 as an example for illustration:
[0130] Perform registration on (Image 1, Point Cloud 1) at the photographing position P1 and (Image 2, Point Cloud 2) at the photographing position P2. First, use the SuperPoint+LightGlue algorithm to extract and match feature points from Image 1 and Image 2. Obtain the correct matching points on the images. Then, select the corresponding point clouds from Point Cloud 1 and Point Cloud 2 according to the correct matching positions on Image 1 and Image 2. Suppose they are recorded as [kpt1-1(100, 97), kpt2-1(200, 244)]. This represents a key point in Image 1, and its coordinates in Image 1 are 100 rows and 97 columns. It corresponds to a point at 200 rows and 244 columns in Image 2. The 3D position correspondence can be obtained from Point Cloud 1 and Point Cloud 2 according to the row and column. For example, the coordinates of the point at 100 rows and 97 columns in Point Cloud 1 are (0.1, 0.35, 1.1), corresponding to the point (2.12, 3.2, 2.11) at 200 rows and 244 columns in Point Cloud 2. If 100 key points in Image 1 are matched with the key points in Image 2, then 100 corresponding three-dimensional coordinate pairs can be obtained from Point Cloud 1 and Point Cloud 2.
[0131] In addition, because of the correspondence of the key points, the neighborhoods around the key points can be extracted. Taking the above-mentioned point pair [kpt1-1(100, 97), kpt2-1(200, 244)] as an example, the 5*5 neighborhood of kpt1-1(100, 97) is:
[0132]
[0133] Obtain the point cloud corresponding to this area, that is, obtain the point cloud corresponding to the area enclosed by the coordinates (98, 95), (98, 97), (98, 99), (102, 95), (102, 97), (102, 99). Similarly, the point clouds of the 5*5 areas around each key point can be obtained.
[0134] In this way, the three-dimensional point pairs with matching relationships between the photographing position P1 and the photographing position P2, and the point clouds of the neighborhoods around each point pair are obtained. Then, the MAC algorithm can be used to register these point pairs to obtain the transformation relationship between the photographing position P1 and the photographing position P2, such as recorded as [r12, t12]. That is, Point Cloud 2 can be transformed from the coordinate system of Point Cloud 2 to the coordinate system of Point Cloud 1 through this set of transformation relationships.
[0135] Then, use the ICP algorithm, take [r12, t12] as the initial value of ICP, and finely adjust the point clouds of the neighborhoods around each key point obtained above. Obtain the finely adjusted pose transformation relationship.
[0136] After obtaining the pose transformation relationship, perform corresponding pose optimization, and then perform target reconstruction.
[0137] The pose graph optimization can be achieved by minimizing the cumulative error existing in continuous observations. For example, optimize the pose transformation relationship to minimize the error of the transformation relationship between different observations.
[0138] First, take the photographing position P1 as the starting node of the pose graph, and its pose is the identity matrix Then the pose of the node of the photographing position P2 relative to the photographing position P1 is The pose of the node of the photographing position P3 relative to the photographing position P1 is After adding the node, take the transfer matrix corresponding to the photographing position as an edge to add. Then perform optimization to obtain the best estimated pose of each photographing position relative to the first node.
[0139] Then transform the point cloud corresponding to each photographing position according to the best estimated pose of each photographing position, and all the point clouds corresponding to the photographing positions can be uniformly transformed into the coordinate system of the photographing position P1 to complete the stitching of the point clouds.
[0140] Refer to Figure 8 , Figure 8 FIG. is a schematic structural diagram of an embodiment of the target reconstruction system provided by the present application. The target reconstruction system 100 includes: an identification projection component 10, an image acquisition component 20, and a processing component 30.
[0141] The identification projection component 10 is used to project an identification to the area where the target is located. In some embodiments, the identification projection component 10 may be a laser projection component, which can project speckles to the area where the target is located. That is, the speckles can be used as an identification.
[0142] The image acquisition component 20 is used to collect the 2D image and 3D image corresponding to the target at different positions after projecting the identification to the area where the target is located. The image acquisition component 20 may be a device with a depth information acquisition function such as a 3D camera.
[0143] The processing component 30 is connected to the image acquisition component 20, and is used to receive the 2D image and 3D image, and convert the 3D image into a point cloud; determine the pose transformation relationship between every two positions according to the 2D image and the point cloud; and perform target reconstruction according to the pose transformation relationship and the point cloud at different positions.
[0144] In some embodiments, the processing component 30 is further used to extract the identification in each 2D image; for each identification, determine the target point cloud corresponding to the identification from the point cloud corresponding to the 2D image; perform feature extraction on the target point cloud to obtain the feature vector corresponding to the target point cloud; wherein, the feature vector includes the distance between the target point cloud and the K nearest neighbor point clouds and the angle between adjacent nearest neighbor point clouds; use the feature vector corresponding to the 2D image to determine the pose transformation relationship between every two positions.
[0145] In some embodiments, the processing component 30 is further configured to compare the distance relationship between the feature vectors corresponding to the 2D images at two positions; in response to the distance relationship satisfying a preset condition, the feature vectors corresponding to the 2D images at the two positions are matched to obtain a first matching relationship; and determine the pose transformation relationship between each two positions according to the first matching relationship.
[0146] In some embodiments, the processing component 30 is further configured to perform a first registration on the first matching relationship to obtain an initial transformation relationship between each two positions; extract corresponding point cloud clusters according to the first matching relationship, and perform a second registration according to the point cloud clusters and the initial transformation relationship to obtain the pose transformation relationship between each two positions.
[0147] In some embodiments, the processing component 30 is further configured to obtain the point cloud center of the target point cloud; perform feature extraction on the point cloud center to obtain the feature vector corresponding to the point cloud center; wherein, the feature vector includes the distances between the point cloud center and the K nearest neighbor point cloud centers and the angles between adjacent nearest neighbor point cloud centers.
[0148] In some embodiments, the processing component 30 is further configured to perform feature point matching on the 2D images at each two positions; determine corresponding key point pairs from the point clouds of the 2D images at the two positions according to the matching relationship; and determine the pose transformation relationship between each two positions according to the key point pairs.
[0149] In some embodiments, the processing component 30 is further configured to determine the corresponding first neighborhood point cloud according to the first key point in the key point pair, and determine the corresponding second neighborhood point cloud according to the second key point in the key point pair; perform a first registration on the key points to obtain an initial transformation relationship between each two positions; and perform a second registration according to the initial transformation relationship, the first neighborhood point cloud and the second neighborhood point cloud to obtain the pose transformation relationship between each two positions.
[0150] In some embodiments, the processing component 30 is further configured to determine a target position from different positions as the pose starting node; obtain the best estimated pose of each position relative to the pose starting node according to the pose starting node and the pose transformation relationship; transform the point cloud corresponding to each position according to its corresponding best estimated pose, and splice all the point clouds after the pose transformation to complete the target reconstruction.
[0151] In some embodiments, the processing component 30 is further configured to implement the method of any of the above embodiments.
[0152] See Figure 9 , Figure 9It is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by this application. The computer-readable storage medium 90 is used to store a computer program 91, and when the computer program 91 is executed by a processor, it is used to implement the following method:
[0153] After projecting an identifier in the area where the target is located, obtain the 2D image and 3D image corresponding to the target collected at different positions; convert the 3D image into a point cloud; determine the pose transformation relationship between every two positions based on the 2D image and the point cloud; and perform target reconstruction based on the pose transformation relationship and the point clouds at different positions.
[0154] In some embodiments, when the computer program 91 is executed by a processor, it is also used to implement the method of any of the above embodiments.
[0155] In summary, for the target reconstruction method, target reconstruction system, and computer-readable storage medium provided by this application, after projecting an identifier in the area where the target is located, obtain the 2D image and 3D image corresponding to the target collected at different positions; convert the 3D image into a point cloud; then determine the pose transformation relationship between every two positions based on the 2D image with the identifier and the point cloud; and perform target reconstruction based on the pose transformation relationship and the point clouds at different positions. That is, based on the point cloud, this application combines the identifier features in the 2D image to construct the pose transformation relationship between different positions, thereby improving the accuracy of the pose transformation relationship and further improving the accuracy of target reconstruction.
[0156] Furthermore, this application combines the speckle identifier of the laser projector and 3D image acquisition technology to handle the reconstruction problem of large workpieces with missing geometric information. For example, it can handle large workpieces with simple geometric structures such as flat plates, cylinders, and spheres, and can also handle workpieces with complex geometric structures. Its robustness is stronger than the solution that only relies on the geometric features of the point cloud.
[0157] Furthermore, reconstruction can be performed without the prior pose information of the multi-view photographing positions.
[0158] In several implementation manners provided by this application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the device implementation manner described above is only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0159] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processing circuit component (processor) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.
[0160] The above are only the embodiments of this application, and do not limit the patent scope of this application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of this application.
Claims
1. A target reconstruction method, characterized in that, The method includes: After projecting an identifier in the area where the target is located, obtaining 2D images and 3D images corresponding to the target collected at different positions; Converting the 3D image into a point cloud; Determining the pose transformation relationship between every two positions based on the 2D image and the point cloud; Performing target reconstruction based on the pose transformation relationship and the point clouds at different positions.
2. The target reconstruction method according to claim 1, wherein The determining the pose transformation relationship between every two positions based on the 2D image and the point cloud includes: Extracting the identifier in each 2D image; For each identifier, determining the target point cloud corresponding to the identifier from the point cloud corresponding to the 2D image; Performing feature extraction on the target point cloud to obtain a feature vector corresponding to the target point cloud; wherein, the feature vector includes the distances between the target point cloud and K nearest neighbor point clouds and the angles between adjacent nearest neighbor point clouds; Determining the pose transformation relationship between every two positions by using the feature vector corresponding to the 2D image.
3. The target reconstruction method according to claim 2, characterized in that, The determining the pose transformation relationship between every two positions by using the feature vector corresponding to the 2D image includes: Comparing the distance relationship between the feature vectors corresponding to the 2D images at two positions; In response to the distance relationship satisfying a preset condition, the feature vectors corresponding to the 2D images at the two positions are matched to obtain a first matching relationship; Determining the pose transformation relationship between every two positions according to the first matching relationship.
4. The target reconstruction method according to claim 3, wherein The determining the pose transformation relationship between every two positions according to the first matching relationship includes: Performing a first registration on the first matching relationship to obtain an initial transformation relationship between every two positions; According to the first matching relationship, extracting corresponding point cloud clusters, and performing a second registration according to the point cloud clusters and the initial transformation relationship to obtain the pose transformation relationship between every two positions.
5. The target reconstruction method according to claim 2, wherein The performing feature extraction on the target point cloud to obtain a feature vector corresponding to the target point cloud includes: Obtaining the point cloud center of the target point cloud; Performing feature extraction on the point cloud center to obtain a feature vector corresponding to the point cloud center; wherein, the feature vector includes the distances between the point cloud center and K nearest neighbor point cloud centers and the angles between adjacent nearest neighbor point cloud centers.
6. The target reconstruction method according to claim 1, wherein The determining the pose transformation relationship between every two positions based on the 2D image and the point cloud includes: Performing feature point matching on the 2D images at every two positions; According to the matching relationship, determining corresponding key point pairs from the point clouds of the 2D images at the two positions; Determining the pose transformation relationship between every two positions according to the key point pairs.
7. The target reconstruction method according to claim 6, wherein The determining the pose transformation relationship between every two positions according to the key point pairs includes: Determining the first neighborhood point cloud corresponding to the first key point in the key point pair, and determining the second neighborhood point cloud corresponding to the second key point in the key point pair; Performing a first registration on the key points to obtain an initial transformation relationship between every two positions; Perform a second registration based on the initial transformation relationship, the first neighborhood point cloud, and the second neighborhood point cloud to obtain the pose transformation relationship between each two positions.
8. The target reconstruction method according to claim 1, wherein The target reconstruction based on the pose transformation relationship and the point clouds at different positions includes: Determine a target position from the different positions as the pose starting node; Based on the pose starting node and the pose transformation relationship, obtain the best estimated pose of each position relative to the pose starting node; Transform the point cloud corresponding to each position according to its corresponding best estimated pose, and splice the point clouds after all pose transformations to complete the target reconstruction.
9. A target reconstruction system, characterized in that, The target reconstruction system includes: An identification projection component for projecting an identification to the area where the target is located; An image acquisition component for acquiring the 2D image and 3D image corresponding to the target at different positions after projecting the identification to the area where the target is located; A processing component connected to the image acquisition component for receiving the 2D image and the 3D image, converting the 3D image into a point cloud; determining the pose transformation relationship between each two positions based on the 2D image and the point cloud; and performing target reconstruction based on the pose transformation relationship and the point clouds at different positions.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which when executed by a processor, is used to implement the target reconstruction method according to any one of claims 1-8.