Pose estimation method, medium and device combining two-dimensional image and three-dimensional point cloud

Abstract

The present invention relates to the field of automation technology, and in particular to a pose estimation method, medium, and device combining two-dimensional images with three-dimensional point clouds. The method comprises the following steps: real-time data acquisition; YOLOv8 image instance segmentation; point cloud mapping segmentation; performing instance segmentation on the image, and then segmenting it using the correspondence between the two-dimensional image and the three-dimensional point cloud; point cloud registration sequence decision-making; comprehensively deciding the order of point cloud registration based on the confidence of the segmented instance identified by YOLOv8, the average height of the point cloud instance, and the relationship between the number of points of the point cloud instance and the point cloud registration template; point cloud preprocessing; point cloud registration; and outputting the target category and pose. The present invention uses deep learning to identify and segment instance images, obtains a point cloud instance of the target through point cloud mapping segmentation, and proposes a new registration decision method that can solve the problems of low accuracy and poor efficiency in target recognition and point cloud segmentation in complex aliasing scenes.

Description

Technical Field

[0001] The present invention relates to the field of automation technology, and in particular to a pose estimation method, medium, and device combining a two-dimensional image with a three-dimensional point cloud. Background Art

[0002] Traditional point cloud object recognition, segmentation, and point cloud registration methods typically utilize methods based on traditional machine learning and point cloud segmentation techniques. However, in complex scenarios, these methods often struggle to meet the accuracy and efficiency requirements for point cloud segmentation and registration. This is because point cloud data in complex scenarios typically has higher dimensions and larger data volumes, which traditional machine learning methods often cannot effectively process. Furthermore, traditional point cloud segmentation methods struggle to handle issues such as occlusion, noise, and point cloud overlap in complex scenes, which impacts the accuracy and efficiency of point cloud segmentation and registration. Currently, point cloud segmentation and registration is a major challenge in industrial fields such as unordered grasping, industrial sorting, and autonomous loading and unloading, and is a must-solve issue for achieving industrial automation. Existing technologies are mostly based on traditional 2D images or 3D point clouds. 2D image-based methods are generally more efficient, but because they lack object depth and distance information, they are only suitable for industrial applications in simple, regular scenes. Methods based on three-dimensional point clouds contain rich object feature information and can accurately describe the shape and spatial position relationship of objects. However, the high dimensionality of point cloud data leads to high complexity in some data processing and calculations, requiring efficient algorithms and computing resources to achieve real-time operations. Summary of the Invention

[0003] In response to the problems of the prior art, the present invention provides a pose estimation method that combines two-dimensional images with three-dimensional point clouds. It uses deep learning to identify and segment instance images, obtains point cloud instances of the target through point cloud mapping and segmentation, and proposes a new registration decision method, which can solve the problems of low accuracy and poor efficiency of target recognition and point cloud segmentation in complex aliasing scenes.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] The present invention provides a pose estimation method combining a two-dimensional image and a three-dimensional point cloud, which comprises the following steps:

[0006] Step S10, real-time data acquisition; real-time data acquisition is divided into two-dimensional image acquisition and three-dimensional point cloud real-time data acquisition;

[0007] Step S20, YOLOv8 image instance segmentation; first, a deep convolutional neural network is used to extract features from the input image to capture key information in the image; next, the feature map is processed using convolutional layers and pooling layers to further extract contextual information of the image; then, a fully connected layer is used to map the extracted feature map to a high-dimensional feature vector to represent different target instances in the image; based on this, candidate target regions are extracted from the image to further narrow the scope of instance segmentation; finally, a segmentation network is used to classify and segment the candidate target regions at the pixel level to obtain the final instance segmentation result;

[0008] Step S30, point cloud mapping segmentation: performing instance segmentation on the image, and then performing segmentation using the correspondence between the two-dimensional image and the three-dimensional point cloud;

[0009] Step S40, point cloud registration order decision: The order of point cloud registration is decided based on the confidence of the segmented instances identified by YOLOv8, the average height of the point cloud instances, and the relationship between the number of points of the point cloud instances and the point cloud registration template.

[0010] Step S50: point cloud preprocessing;

[0011] Step S60: point cloud registration;

[0012] Step S70: Output the target category and pose.

[0013] Wherein, in the step S10, the two-dimensional image acquisition uses a camera device to capture the image, and images of different angles and clarity are obtained by adjusting the parameters of the camera.

[0014] In step S10, the three-dimensional point cloud real-time data acquisition uses a laser scanner to obtain point cloud data on the surface of the object; the point cloud coordinates on the surface of the object are obtained by transmitting a laser beam from the laser scanner and receiving a reflected light signal from the receiver.

[0015] Wherein, in step S30, the point cloud mapping segmentation method is:

[0016] Step S31, camera parameter calibration: camera parameter calibration obtains camera parameters;

[0017] Step S32, point cloud segmentation: Based on the spatial mapping relationship between the pixel coordinate system and the camera coordinate system, a mapping relationship between the two-dimensional image and the three-dimensional point cloud depth image is established; based on the mapping relationship, the pixel coordinate values ​​of the non-zero pixel areas in the binary mask of the image instance segmentation are traversed to locate the point cloud area of ​​each instance, and finally the target point cloud instance is segmented; the method of mapping the binary mask image to the three-dimensional point cloud is as follows:

[0018]

[0019] Among them, (p1, p2, p3) is the point cloud camera coordinate of point p in the target point cloud area, p d is the depth c of point p f is the camera depth scale factor, (m,n) is the pixel coordinate of the color image, and f x , f y , c x , c y is the intrinsic parameter of the camera, R is the rotation matrix, and t is the translation vector.

[0020] In step S40, the decision-making method for the point cloud registration order is:

[0021]

[0022] Among them, s confidence The confidence level of YOLOv8 segmentation instance recognition, H average is the average height of the entire point cloud, H i The average height of different point cloud instances, Num template is the number of points in the point cloud template, Num i is the number of points in different point cloud instances, w c , w h , w p are the weights assigned to the three dimensions respectively.

[0023] Wherein, in step S50, the point cloud preprocessing method is:

[0024] Step S51, octree downsampling: using octree downsampling to pre-process the point cloud instance before registration, by dividing the point cloud into an octree structure and sampling each leaf node, and finally obtaining a downsampled point cloud;

[0025] Step S52: Radius filtering: Determine whether each point in the point cloud is an outlier based on the neighborhood information around it, and remove the outlier.

[0026] Wherein, in step S60, the point cloud registration method is:

[0027] Step S61, Teaser++-based point cloud coarse registration: perform preliminary alignment on the input point cloud dataset;

[0028] Step S62: ICP-based point cloud precise registration: align one point cloud dataset to another point cloud dataset through iterative optimization.

[0029] The present invention also provides a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the pose estimation method combining the two-dimensional image and the three-dimensional point cloud.

[0030] The present invention also provides an electronic device, wherein the electronic device includes: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, enable the processor to execute the pose estimation method combining the two-dimensional image with the three-dimensional point cloud.

[0031] Beneficial effects of the present invention:

[0032] The present invention is cleverly designed. By using YOLOv8 to perform instance segmentation on a two-dimensional image, we can obtain the location and category information of each object in the image. Then, by mapping the object instances in the two-dimensional image with the three-dimensional point cloud, we can obtain the object instances in the corresponding three-dimensional point cloud; this allows each object to be accurately segmented in three-dimensional space; by utilizing the results of two-dimensional image instance segmentation, the present invention can greatly reduce the amount of data required for point cloud segmentation. At the same time, due to the mapping relationship between the two-dimensional image and the three-dimensional point cloud, the point cloud can be segmented more accurately, improving the accuracy of the segmentation; in addition, the point cloud registration order decision strategy proposed in the present invention comprehensively determines the order of point cloud registration from three dimensions: the confidence of the segmentation instance identified by YOLOv8, the average height of the point cloud instance, and the relationship between the point cloud instance and the number of points in the point cloud registration template. This can more quickly obtain a target that meets the registration accuracy requirements and threshold, thereby significantly reducing the time consumption of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the pose estimation method combining two-dimensional images and three-dimensional point clouds of the present invention. DETAILED DESCRIPTION

[0034] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the embodiments and the accompanying drawings. The contents mentioned in the embodiments are not intended to limit the present invention. The present invention will be described in detail below with reference to the accompanying drawings.

[0035] Example 1

[0036] Embodiment 1 of the present application provides a pose estimation method combining a two-dimensional image and a three-dimensional point cloud, which includes the following steps:

[0037] Step S10, real-time data acquisition; real-time data acquisition is divided into two-dimensional image acquisition and three-dimensional point cloud real-time data acquisition;

[0038] The two-dimensional image acquisition process uses a camera to capture images. Adjusting camera parameters, such as focal length and exposure time, allows for the acquisition of images of varying angles and resolutions. During real-time data acquisition, the distance and angle between the camera and the subject must be maintained to ensure accurate image information. Furthermore, multiple cameras can be used to capture images from multiple perspectives, improving both accuracy and comprehensiveness.

[0039] Among them, the real-time three-dimensional point cloud data acquisition uses a laser scanner to obtain point cloud data on the surface of an object; the point cloud coordinates on the surface of the object are obtained by transmitting the laser beam of the laser scanner and receiving the reflected light signal of the receiver. These coordinates can represent the shape, texture and other information of the object for subsequent three-dimensional reconstruction and analysis.

[0040] Step S20, YOLOv8 image instance segmentation; YOLOv8 image instance segmentation mainly utilizes the characteristics of YOLOv8 with high precision and efficiency in image instance segmentation; it can efficiently and accurately segment and classify target instances in complex aliasing scenes; first, a deep convolutional neural network is used to extract features of the input image to capture key information in the image; next, a series of convolutional layers and pooling layers are used to process the feature map to further extract the context information of the image; then, a fully connected layer is used to map the extracted feature map to a high-dimensional feature vector to represent different target instances in the image; on this basis, candidate target regions are extracted from the image to further narrow the scope of instance segmentation; finally, a segmentation network is used to classify and segment the candidate target regions at the pixel level to obtain the final instance segmentation result; through this series of steps, YOLOv8 can achieve accurate segmentation and recognition of target instances in the image;

[0041] Step S30, point cloud mapping segmentation; perform instance segmentation on the image, and then use the correspondence between the two-dimensional image and the three-dimensional point cloud to segment it; point cloud mapping segmentation combines deep learning with traditional algorithms, uses a deep learning network to perform instance segmentation on the image, and then uses the correspondence between the two-dimensional image and the three-dimensional point cloud to segment it; it can fully utilize the advantages of deep learning in image instance segmentation in aliasing scenes, greatly reducing the amount of calculation and improving segmentation accuracy.

[0042] Step S40, point cloud registration order decision; the order of point cloud registration is determined based on the confidence of the segmentation instance identified by YOLOv8, the average height of the point cloud instance, and the relationship between the number of points of the point cloud instance and the point cloud registration template. Through the above steps S10 to S30, multiple individual point cloud instances of different target categories can be obtained. However, when performing registration to solve the pose, if the registration operation is performed on each point cloud instance, it will not only consume a large amount of device computing resources, but also greatly reduce the computing efficiency of the entire system, and it will not be possible to achieve the requirement of real-time registration to solve the target pose. Therefore, the role of the point cloud registration order decision is to comprehensively consider the registration order of each point cloud instance, to register as quickly as possible and meet other requirements such as thresholds. This comprehensive sorting method can enable the system to complete the registration task more efficiently. In view of the characteristics of point clouds in aliasing scenarios, the embodiment of the present application uses the confidence of the segmentation instance identified by YOLOv8, the average height of the point cloud instance, and the relationship between the number of points of the point cloud instance and the point cloud registration template to determine the order of point cloud registration. In addition, considering that in actual applications, there are usually different targets, different scenarios and differences in equipment, the decision-making method of the embodiment of the present application assigns flexibly adjustable weights to the indicators of the three dimensions respectively.

[0043] Step S50, point cloud preprocessing; in order to solve the problems of large amount of point cloud instance data and many outliers, point cloud preprocessing must be performed before point cloud registration; by preprocessing the point cloud data, noise and redundancy can be reduced, and data quality can be improved, thereby providing more accurate and efficient input for subsequent point cloud registration.

[0044] Step S60: Point cloud registration. After these steps, a highly accurate point cloud is obtained, preserving its geometric structure and features while reducing the amount of data required. This point cloud meets the requirements for point cloud registration and can be used as input for the point cloud registration algorithm, improving registration accuracy and efficiency. Point cloud registration is divided into coarse point cloud registration based on Teaser++ and fine point cloud registration based on ICP.

[0045] Step S70: Output the target category and pose. Through all the above steps, the category and pose information of each two-dimensional image instance and three-dimensional point cloud can be obtained. This information is of great significance for tasks such as target detection, object tracking, and three-dimensional reconstruction.

[0046] Specifically, the embodiments of the present application are ingeniously designed. By using YOLOv8 to perform instance segmentation on a two-dimensional image, the position and category information of each object in the image can be obtained. Then, by mapping the object instances in the two-dimensional image to the three-dimensional point cloud, the corresponding object instances in the three-dimensional point cloud can be obtained. This allows for accurate segmentation of each object in three-dimensional space. By utilizing the results of two-dimensional image instance segmentation, the embodiments of the present application can significantly reduce the amount of data required for point cloud segmentation. At the same time, due to the mapping relationship between the two-dimensional image and the three-dimensional point cloud, the point cloud can be segmented more accurately, improving the accuracy of the segmentation. Furthermore, the point cloud registration order decision strategy proposed in the embodiments of the present application comprehensively determines the order of point cloud registration based on three dimensions: the confidence level of the segmented instances identified by YOLOv8, the average height of the point cloud instances, and the relationship between the number of points in the point cloud instances and the point cloud registration template. This allows for faster acquisition of targets that meet the registration accuracy requirements and thresholds, thereby significantly reducing the time required for practical applications.

[0047] In the embodiment of the present application, in step S30, the method for point cloud mapping segmentation is as follows: step S31, camera parameter calibration: camera parameter calibration obtains camera parameters; point cloud mapping segmentation is based on the correspondence between the two-dimensional image and the three-dimensional point cloud, therefore, camera parameter calibration must be performed before obtaining camera parameters;

[0048] Step S32, point cloud segmentation: Based on the spatial mapping relationship between the pixel coordinate system and the camera coordinate system, a mapping relationship between the two-dimensional image and the three-dimensional point cloud depth image is established; based on the mapping relationship, the pixel coordinate values ​​of the non-zero pixel areas in the binary mask of the image instance segmentation are traversed to locate the point cloud area of ​​each instance, and finally the target point cloud instance is segmented; the method of mapping the binary mask image to the three-dimensional point cloud is as follows:

[0049]

[0050] Among them, (p1, p2, p3) is the point cloud camera coordinate of point p in the target point cloud area, p d is the depth c of point p f is the camera depth scale factor, (m,n) is the pixel coordinate of the color image, and f x , f y , c x , c y is the intrinsic parameter of the camera, R is the rotation matrix, and t is the translation vector.

[0051] In the embodiment of the present application, in step S40, the decision method for determining the point cloud registration order is:

[0052]

[0053] Among them, sconfidence The confidence level of YOLOv8 segmentation instance recognition, H average is the average height of the entire point cloud, H i The average height of different point cloud instances, Num template is the number of points in the point cloud template, Num i is the number of points in different point cloud instances, w c , w h , w p The weights assigned to the three dimensions are respectively; the higher the score, the closer the registration order is, and vice versa.

[0054] In the embodiment of the present application, in step S50, the point cloud preprocessing method is as follows: Step S51, octree downsampling; octree downsampling is used to preprocess the point cloud instance before registration, by dividing the point cloud into an octree structure, sampling each leaf node, and finally obtaining a downsampled point cloud; to address the problems of large point cloud data volume and feature redundancy, the embodiment of the present application uses octree downsampling to preprocess the point cloud instance before registration; octree downsampling is a commonly used point cloud downsampling method, which divides the point cloud into an octree structure, samples each leaf node, and finally obtains a downsampled point cloud; this method does not change the morphology and features of the point cloud, can maintain the geometric structure and topological relationship of the point cloud, and is applicable to various types of point cloud data, including ordered point clouds and unordered point clouds. Therefore, using octree downsampling before registration can preserve point cloud features while reducing the amount of redundant data. Step S52, radius filtering; based on the neighborhood information around each point in the point cloud, determine whether it is an outlier and remove the outlier. After the octree downsampling, the point cloud instances still have a large number of outliers and noise points. The embodiment of the present application uses radius filtering to pre-process the point cloud instances. Point cloud radius filtering is a commonly used point cloud denoising method. It determines whether it is an outlier based on the neighborhood information around each point in the point cloud and removes the outliers. The principle of this method is to calculate the distance between each point and its neighboring points, and determine the points whose distance exceeds the set radius threshold as outliers. This method has the characteristics of being simple and fast, having adjustable parameters, preserving local structure, and being applicable to sparse point clouds. Therefore, it is suitable for pre-processing before point cloud registration.

[0055] In the embodiment of the present application, in the step S60, the point cloud registration method is: step S61, coarse point cloud registration based on Teaser++: the input point cloud data set is preliminarily aligned; in point cloud registration, coarse registration is an important step, and its goal is to preliminarily align the input point cloud data set. Teaser++ uses an efficient iterative optimization algorithm by utilizing the geometric information between point clouds, which can achieve coarse registration of point clouds in a relatively short time. Teaser++ not only performs well in registration accuracy, but also has high computational efficiency; its optimization algorithm adopts an adaptive strategy, which can automatically adjust the parameters in the optimization process according to the characteristics of the point cloud, so as to obtain better registration results on different data sets. In addition, Teaser++ also supports parallel computing, which can take advantage of the advantages of multi-core processors to further improve the efficiency of registration; therefore, it is suitable for target coarse registration operations in aliasing scenarios.

[0056] Step S62: ICP-based point cloud registration: Align one point cloud dataset to another through iterative optimization. The ICP algorithm is one of the most commonly used methods for point cloud registration, and is also the most widely used and mature registration algorithm. It uses iterative optimization to align one point cloud dataset (called the target point cloud) to another point cloud dataset (called the reference point cloud). The core idea of ​​the ICP algorithm is to minimize the distance between the target point cloud and the reference point cloud to find the optimal rigid body transformation (rotation and translation) between the two point clouds, thereby maximizing their overlap. The main steps of the ICP algorithm include finding the nearest neighbor point pair, calculating the rigid body transformation, updating the target point cloud, and iterating until convergence. In each iteration, the ICP algorithm maps each point in the target point cloud to the reference point cloud based on the current transformation matrix and finds the point with the closest distance to it. Then, the optimal rotation and translation matrix is ​​solved using the least squares method to transform the target point cloud. As the number of iterations increases, the target point cloud gradually aligns with the reference point cloud until an accurate match is achieved.

[0057] Example 2

[0058] Embodiment 2 of the present application provides a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the pose estimation method combining the two-dimensional image and the three-dimensional point cloud.

[0059] Example 3

[0060] Embodiment 3 of the present application provides an electronic device, wherein the electronic device includes: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to execute the pose estimation method combining the two-dimensional image with the three-dimensional point cloud.

[0061] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention is disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technology of the present invention are all within the scope of the technical solution of the present invention without departing from the content of the technical solution of the present invention.

Claims

1. A pose estimation method combining two-dimensional images and three-dimensional point clouds, characterized in that: The following steps are involved: Step S10, real-time data acquisition; real-time data acquisition is divided into two-dimensional image acquisition and three-dimensional point cloud real-time data acquisition; Step S20: YOLOv8 image instance segmentation. First, a deep convolutional neural network is used to extract features from the input image to capture key information in the image. Next, the feature map is processed using convolutional layers and pooling layers to further extract contextual information of the image. Then, the extracted feature map is mapped to a high-dimensional feature vector through a fully connected layer to represent different target instances in the image. On this basis, candidate target regions are extracted from the image to further narrow the scope of instance segmentation. Finally, a segmentation network is used to classify and segment the candidate target regions at the pixel level to obtain the final instance segmentation result. Step S30, point cloud mapping and segmentation; Perform instance segmentation on the image and then segment it using the correspondence between the 2D image and the 3D point cloud; Step S40, point cloud registration order decision: The order of point cloud registration is decided based on the confidence of the segmented instances identified by YOLOv8, the average height of the point cloud instances, and the relationship between the number of points of the point cloud instances and the point cloud registration template. Step S50: point cloud preprocessing; Step S60: point cloud registration; Step S70: output target category and pose; In step S40, the decision-making method for the point cloud registration order is: Among them, s confidence The confidence level of YOLOv8 segmentation instance recognition, H average is the average height of the entire point cloud, H i The average height of different point cloud instances, Num template is the number of points in the point cloud template, Num i is the number of points in different point cloud instances, w c , w h , w p The weights given to the three dimensions are respectively; the higher the score, the higher the registration order, and vice versa.

2. The pose estimation method combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S10, the two-dimensional image acquisition uses a camera device to capture images, and images of different angles and sharpness are obtained by adjusting the parameters of the camera.

3. The pose estimation method combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S10, the three-dimensional point cloud real-time data acquisition uses a laser scanner to obtain point cloud data on the surface of the object; the point cloud coordinates on the surface of the object are obtained by transmitting a laser beam from the laser scanner and receiving a reflected light signal from the receiver.

4. The pose estimation method combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S30, the point cloud mapping segmentation method is: Step S31, camera parameter calibration: camera parameter calibration obtains camera parameters; Step S32, point cloud segmentation: Based on the spatial mapping relationship between the pixel coordinate system and the camera coordinate system, a mapping relationship between the two-dimensional image and the three-dimensional point cloud depth image is established; based on the mapping relationship, the pixel coordinate values ​​of the non-zero pixel areas in the binary mask of the image instance segmentation are traversed to locate the point cloud area of ​​each instance, and finally the target point cloud instance is segmented; the method of mapping the binary mask image to the three-dimensional point cloud is as follows: Among them, (p1, p2, p3) is the point cloud camera coordinate of point p in the target point cloud area, p d is the depth of point p, c f is the camera depth scale factor, (m,n) is the pixel coordinate of the color image, and f x , f y , c x , c y is the intrinsic parameter of the camera, R is the rotation matrix, and t is the translation vector.

5. The pose estimation method combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S50, the point cloud preprocessing method is: Step S51, octree downsampling; Use octree downsampling to pre-process the point cloud instance before registration. By dividing the point cloud into an octree structure and sampling each leaf node, the downsampled point cloud is finally obtained. Step S52: Radius filtering: Determine whether each point in the point cloud is an outlier based on the neighborhood information around it, and remove the outlier.

6. The pose estimation method combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S60, the point cloud registration method is: Step S61, Teaser++-based point cloud coarse registration: perform preliminary alignment on the input point cloud dataset; Step S62: ICP-based point cloud precise registration: align one point cloud dataset to another point cloud dataset through iterative optimization.

7. A computer storage medium storing computer instructions, which, when called, are used to execute the pose estimation method combining a two-dimensional image and a three-dimensional point cloud as described in any one of claims 1 to 6.

8. An electronic device, wherein: The electronic device includes: a processor; and a memory arranged to store computer-executable instructions, wherein when the executable instructions are executed, the processor executes the pose estimation method for combining a two-dimensional image with a three-dimensional point cloud as described in any one of claims 1 to 6.

Images