Two-dimensional image and three-dimensional point cloud combined pose estimation method, medium and equipment

Through the pose estimation method of combining two-dimensional images and three-dimensional point clouds, YOLOv8 and deep learning technology are used to perform image instance segmentation and point cloud mapping segmentation, and point cloud registration is combined with Teaser++ and ICP algorithms to solve point clouds in complex scenarios. The problems of low segmentation and registration accuracy and poor efficiency achieve efficient point cloud segmentation and registration.

CN118967702AActive Publication Date: 2024-11-15东莞市德普特电子有限公司

Patent Information

Application Number
CN202411022368.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

Traditional point cloud segmentation and registration methods are difficult to meet the accuracy and efficiency requirements in complex scenarios, especially when dealing with high-dimensional point cloud data, and are difficult to deal with occlusion, noise and overlap problems, which affects the implementation of industrial automation.

Method used

Using a pose estimation method combining two-dimensional images and three-dimensional point clouds, image instance segmentation is performed through YOLOv8, point cloud mapping segmentation and deep learning technology, combined with Teaser++ and ICP algorithm for point cloud registration, and a new registration decision method is proposed. , including steps such as real-time data acquisition, image instance segmentation, point cloud mapping segmentation, point cloud preprocessing and point cloud registration.

Benefits of technology

Improve the accuracy and efficiency of point cloud segmentation, reduce the amount of data of point cloud segmentation, and segment more accurately through the mapping relationship between two-dimensional images and three-dimensional point clouds. The proposed point cloud registration order decision strategy quickly achieves the accuracy requirements. The goal of reducing the time-consuming application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967702A_ABST
    Figure CN118967702A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of automation, in particular to a two-dimensional image and three-dimensional point cloud combined pose estimation method, medium and equipment, and the method comprises the following steps: collecting real-time data; carrying out YOLOv8 image instance segmentation; carrying out point cloud mapping segmentation; performing instance segmentation on the image, and then performing segmentation by using a corresponding relationship between a two-dimensional image and a three-dimensional point cloud; point cloud registration sequence decision making; the point cloud registration sequence is decided by combining three dimensions including the confidence coefficient of the segmentation instance identified by YOLOv8, the average height of the point cloud instance and the point relation between the point cloud instance and the point cloud registration template; preprocessing the point cloud; point cloud registration is carried out; and outputting a target category and a pose. According to the method, the instance image is identified and segmented by using deep learning, the point cloud instance of the target is obtained through a point cloud mapping segmentation mode, and a brand new registration decision method is provided, so that the problems of low precision and poor efficiency of target identification and point cloud segmentation in a complex aliasing scene can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of automation technology, and in particular to a posture estimation method, medium and device combining a two-dimensional image with a three-dimensional point cloud. Background Art

[0002] Traditional point cloud target recognition segmentation and point cloud registration methods are usually implemented based on traditional machine learning and point cloud segmentation technology. However, in complex scenes, these methods often fail to meet the accuracy and efficiency requirements of point cloud segmentation and registration. This is because point cloud data in complex scenes usually have higher dimensions and larger data volumes, and traditional machine learning methods often cannot effectively process these data. In addition, traditional point cloud segmentation methods also have difficulty in dealing with problems such as occlusion, noise, and overlap between point clouds in complex scenes, which affects the accuracy and efficiency of point cloud segmentation and registration. At present, point cloud segmentation and registration is a major difficulty in industrial fields such as disordered grasping, industrial sorting, and autonomous loading and unloading, and is a problem that must be solved to achieve industrial automation. In the existing technology, most of them are based on traditional two-dimensional images or three-dimensional point clouds; the method based on two-dimensional images is generally more efficient, but because there is no depth distance information of the object, it can only meet the industrial application of simple and regular scenes. The method based on three-dimensional point cloud contains rich object feature information and can accurately describe the shape and spatial position relationship of the object; however, the dimensionality of point cloud data is high, which leads to high complexity of some data processing and calculation, and requires efficient algorithms and computing resources to achieve real-time calculations. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a pose estimation method combining two-dimensional images with three-dimensional point clouds, uses deep learning to identify and segment instance images, obtains point cloud instances of the target through point cloud mapping and segmentation, and proposes a new registration decision method, which can solve the problems of low accuracy and poor efficiency of target recognition and point cloud segmentation in complex aliasing scenes.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] The present invention provides a posture estimation method combining a two-dimensional image and a three-dimensional point cloud, which comprises the following steps:

[0006] Step S10, real-time data collection; real-time data collection is divided into two-dimensional image collection and three-dimensional point cloud real-time data collection;

[0007] Step S20, YOLOv8 image instance segmentation; first, a deep convolutional neural network is used to extract features of the input image to capture key information in the image; next, the feature map is processed using a convolutional layer and a pooling layer to further extract contextual information of the image; then, a fully connected layer is used to map the extracted feature map to a high-dimensional feature vector to represent different target instances in the image; on this basis, candidate target regions are extracted from the image to further narrow the scope of instance segmentation; finally, a segmentation network is used to classify and segment the candidate target regions at the pixel level to obtain the final instance segmentation result;

[0008] Step S30, point cloud mapping segmentation: performing instance segmentation on the image, and then performing segmentation using the correspondence between the two-dimensional image and the three-dimensional point cloud;

[0009] Step S40, point cloud registration order decision: the order of point cloud registration is decided based on the confidence of the segmentation instance identified by YOLOv8, the average height of the point cloud instance, and the relationship between the point cloud instance and the point cloud registration template points.

[0010] Step S50, point cloud preprocessing;

[0011] Step S60, point cloud registration;

[0012] Step S70: output the target category and pose.

[0013] Wherein, in the step S10, the two-dimensional image acquisition uses a camera device to capture the image, and the images of different angles and clarity are obtained by adjusting the parameters of the camera.

[0014] Among them, in the step S10, the three-dimensional point cloud real-time data acquisition uses a laser scanner to obtain point cloud data on the surface of the object; the point cloud coordinates on the surface of the object are obtained by transmitting a laser beam of the laser scanner and receiving a reflected light signal of the receiver.

[0015] Wherein, in step S30, the method of point cloud mapping segmentation is:

[0016] Step S31, camera parameter calibration: camera parameter calibration obtains camera parameters;

[0017] Step S32, point cloud segmentation: According to the spatial mapping relationship between the pixel coordinate system and the camera coordinate system, a mapping relationship between the two-dimensional image and the three-dimensional point cloud depth image is established; according to the mapping relationship, the pixel coordinate values ​​of the non-zero pixel area in the binary mask of the image instance segmentation are traversed to locate the point cloud area of ​​each instance, and finally the target point cloud instance is segmented; the method of mapping the binary mask image to the three-dimensional point cloud is as follows:

[0018]

[0019] Among them, (p1, p2, p3) is the point cloud camera coordinates of point p in the target point cloud area, p d is the depth c of point p f is the camera depth scale factor, (m,n) is the pixel coordinate of the color image, and f x , f y , c x , c y is the intrinsic parameter of the camera, R is the rotation matrix, and t is the translation vector.

[0020] Among them, in step S40, the decision method for point cloud registration sequence decision is:

[0021]

[0022] Among them, s confidence H is the confidence of YOLOv8 identifying the segmented instance, average is the average height of the entire point cloud, H i The average height of different point cloud instances, Num template is the number of points in the point cloud template, Num i is the number of points in different point cloud instances, w c , w h , w p are the weights assigned to the three dimensions respectively.

[0023] Wherein, in step S50, the point cloud preprocessing method is:

[0024] Step S51, octree downsampling: using octree downsampling to pre-process the point cloud instance before registration, by dividing the point cloud into an octree structure, sampling each leaf node, and finally obtaining a downsampled point cloud;

[0025] Step S52: Radius filtering: judging whether it is an outlier based on the neighborhood information around each point in the point cloud, and removing the outlier.

[0026] Wherein, in step S60, the point cloud registration method is:

[0027] Step S61, coarse point cloud registration based on Teaser++: preliminary alignment of the input point cloud data set;

[0028] Step S62, ICP-based point cloud precise registration: align one point cloud dataset to another point cloud dataset by iterative optimization.

[0029] The present invention also provides a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the posture estimation method combining the two-dimensional image with the three-dimensional point cloud.

[0030] The present invention also provides an electronic device, wherein the electronic device includes: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the pose estimation method combining the two-dimensional image with the three-dimensional point cloud.

[0031] Beneficial effects of the present invention:

[0032] The present invention is cleverly designed. By using YOLOv8 to perform instance segmentation on a two-dimensional image, we can obtain the position and category information of each object in the image. Then, by mapping the object instance in the two-dimensional image with the three-dimensional point cloud, the object instance in the corresponding three-dimensional point cloud can be obtained; in this way, each object can be accurately segmented in three-dimensional space; the present invention can greatly reduce the amount of data required for point cloud segmentation by utilizing the result of instance segmentation of the two-dimensional image. At the same time, due to the mapping relationship between the two-dimensional image and the three-dimensional point cloud, the point cloud can be segmented more accurately, and the accuracy of segmentation can be improved; in addition, the point cloud registration order decision strategy proposed by the present invention comprehensively makes decisions on the order of point cloud registration from three dimensions: the confidence of the segmentation instance identified by YOLOv8, the average height of the point cloud instance, and the relationship between the point cloud instance and the point cloud registration template point count, so that the target that meets the registration accuracy requirements and threshold can be obtained more quickly, thereby greatly reducing the time consumption of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flowchart of a method for estimating a posture by combining a two-dimensional image with a three-dimensional point cloud. DETAILED DESCRIPTION

[0034] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with the embodiments and drawings, and the contents mentioned in the embodiments are not intended to limit the present invention. The present invention is described in detail below in conjunction with the drawings.

[0035] Embodiment 1

[0036] Embodiment 1 of the present application provides a method for posture estimation combining a two-dimensional image and a three-dimensional point cloud, which includes the following steps:

[0037] Step S10, real-time data collection; real-time data collection is divided into two-dimensional image collection and three-dimensional point cloud real-time data collection;

[0038] The two-dimensional image acquisition uses a camera device to capture images, and images of different angles and clarity are obtained by adjusting the parameters of the camera; images of different angles and clarity can be obtained by adjusting the parameters of the camera, such as focal length, exposure time, etc. In the real-time data acquisition process, it is necessary to ensure the distance and angle between the camera and the object being photographed to obtain accurate image information; in addition, multiple cameras can be used for multi-view image acquisition to improve the accuracy and comprehensiveness of the image.

[0039] Among them, the real-time three-dimensional point cloud data acquisition uses a laser scanner to obtain point cloud data on the surface of an object; the point cloud coordinates on the surface of the object are obtained by transmitting a laser beam from the laser scanner and receiving a reflected light signal from the receiver. These coordinates can represent information such as the shape and texture of the object, which is used for subsequent three-dimensional reconstruction and analysis.

[0040] Step S20, YOLOv8 image instance segmentation; YOLOv8 image instance segmentation mainly utilizes the characteristics of high precision and efficiency of YOLOv8 in image instance segmentation; it can efficiently and accurately segment and classify target instances in complex aliasing scenes; first, a deep convolutional neural network is used to extract features of the input image to capture key information in the image; next, a series of convolutional layers and pooling layers are used to process the feature map to further extract contextual information of the image; then, the extracted feature map is mapped to a high-dimensional feature vector through a fully connected layer to represent different target instances in the image; on this basis, candidate target areas are extracted from the image to further narrow the scope of instance segmentation; finally, a segmentation network is used to classify and segment the candidate target areas at the pixel level to obtain the final instance segmentation result; through this series of steps, YOLOv8 can achieve accurate segmentation and recognition of target instances in the image;

[0041] Step S30, point cloud mapping segmentation; perform instance segmentation on the image, and then use the correspondence between the two-dimensional image and the three-dimensional point cloud for segmentation; point cloud mapping segmentation combines deep learning with traditional algorithms, uses a deep learning network to perform instance segmentation on the image, and then uses the correspondence between the two-dimensional image and the three-dimensional point cloud for segmentation; it can fully utilize the advantages of deep learning in image instance segmentation in aliasing scenes, greatly reduce the amount of calculation and improve segmentation accuracy.

[0042] Step S40, point cloud registration order decision; from the confidence of YOLOv8 recognition segmentation instance, the average height of the point cloud instance, and the point cloud instance and the point cloud registration template point number relationship, the order of point cloud registration is comprehensively decided; through the above steps S10 to S30, multiple individual point cloud instances of different target categories can be obtained; however, when performing registration to solve the pose, if each point cloud instance is registered, it will not only consume a lot of equipment computing resources, but also greatly reduce the computing efficiency of the entire system, and it is impossible to achieve the requirement of real-time registration to solve the target pose. Therefore, the role of point cloud registration order decision is to comprehensively consider the registration order of each point cloud instance, register at the fastest speed and meet the threshold and other requirements; this comprehensive sorting method can make the system complete the registration task more efficiently; in view of the characteristics of point clouds in aliasing scenes, the embodiment of the present application comprehensively decides the order of point cloud registration from the confidence of YOLOv8 recognition segmentation instance, the average height of the point cloud instance, and the point cloud instance and the point cloud registration template point number relationship, three dimensions. In addition, considering that in actual applications, there are usually different targets, different scenarios and differences in equipment, the decision-making method of the embodiment of the present application assigns flexibly adjustable weights to the indicators of the three dimensions respectively.

[0043] Step S50, point cloud preprocessing; in order to solve the problems of large amount of point cloud instance data and many outliers, point cloud preprocessing must be performed before point cloud registration; by preprocessing the point cloud data, noise and redundancy can be reduced, and data quality can be improved, thereby providing more accurate and efficient input for subsequent point cloud registration.

[0044] Step S60, point cloud registration: After the above steps, a point cloud with high accuracy can be obtained, which retains the geometric structure and features of the point cloud and reduces the amount of data of the point cloud; such a point cloud can meet the requirements of point cloud registration and can be used as the input of the point cloud registration algorithm to improve the accuracy and efficiency of registration. Point cloud registration is divided into coarse point cloud registration based on Teaser++ and fine point cloud registration based on ICP.

[0045] Step S70, output target category and pose; through all the above steps, the category and pose information of each two-dimensional image instance and three-dimensional point cloud can be obtained, which is of great significance for tasks such as target detection, object tracking, and three-dimensional reconstruction.

[0046] Specifically, the embodiment of the present application is cleverly designed. By using YOLOv8 to perform instance segmentation on a two-dimensional image, we can obtain the position and category information of each object in the image. Then, by mapping the object instance in the two-dimensional image with the three-dimensional point cloud, the object instance in the corresponding three-dimensional point cloud can be obtained. In this way, each object can be accurately segmented in three-dimensional space; the embodiment of the present application can greatly reduce the amount of data required for point cloud segmentation by utilizing the result of instance segmentation of the two-dimensional image. At the same time, due to the mapping relationship between the two-dimensional image and the three-dimensional point cloud, the point cloud can be segmented more accurately and the accuracy of segmentation can be improved; in addition, the point cloud registration order decision strategy proposed in the embodiment of the present application comprehensively makes decisions on the order of point cloud registration from three dimensions: the confidence of the segmentation instance identified by YOLOv8, the average height of the point cloud instance, and the relationship between the point cloud instance and the point cloud registration template point count, so that the target that meets the registration accuracy requirements and thresholds can be obtained more quickly, thereby greatly reducing the time consumption of practical applications.

[0047] In the embodiment of the present application, in the step S30, the method of point cloud mapping segmentation is as follows: step S31, camera parameter calibration: the camera parameter calibration obtains the parameters of the camera; the point cloud mapping segmentation is segmented based on the correspondence between the two-dimensional image and the three-dimensional point cloud, therefore, the camera parameter calibration must be performed before obtaining the parameters of the camera;

[0048] Step S32, point cloud segmentation: According to the spatial mapping relationship between the pixel coordinate system and the camera coordinate system, a mapping relationship between the two-dimensional image and the three-dimensional point cloud depth image is established; according to the mapping relationship, the pixel coordinate values ​​of the non-zero pixel area in the binary mask of the image instance segmentation are traversed to locate the point cloud area of ​​each instance, and finally the target point cloud instance is segmented; the method of mapping the binary mask image to the three-dimensional point cloud is as follows:

[0049]

[0050] Among them, (p1, p2, p3) is the point cloud camera coordinates of point p in the target point cloud area, p d is the depth c of point p f is the camera depth scale factor, (m,n) is the pixel coordinate of the color image, and f x , f y , c x , c y is the intrinsic parameter of the camera, R is the rotation matrix, and t is the translation vector.

[0051] In the embodiment of the present application, in step S40, the decision-making method for point cloud registration order is:

[0052]

[0053] Among them, sconfidence H is the confidence of YOLOv8 identifying the segmented instance, average is the average height of the entire point cloud, H i The average height of different point cloud instances, Num template is the number of points in the point cloud template, Num i is the number of points in different point cloud instances, w c , w h , w p They are the weights given to the three dimensions respectively; the higher the score, the closer the registration order is, and vice versa.

[0054] In the embodiment of the present application, in the step S50, the method of point cloud preprocessing is as follows: step S51, octree downsampling; octree downsampling is used to preprocess the point cloud instance before registration, by dividing the point cloud into an octree structure, sampling each leaf node, and finally obtaining the downsampled point cloud; in view of the large amount of point cloud data, feature redundancy and other problems, the embodiment of the present application uses octree downsampling to preprocess the point cloud instance before registration; octree downsampling is a commonly used point cloud downsampling method, which divides the point cloud into an octree structure, samples each leaf node, and finally obtains the downsampled point cloud; this method does not change the morphology and features of the point cloud, can maintain the geometric structure and topological relationship of the point cloud, and is applicable to various types of point cloud data, including ordered point clouds and unordered point clouds, so the use of octree downsampling before registration can reduce the amount of redundant data while preserving the point cloud features. Step S52, radius filtering; based on the neighborhood information around each point in the point cloud, determine whether it is an outlier point, and remove the outlier point. There are still a large number of outliers and noise points in the point cloud instance sampled by the octree. The embodiment of the present application uses radius filtering to pre-process the point cloud instance. Point cloud radius filtering is a commonly used point cloud denoising method. It determines whether it is an outlier based on the neighborhood information around each point in the point cloud and removes the outliers. The principle of this method is to calculate the distance between each point and its neighboring points, and determine the points whose distance exceeds the set radius threshold as outliers. This method has the characteristics of being simple and fast, adjustable parameters, retaining local structure, and being suitable for sparse point clouds. Therefore, it is suitable for pre-processing before point cloud registration.

[0055] In the embodiment of the present application, in the step S60, the method of point cloud registration is: step S61, coarse point cloud registration based on Teaser++: preliminarily align the input point cloud data set; in point cloud registration, coarse registration is an important step, and its goal is to preliminarily align the input point cloud data set. Teaser++ uses an efficient iterative optimization algorithm by utilizing the geometric information between point clouds, and can achieve coarse registration of point clouds in a shorter time. Teaser++ not only performs well in registration accuracy, but also has high computational efficiency; its optimization algorithm adopts an adaptive strategy, which can automatically adjust the parameters in the optimization process according to the characteristics of the point cloud, so as to obtain better registration results on different data sets. In addition, Teaser++ also supports parallel computing, and can take advantage of multi-core processors to further improve the efficiency of registration; therefore, it is suitable for target coarse registration operations in aliasing scenarios.

[0056] Step S62, ICP-based point cloud precise registration: align one point cloud data set to another point cloud data set by iterative optimization; the ICP algorithm is one of the most commonly used methods in point cloud precise registration, and is also the most widely used and mature precise registration algorithm. It aligns one point cloud data set (called the target point cloud) to another point cloud data set (called the reference point cloud) by iterative optimization. The core idea of ​​the ICP algorithm is to solve the optimal rigid body transformation (rotation and translation) between the two point clouds by minimizing the distance between the target point cloud and the reference point cloud, so that their overlap is maximized. The main steps of the ICP algorithm include: finding the nearest neighbor point pair, calculating the rigid body transformation, updating the target point cloud, and iterating until convergence. In each iteration, the ICP algorithm maps each point of the target point cloud to the reference point cloud according to the current transformation matrix, and finds the point closest to it. Then, the optimal rotation and translation matrix is ​​solved by the least squares method to transform the target point cloud. As the number of iterations increases, the target point cloud gradually aligns with the reference point cloud until an accurate matching result is achieved.

[0057] Embodiment 2

[0058] Embodiment 2 of the present application provides a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the pose estimation method combining the two-dimensional image with the three-dimensional point cloud.

[0059] Embodiment 3

[0060] Embodiment 3 of the present application provides an electronic device, wherein the electronic device includes: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to execute the pose estimation method combining the two-dimensional image with the three-dimensional point cloud.

[0061] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention is disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments according to the technology of the present invention are within the scope of the technical solution of the present invention without departing from the content of the technical solution of the present invention.

Claims

1. A pose estimation method combining a two-dimensional image with a three-dimensional point cloud, characterized in that: The following steps are involved: Step S10, real-time data collection; real-time data collection is divided into two-dimensional image collection and three-dimensional point cloud real-time data collection; Step S20, YOLOv8 image instance segmentation; first, a deep convolutional neural network is used to extract features of the input image to capture key information in the image; then, the feature map is processed using a convolutional layer and a pooling layer to further extract contextual information of the image; Then, the extracted feature map is mapped to a high-dimensional feature vector through a fully connected layer to represent different target instances in the image; on this basis, the candidate target regions are extracted from the image to further narrow the scope of instance segmentation; finally, a segmentation network is used to classify and segment the candidate target regions at the pixel level to obtain the final instance segmentation result; Step S30, point cloud mapping segmentation; Perform instance segmentation on the image and then segment it using the correspondence between the 2D image and the 3D point cloud; Step S40, point cloud registration order decision: the order of point cloud registration is decided based on the confidence of the segmentation instance identified by YOLOv8, the average height of the point cloud instance, and the relationship between the point cloud instance and the point cloud registration template points. Step S50, point cloud preprocessing; Step S60, point cloud registration; Step S70: output the target category and pose.

2. The method for pose estimation combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S10, the two-dimensional image acquisition uses a camera device to capture images, and images of different angles and sharpness are obtained by adjusting the parameters of the camera.

3. The method for pose estimation combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S10, the three-dimensional point cloud real-time data acquisition uses a laser scanner to obtain point cloud data on the surface of the object; the point cloud coordinates on the surface of the object are obtained by transmitting a laser beam of the laser scanner and receiving a reflected light signal of the receiver.

4. The method for pose estimation combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S30, the method of point cloud mapping segmentation is: Step S31, camera parameter calibration: camera parameter calibration obtains camera parameters; Step S32, point cloud segmentation: According to the spatial mapping relationship between the pixel coordinate system and the camera coordinate system, a mapping relationship between the two-dimensional image and the three-dimensional point cloud depth image is established; according to the mapping relationship, the pixel coordinate values ​​of the non-zero pixel area in the binary mask of the image instance segmentation are traversed to locate the point cloud area of ​​each instance, and finally the target point cloud instance is segmented; the method of mapping the binary mask image to the three-dimensional point cloud is as follows: Among them, (p1, p2, p3) is the point cloud camera coordinates of point p in the target point cloud area, p d is the depth c of point p f is the camera depth scale factor, (m,n) is the pixel coordinate of the color image, and f x , f y , c x , c y is the intrinsic parameter of the camera, R is the rotation matrix, and t is the translation vector.

5. The method for pose estimation combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S40, the decision-making method for point cloud registration sequence is: Among them, s confidence H is the confidence of YOLOv8 identifying the segmented instance, average is the average height of the entire point cloud, H i The average height of different point cloud instances, Num template is the number of points in the point cloud template, Num i is the number of points in different point cloud instances, w c , w h , w p are the weights assigned to the three dimensions respectively.

6. The method for pose estimation combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S50, the point cloud preprocessing method is: Step S51, octree downsampling; Use octree downsampling to pre-process the point cloud instance before registration. By dividing the point cloud into an octree structure and sampling each leaf node, the downsampled point cloud is finally obtained. Step S52: Radius filtering: judging whether it is an outlier based on the neighborhood information around each point in the point cloud, and removing the outlier.

7. The method for pose estimation combining a two-dimensional image and a three-dimensional point cloud according to claim 1, characterized in that: In step S60, the point cloud registration method is: Step S61, coarse point cloud registration based on Teaser++: preliminary alignment of the input point cloud data set; Step S62, ICP-based point cloud precise registration: align one point cloud dataset to another point cloud dataset by iterative optimization.

8. A computer storage medium storing computer instructions, which, when called, are used to execute the pose estimation method combining a two-dimensional image with a three-dimensional point cloud as described in any one of claims 1 to 7.

9. An electronic device, wherein: The electronic device comprises: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the pose estimation method combining a two-dimensional image with a three-dimensional point cloud as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Indoor scene positioning method based on hybrid camera

    CN104517289A

  • Grabbing attitude estimation method based on image instance segmentation and point cloud PCA algorithm

    CN113327298A

  • Disordered workpiece three-dimensional visual pose estimation method based on deep learning

    CN114140526A

  • Workpiece recognition and pose detection method and system and grabbing control method of mechanical arm

    CN114952809A

  • Stacked workpiece pose estimation method based on combination of two-dimensional image and three-dimensional point cloud

    CN116309847A

Cited By

  • Spatial positioning method based on multi-modal visual fusion

    CN119251303A

  • Three-dimensional image reconstruction method and device

    CN119417992A