Pose estimation method and electronic equipment

Abstract

The embodiment of the invention provides a pose estimation method and electronic equipment, and belongs to the technical field of robots, and the method comprises the steps: obtaining image data and point cloud data of at least one target object; inputting the image data into a pre-training detection model for feature extraction processing to obtain bounding box data of each target object; performing point cloud segmentation processing on the point cloud data according to the bounding box data and a preset internal reference matrix to obtain first point cloud data; preprocessing the first point cloud data to obtain second point cloud data; performing density clustering processing on the second point cloud data to obtain at least one point cloud cluster; determining the point cloud centroid of each target object according to each point cloud cluster; and performing principal component analysis processing on the point cloud cluster according to the point cloud centroid to obtain a first feature vector and a second feature vector of the target object, and determining attitude data of each target object according to the first feature vector and the second feature vector of each target object. The method is used for achieving the effect of accurately estimating the pose of the target object.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a posture estimation method and electronic equipment. Background Art

[0002] With the rapid development of robotics, it has become a key driver of progress across various industries. Robots can replace humans in completing various tasks, and accurate pose estimation is crucial for robots to successfully complete these tasks. Therefore, accurately estimating the pose of the task object, enabling humanoid robots to complete tasks more efficiently and accurately, has become a hot research topic within the industry. Summary of the Invention

[0003] The embodiments of the present application provide a posture estimation method and an electronic device for achieving accurate posture estimation of a target object.

[0004] In a first aspect, an embodiment of the present application provides a pose estimation method, comprising:

[0005] Acquiring image data and point cloud data of at least one target object;

[0006] Inputting the image data into a pre-trained detection model for feature extraction processing to obtain bounding box data of each target object;

[0007] Performing point cloud segmentation processing on the point cloud data according to the bounding box data and a preset intrinsic parameter matrix to obtain first point cloud data;

[0008] Preprocessing the first point cloud data to obtain second point cloud data, wherein the point cloud density of the second point cloud data is lower than that of the first point cloud data;

[0009] performing density clustering processing on the second point cloud data to obtain at least one point cloud cluster;

[0010] According to each point cloud cluster, determine the point cloud centroid of each target object;

[0011] The point cloud cluster is subjected to principal component analysis according to the point cloud centroid to obtain a first eigenvector and a second eigenvector of the target object, and the posture data of each target object is determined according to the first eigenvector and the second eigenvector of each target object.

[0012] In one possible implementation, the point cloud data includes multiple initial discrete points, and the bounding box data includes a bounding box of each target object. Accordingly, the point cloud segmentation processing is performed on the point cloud data according to the bounding box data and a preset intrinsic parameter matrix to obtain first point cloud data, including: projecting each initial discrete point onto a two-dimensional image plane according to the preset intrinsic parameter matrix to obtain a point cloud projection point corresponding to each initial discrete point; traversing each point cloud projection point according to the bounding box of each target object to retain the point cloud projection point located within each bounding box; and extracting a corresponding initial discrete point from the multiple initial discrete points as a first discrete point according to the point cloud projection point located within each bounding box to obtain first point cloud data.

[0013] In one possible implementation, the first point cloud data includes a plurality of first discrete points; accordingly, preprocessing the first point cloud data to obtain the second point cloud data includes: calculating a point cloud normal vector based on the first discrete points; and comparing the point cloud normal vector with a preset reference normal vector, so that when a preset point cloud rejection condition is met, the first discrete point corresponding to the point cloud normal vector is rejected from the first point cloud data to obtain intermediate point cloud data.

[0014] Downsampling is performed on the intermediate point cloud data to obtain second point cloud data.

[0015] In one possible implementation, the second point cloud data includes multiple second discrete points; accordingly, the density clustering processing is performed on the second point cloud data to obtain at least one point cloud cluster, including: dividing the multiple second discrete points into core points, boundary points, and noise points according to a preset neighborhood radius and a lower limit point number threshold; starting from the core point, expanding, removing the noise points in the multiple discrete points, and grouping adjacent second discrete points into the same cluster with the core point as the cluster center to form multiple point cloud clusters; wherein each point cloud cluster corresponds to a target object.

[0016] In one possible implementation, each of the point cloud clusters includes multiple second discrete points; accordingly, determining the point cloud centroid of each target object based on each point cloud cluster includes: calculating the three-dimensional coordinate mean corresponding to each point cloud cluster based on the second discrete points in each point cloud cluster; and determining the three-dimensional coordinate mean as the coordinates of the point cloud centroid of the target object corresponding to the point cloud cluster.

[0017] In a possible embodiment, each of the point cloud clusters includes multiple second discrete points; accordingly, the point cloud cluster is subjected to principal component analysis according to the point cloud centroid to obtain the first eigenvector and the second eigenvector of the target object, including: calculating the deviation of each second discrete point in the corresponding point cloud cluster relative to the point cloud centroid according to the point cloud centroid, and constructing the covariance matrix of the point cloud cluster according to the deviation of each second discrete point relative to the point cloud centroid; performing eigenvalue decomposition on the covariance matrix of the point cloud cluster to obtain multiple target eigenvalues and multiple corresponding target eigenvectors, wherein the target eigenvalue is proportional to the directional importance corresponding to the target eigenvector; extracting the maximum target eigenvalue and the second largest target eigenvalue from the multiple target eigenvalues, and determining the target eigenvector corresponding to the maximum target eigenvalue as the first eigenvector, and determining the target eigenvector corresponding to the second largest target eigenvalue as the second eigenvector.

[0018] In one possible embodiment, determining the posture data of each target object based on the first eigenvector and the second eigenvector of each target object includes: performing vector product on the first eigenvector and the second eigenvector of each target object to obtain a third eigenvector; constructing a local coordinate system of the target object based on the first eigenvector, the second eigenvector and the third eigenvector, wherein the local coordinate system of each target object represents the posture data of the target object.

[0019] In one possible embodiment, the pre-trained detection model is obtained after model training using training image data collected in a multi-target scene, wherein the training image data collected in the multi-target scene includes images of multiple training targets at multiple angles, multiple lighting conditions, and multiple occlusion conditions.

[0020] In a possible implementation, the method further includes: inputting the image data into the pre-trained detection model for attribute extraction processing to obtain attribute data of each target object.

[0021] In a second aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0022] The memory stores computer-executable instructions;

[0023] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the first aspect and / or various possible implementations of the first aspect.

[0024] A posture estimation method and electronic device provided in an embodiment of the present application first collect image data and point cloud data of a target object, then use a pre-trained detection model to obtain bounding box data of each target object, and then sequentially undergo point cloud segmentation processing, preprocessing and clustering processing to eliminate interfering redundant data and background noise to obtain at least one point cloud cluster, and then use the point cloud cluster to determine the point cloud centroid of the target object as the position of the target object, and finally use the point cloud centroid to perform principal component analysis on the point cloud cluster until the posture data of each target object is calculated, thereby providing target posture data for the humanoid robot to perform tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] Figure 1 A schematic diagram of a scene of a pose estimation method provided in an embodiment of the present application;

[0027] Figure 2 A flowchart of a pose estimation method provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a process for performing point cloud segmentation processing on point cloud data provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a process for preprocessing first point cloud data provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a process for performing density clustering processing on second point cloud data provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the target detection process of a pre-trained detection model provided in an embodiment of the present application;

[0032] Figure 7 A schematic structural diagram of a posture estimation device provided in this application;

[0033] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application.

[0034] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0036] Existing pose estimation methods based on feature point matching have relatively low requirements for the target object model. They only need to extract the target's feature points using the scale-invariant feature transform (SIFT) algorithm and the accelerated robust feature (SURF) algorithm to match them with the pre-established feature points of the target model and estimate the pose. Complex 3D models are not required. While these methods can quickly and accurately estimate the target's pose in some simple scenarios, they become difficult to extract and match when there are occlusions, complex backgrounds, or unclear feature points in the scene. This can easily lead to mismatches, thus reducing the accuracy of pose estimation.

[0037] In response to the above technical problems, an embodiment of the present application provides an inventive concept of a pose estimation method as follows: by combining target detection, point cloud clustering and PCA principal component analysis technology, fast and accurate pose estimation can be achieved in complex environments, providing a guarantee for the robot to better perform tasks.

[0038] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0039] Figure 1 A schematic diagram of a scene of a pose estimation method provided in an embodiment of the present application, such as Figure 1 As shown, the application scenario includes: a humanoid robot 101 and multiple targets 102.

[0040] Among them, a data acquisition device and a server are installed on the humanoid robot 101, and the server is communicatively connected to the data acquisition device. The data acquisition device is used to collect image data and point cloud data of the target object, and to obtain image data and point cloud data by real-time shooting of multiple target objects 102. Then, the server receives the collected image data and point cloud data, and executes relevant steps of the posture estimation method for the image data and point cloud data, and finally generates the position and posture of the humanoid robot 101 relative to the target object 102, providing a basis for the humanoid robot to subsequently perform related tasks.

[0041] In an optional embodiment of the present application, the processor of the data acquisition device of the humanoid robot can also execute the relevant steps of the posture estimation method, and finally generate the posture estimation result of the humanoid robot 101 for the target object 10, and transmit it to the server inside the humanoid robot 101.

[0042] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0043] Figure 2 A flow chart of a posture estimation method provided in an embodiment of the present application is provided. The execution subject of the posture estimation method may be as follows Figure 1 The server in the humanoid robot 101 shown, or the computer-related device wirelessly connected to the humanoid robot 101, is not limited in this embodiment.

[0044] like Figure 2 As shown, the pose estimation method includes:

[0045] S201: Acquire image data and point cloud data of at least one target object.

[0046] In this embodiment, the image data can be color image data captured by a camera, which is a two-dimensional image. The point cloud data can be a data set composed of a large number of discrete points in a three-dimensional space captured by a depth camera, each of which includes information such as three-dimensional coordinates, color, reflection intensity, and normal vector.

[0047] S202: Input the image data into the pre-trained detection model for feature extraction processing to obtain the bounding box data of each target object.

[0048] In this embodiment, the pre-trained detection model can be a trained object detection model, such as a YOLO model. The pre-trained detection model can detect objects in an input two-dimensional image and extract bounding box data of the object. The bounding box data of the object can include the location of the object in the image, such as the pixel coordinates of the upper left corner and the lower right corner.

[0049] S203: Performing point cloud segmentation processing on the point cloud data according to the bounding box data and the preset intrinsic parameter matrix to obtain first point cloud data.

[0050] In this embodiment, the preset intrinsic parameter matrix can be based on the camera intrinsic parameters (such as focal length , and optical center coordinates , ) The obtained matrix, the preset internal parameter matrix is as follows:

[0051]

[0052] Where K represents a preset intrinsic parameter matrix, and R represents a set of all real numbers. Point cloud segmentation processing can be a process of projecting point cloud data onto a two-dimensional image plane to obtain the coordinates of each pixel point.

[0053] In some optional embodiments, when the target object volume is larger than a first preset threshold, the anchor box size of the pre-trained detection model is larger than a second preset threshold; when the target object volume is smaller than the first preset threshold, the anchor box size of the pre-trained detection model is smaller than a third preset threshold; wherein the third preset threshold is smaller than the second preset threshold. This is because for very small targets, if the anchor box is too large, the target may not be accurately detected; and for very large targets, if the anchor box is too small, the entire target may not be covered. Therefore, this helps improve the accuracy of detecting the target object's point cloud centroid, i.e., its position.

[0054] Figure 3 A schematic diagram of a process for performing point cloud segmentation processing on point cloud data provided in an embodiment of the present application.

[0055] In this embodiment, the point cloud data is a data set consisting of a large number of discrete points in three-dimensional space.

[0056] like Figure 3 As shown, based on the above embodiment, as an optional embodiment of the present application, the pose estimation method provided, the point cloud data includes multiple initial discrete points, and the bounding box data includes the bounding box of each target object; accordingly, step S203 includes:

[0057] S203a: Projecting each initial discrete point onto a two-dimensional image plane according to a preset intrinsic parameter matrix to obtain a point cloud projection point corresponding to each initial discrete point.

[0058] In this embodiment, the initial discrete points in the point cloud data are (x i ,y i ,z i ) where x i is the x-axis value corresponding to the i-th initial discrete point, y i is the y-axis value corresponding to the i-th initial discrete point, z i is the z-axis value of the i-th initial discrete point. The initial discrete point can be projected onto the two-dimensional image plane using the projection formula to obtain the point cloud projection point (u, v).

[0059] In an optional embodiment of the present application, the projection formula for obtaining the point cloud projection point corresponding to each initial discrete point in step S203a is as follows:

[0060] ,

[0061] Where, the coordinates of the initial discrete point are (x, y, z), u and v represent the horizontal and vertical coordinate values of the point cloud projection point on the two-dimensional image plane, , is the camera focal length, , is the optical center coordinate.

[0062] In this embodiment, the corresponding relationship between the i-th initial discrete point in the point cloud data and the corresponding point cloud projection point can be obtained from the projection formula and the preset internal parameter matrix as follows:

[0063]

[0064] Using the corresponding relationship in the above formula, the coordinates of all point cloud projection points in the two-dimensional image plane can be obtained.

[0065] S203b: traverse each point cloud projection point according to the bounding box of each target object to retain the point cloud projection points located within each bounding box.

[0066] In this embodiment, traversing each point cloud projection point according to the bounding box of each target object refers to traversing all bounding box data and using each bounding box data to detect whether each point cloud projection point is within the bounding box range, wherein the bounding box range can be determined by the upper left corner coordinate (x min ,y min ) and the lower right corner coordinate (x max ,y max If it is within the bounding box, the point cloud projection point is retained; otherwise, the point cloud projection points outside the bounding box are discarded.

[0067] S203c: Extracting corresponding initial discrete points from the plurality of initial discrete points as first discrete points according to the point cloud projection points located within each bounding box, to obtain first point cloud data.

[0068] In this embodiment, step S203c refers to calculating the discrete points within each bounding box and the corresponding projection relationship, obtaining initial discrete points corresponding to each point cloud projection point from multiple initial discrete points, and using these initial discrete points as first discrete points. All first discrete points form first point cloud data.

[0069] S204: Preprocess the first point cloud data to obtain second point cloud data, wherein the point cloud density of the second point cloud data is lower than that of the first point cloud data.

[0070] In this embodiment, preprocessing can be the process of removing redundant information from point cloud data. The second point cloud data generated after preprocessing the first point cloud data has a lower point cloud density than the first point cloud data due to the removal of redundant information, thereby improving subsequent computational efficiency. For example, preprocessing can include desktop removal and downsampling. Desktop removal can be implemented based on existing desktop removal algorithms.

[0071] Figure 4 A schematic diagram of a process for preprocessing first point cloud data provided in an embodiment of the present application.

[0072] like Figure 4 As shown, based on the above embodiment, in an optional embodiment of the present application, the first point cloud data includes a plurality of first discrete points; accordingly, step S204 includes:

[0073] S204a: Calculate the point cloud normal vector according to the first discrete point.

[0074] In this embodiment, the point cloud normal vector may be calculated by performing a normal solution using the first point cloud coordinate point and the surface on which the target object is placed. The direction of the point cloud normal vector may be perpendicular to the plane on which the target object is placed or may be at a certain angle to the plane on which the target object is placed.

[0075] S204b: Compare the point cloud normal vector with a preset reference normal vector, and when a preset point cloud rejection condition is met, remove the first discrete point corresponding to the point cloud normal vector from the first point cloud data to obtain intermediate point cloud data.

[0076] In this embodiment, comparing the point cloud normal vector with a preset reference normal vector may include calculating an angle between the point cloud normal vector and the preset reference normal vector, and then comparing the angle with a preset angle threshold. The preset reference normal vector may be a pre-specified specific normal vector for reference, for example, any normal vector perpendicular to the target object.

[0077] In this embodiment, the basis for determining whether the point cloud rejection conditions are met can be that the angle of a first discrete point is calculated to be less than a preset angle threshold, and the value representing the height in the first discrete point is consistent with the height of the plane on which the target object is placed. In this case, it indicates that the first discrete point is not the first discrete point on the target object, and it is necessary to reject the first discrete point. Otherwise, the first discrete point is retained, and all retained first discrete points are stored as intermediate point cloud data.

[0078] S204c: Down-sampling the intermediate point cloud data to obtain second point cloud data.

[0079] In this embodiment, downsampling may involve dividing the intermediate point cloud data into uniformly sized three-dimensional grids, replacing all discrete points in each grid with the execution point or center point within the grid, and preserving the geometric properties of the discrete points. For example, a voxel filtering method may be employed to divide the intermediate point cloud data into multiple uniformly sized three-dimensional grids (hereinafter referred to as voxels), replacing all points in each voxel with the execution point or center point within the voxel, thereby reducing the point cloud density while preserving the geometric properties of the point cloud.

[0080] In this embodiment, a method based on normal vectors for removing redundant information improves data processing efficiency for flat and regular planar scenes while also being more robust to noise points. Downsampling reduces the point cloud density, further supporting efficiency improvements in subsequent data processing.

[0081] S205: Perform density clustering processing on the second point cloud data to obtain at least one point cloud cluster.

[0082] In this embodiment, the density clustering process may be a process of separating different objects in the second point cloud data into independent point cloud clusters corresponding to each object to remove background and noise points. Each point cloud cluster in the plurality of point cloud clusters represents a target object.

[0083] Figure 5 A schematic diagram of a process for performing density clustering processing on second point cloud data provided in an embodiment of the present application.

[0084] like Figure 5 As shown, based on the above embodiment, as an optional embodiment of the present application, the pose estimation method provided, the second point cloud data includes a plurality of second discrete points; accordingly, step S205 includes:

[0085] S205a: Divide the plurality of second discrete points into core points, boundary points, and noise points according to a preset neighborhood radius and a lower point number threshold.

[0086] In this embodiment, the preset neighborhood radius may be a critical value for determining the proximity relationship between second discrete points. The lower limit point count threshold may be the minimum value of the second discrete points included in a neighborhood. In this embodiment, a core point may be a second discrete point. When the number of second discrete points included in a neighborhood divided by a preset neighborhood radius is greater than a predetermined fixed number, the second discrete point may be considered a core point. A boundary point may be a second discrete point within the core point area, and a noise point refers to an isolated second discrete point that is not within any neighborhood.

[0087] For example, in an optional embodiment of the present application, assuming that the preset neighborhood radius in step S205a is ε, if one of the second discrete points P in the point cloud cluster is iand another second discrete point P j The distance between them satisfies the following distance judgment formula:

[0088]

[0089] Where, dist(P i ,p j ) represents the second discrete point P i With the second discrete point P j The distance between the second discrete point P i The coordinates of (x i ,y i ,z i ), the second discrete point P j The coordinates of (x j ,y j ,z j ), where i and j are positive integers and i≠j.

[0090] If the distance judgment formula is satisfied, the second discrete point P is considered i is the second discrete point P j Starting from the core point, the adjacent neighbor points are classified into the same cluster, and finally a point cloud cluster is formed.

[0091] S205b: Starting from the core point, the noise points in the multiple discrete points are removed, and the adjacent second discrete points are grouped into the same cluster with the core point as the cluster center to form multiple point cloud clusters; each point cloud cluster corresponds to a target object.

[0092] In this embodiment, by expanding from the core point, two adjacent second discrete points are grouped into the same cluster, ultimately forming multiple point cloud clusters. Each cluster corresponds to a target object, and each point cloud cluster contains the target object's three-dimensional information. In this embodiment, the algorithm used for density clustering can be the density clustering DBSCAN algorithm. After clustering is completed, each target object in the original point cloud data is separated into an independent point cloud cluster, and background and noise points are removed, providing more accurate data input for subsequent pose estimation, thereby improving the accuracy of the pose estimation results.

[0093] S206: Determine the point cloud centroid of each target object based on each point cloud cluster. The coordinates of the point cloud centroid of the target object are the location information of the target object.

[0094] In this embodiment, each point cloud cluster corresponds to a target object, and all discrete points in the point cloud cluster can be obtained by solving the coordinate average value.

[0095] Specifically, in an optional embodiment of the present application, each point cloud cluster includes a plurality of second discrete points. Accordingly, step S206 includes:

[0096] S206a: Calculate the three-dimensional coordinate mean corresponding to each point cloud cluster according to the second discrete point in each point cloud cluster.

[0097] In this embodiment, step S206a can be performed by calculating the average values of the x-axis coordinates, y-axis coordinates, and z-axis coordinates of all the second discrete points in each point cloud cluster to obtain the mean value μ = (μ x ,μ y , μ z ), where μ x Indicates the mean x-axis coordinate corresponding to the point cloud cluster, μ y Indicates the y-axis coordinate value corresponding to the point cloud cluster, μ z Indicates the z-axis coordinate value corresponding to the point cloud cluster.

[0098] S206b: Determine the three-dimensional coordinate mean as the coordinates of the point cloud centroid of the target object corresponding to the point cloud cluster.

[0099] In this embodiment, the mean of the three-dimensional coordinates of each point cloud cluster can be used to represent the centroid of the target object corresponding to the point cloud cluster. For example, the centroid of the point cloud of the target object is ,The execution of the target object can be used as the corresponding target position when the humanoid robot performs the task, where the total number of the second discrete points in the N point cloud cluster, P i =( x i ,y i ,z i ) represents the coordinates of the i-th second discrete point.

[0100] S207: performing principal component analysis on the point cloud cluster according to the point cloud centroid to obtain the first eigenvector and the second eigenvector of the target object, and determining the posture data of each target object according to the first eigenvector and the second eigenvector of each target object.

[0101] In this embodiment, performing principal component analysis on the point cloud cluster according to the point cloud centroid can be a process of converting the second point cloud data in the point cloud cluster into a set of linearly independent variables through orthogonal transformation, sorting these variables by variance size, and retaining the variability of the data to the greatest extent, and then performing feature decomposition through the point cloud centroid coordinates and linearly independent variables to obtain the feature vector.

[0102] The posture data is obtained by calculating based on principal component analysis, which has the advantages of small calculation amount and high calculation efficiency.

[0103] Based on the above embodiment, in an optional embodiment of the present application, each point cloud cluster includes multiple second discrete points; accordingly, in step S207, principal component analysis is performed on the point cloud cluster according to the point cloud centroid to obtain the first eigenvector and the second eigenvector of the target object, including:

[0104] S207a: Calculate the deviation of each second discrete point in the corresponding point cloud cluster relative to the point cloud centroid according to the point cloud centroid, and construct a covariance matrix of the point cloud cluster according to the deviation of each second discrete point relative to the point cloud centroid.

[0105] In this embodiment, the deviation of each second discrete point in the corresponding point cloud cluster relative to the point cloud mass center can be calculated based on the point cloud mass center by subtracting the coordinates of the point cloud mass center from the coordinates of each second discrete point in turn. For example: the i-th second discrete point P i The deviation from the point cloud centroid is (P i -μ). The covariance matrix can be constructed using the covariance matrix transformation formula.

[0106] In an optional embodiment of the present application, the covariance matrix can be represented by C, where: , where (P i -μ) represents the deviation of the i-th second discrete point compared to the point cloud centroid, C represents a 3×3 covariance matrix, N represents the number of second discrete points in the point cloud cluster, and T represents the transpose of the matrix consisting of the deviations of all second discrete points in the point cloud cluster compared to the point cloud centroid.

[0107] S207b: Perform eigenvalue decomposition on the covariance matrix of the point cloud cluster to obtain multiple target eigenvalues and multiple corresponding target eigenvectors, wherein the importance of the directions corresponding to the target eigenvalues and the target eigenvectors are proportional.

[0108] In this embodiment, eigenvalue decomposition refers to a process of solving a covariance matrix to obtain the eigenvalues and corresponding eigenvectors of the covariance matrix.

[0109] In an optional embodiment of the present application, after eigendecomposition of the covariance matrix, eigenvalues (λ1, λ2, λ3) and corresponding eigenvectors (v1, v2, v3) are obtained. The importance of the direction corresponding to the target eigenvalue and the target eigenvector is proportional. The eigenvalue and eigenvector of the covariance matrix can satisfy the following relationship: , where λ i represents the i-th eigenvalue of the covariance matrix C, v i It represents the characteristic vector corresponding to the i-th eigenvalue, and the larger the eigenvalue, the more important the direction corresponding to the eigenvector is.

[0110] S207c: Extract the maximum target eigenvalue and the second largest target eigenvalue from the multiple target eigenvalues, and determine the target eigenvector corresponding to the maximum target eigenvalue as the first eigenvector, and determine the target eigenvector corresponding to the second largest target eigenvalue as the second eigenvector.

[0111] In this embodiment, the maximum target eigenvalue refers to the target eigenvalue with the largest value obtained after sorting and comparing the values of multiple target eigenvalues in each point cloud cluster. The value of the second largest target eigenvalue is smaller than the value of the maximum target eigenvalue but larger than the values of the remaining target eigenvalues. The first eigenvector is the main direction (long axis) of the target object v long =argmax(λ1,λ2,λ3), the second eigenvector is the secondary direction (minor axis) of the target object v shot =argmin(λ1,λ2,λ3).

[0112] In an optional embodiment of the present application, determining the posture data of each target object according to the first feature vector and the second feature vector of each target object in step S207 includes:

[0113] S207d: Perform vector product on the first eigenvector and the second eigenvector of each target object to obtain a third eigenvector.

[0114] In this embodiment, it is assumed that the first eigenvector is v long , the second eigenvector is v shot , then according to the vector right-hand rule, we can get the third eigenvector v by performing vector multiplication th = v long × v shot .

[0115] S207e: Constructing a local coordinate system of the target object based on the first eigenvector, the second eigenvector, and the third eigenvector, wherein the local coordinate system of each target object represents the posture data of the target object.

[0116] In this embodiment, step S207e refers to forming a local coordinate system that ultimately contains three coordinate axes using the first eigenvector as the major axis, the second eigenvector as the minor axis, and the third eigenvector as the third direction. The local coordinate system of the target object is used to characterize the posture data of the target object, where the posture data may include the posture and rotation direction of the target object.

[0117] In summary, the target pose estimation method provided in the embodiment of the present application first collects the image data and point cloud data of the target object, and then uses the pre-trained detection model to obtain the bounding box data of each target object, and then sequentially undergoes point cloud segmentation processing, preprocessing and clustering processing to eliminate interfering redundant data and background noise to obtain at least one point cloud cluster, and then uses the point cloud cluster to determine the point cloud centroid of the target object as the position of the target object, thereby improving the estimation accuracy of the target object position, and finally uses the point cloud centroid to perform principal component analysis on the point cloud cluster until the posture data of each target object is calculated, thereby improving the accuracy of posture estimation, and thus providing more accurate target pose data for the humanoid robot to perform tasks.

[0118] At the same time, the position of the target object, i.e., the centroid of the point cloud, is used in the subsequent principal component analysis process to further improve the accuracy of the target object's posture data.

[0119] At the same time, it can also perform pose estimation on multiple targets at the same time, and ensure the accuracy of the pose estimation results of multiple targets while ensuring the efficiency of the pose estimation work.

[0120] Based on the above embodiments, in an optional embodiment of the present application, the pre-trained detection model is obtained after model training using training image data collected in a multi-target scene, wherein the training image data collected in the multi-target scene includes images of multiple training targets at multiple angles, multiple lighting conditions, and multiple occlusion conditions.

[0121] In this embodiment, the multi-target scene refers to a scene with multiple training targets, and the multiple training targets can be multiple different objects, for example, multiple training targets can include bottles, boxes, boxes and plates. The training image data refers to the two-dimensional color image and point cloud data used to train the model. Specifically, the target objects in the scene can be captured in real time by using a camera and saved in the form of RGB images to ensure that color images of each training target object are collected at multiple different angles, different lighting conditions and different occlusion conditions, so as to enhance the robustness of the pre-trained detection model. Of course, if the accuracy of the pose estimation results of a single type of target object is to be improved, the multiple training targets can also be multiple identical objects. This enhances the performance of the present application in complex scenes and the accuracy of the pose estimation results of multiple targets in complex scenes.

[0122] In some optional embodiments, in step S206, the point cloud cluster corresponding to the target object can be divided into multiple parts based on the mechanical structure characteristics of the target object. Based on the degree of correlation between each part of the point cloud and the task to be performed by the humanoid robot, the structure corresponding to the part with the highest correlation with the task to be performed is determined as the target structure, and the point cloud centroid is calculated based on the point cloud corresponding to the target structure. In other words, the average value of the coordinates of this part of the point cloud is used as the point cloud centroid. This can improve the accuracy of the point cloud position used to perform the task, thereby improving the accuracy of the target object position estimation used to perform the task.

[0123] For example, a cup, based on its mechanical structure, consists of a body and a handle. Therefore, its corresponding point cloud cluster consists of two parts: the first part corresponding to the body, and the second part corresponding to the handle. If the robot's task is to grasp the cup, then using the average of the point cloud coordinates of the body and handle as the position estimate may result in the robot failing to reach the handle due to the average value not being near the handle, leading to task failure. This approach improves position estimation accuracy, thereby increasing the success rate of task execution.

[0124] In some optional embodiments, in step S207, if the target object is partially occluded, that is, it includes both occluded and unoccluded areas, then during the feature decomposition of the point cloud cluster, the feature granularity of the occluded area is greater than that of the unoccluded area. This is because when local features are occluded, large-scale features can still provide sufficient information to estimate the pose, while features that are too small cannot provide sufficient information, thereby improving the accuracy of pose estimation for objects with occluded areas.

[0125] Figure 6 A schematic diagram of the target detection process of a pre-trained detection model provided in an embodiment of the present application.

[0126] In an optional embodiment of the present application, the process of performing model training to obtain a pre-trained detection model includes: first, manually annotating the collected image data through an annotation tool to generate corresponding training bounding box information, and storing the training bounding box information as a label file in YOLO format, wherein the label file contains the category and position coordinates of each training target object. After the annotation is completed, the data set is divided into a training set and a validation set, which are used for the preliminary model training process and the later model performance evaluation process, respectively. In this embodiment, the YOLOv5 framework can be used to train the collected image data, and the image input is set to be consistent with the resolution of the collected image data during the model training process.

[0127] like Figure 6As shown, the pre-trained detection model is used to detect that the desktop contains a box-type training target object and an orange toy-type training target object, as well as the bounding box information of these two training targets.

[0128] Based on the above embodiment, the pose estimation method provided as an optional embodiment of the present application further includes:

[0129] Step A: Input the image data into the pre-trained detection model for attribute extraction processing to obtain the attribute data of each target object.

[0130] In this embodiment, the pre-trained detection model performs visual inspection on the target object's image data to extract its attributes. The target object's attribute data may include the target's name, microscopic category (e.g., red bottle, orange toy), macroscopic category (e.g., bottles, batteries), and the number of targets. This target attribute data is used to better understand the target object's attributes during the humanoid robot's mission. Furthermore, counting the number of targets of each category provides a reference for the humanoid robot to perform tasks such as grasping and sorting.

[0131] Based on the above embodiment, in an optional embodiment of the present application, a pose estimation method is provided, further comprising:

[0132] Step B: Execute the task operation based on the point cloud centroid, posture data and attribute data of each target object.

[0133] In summary, the posture estimation method provided in the embodiment of the present application also reduces redundant calculations by completing the attribute detection and posture estimation of the target object sequentially in the same data chain, providing more detailed and accurate data basis for the task execution of the humanoid robot, and improving the efficiency of the robot's task execution.

[0134] Figure 7 A schematic diagram of the structure of a posture estimation device provided in this application is shown as follows: Figure 7 As shown, the posture estimation device provided in this embodiment includes: a data acquisition module 71, an identification and detection module 72, a data processing module 73 and a posture calculation module 74.

[0135] The data acquisition module 71 is used to acquire image data and point cloud data of at least one target object.

[0136] The recognition and detection module 72 is used to input the image data into the pre-trained detection model for feature extraction processing to obtain the bounding box data of each target object.

[0137] The data processing module 73 is used to perform point cloud segmentation processing on the point cloud data according to the bounding box data and the preset intrinsic parameter matrix to obtain first point cloud data.

[0138] The data processing module 73 is further configured to pre-process the first point cloud data to obtain second point cloud data, wherein the point cloud density of the second point cloud data is lower than that of the first point cloud data.

[0139] The data processing module 73 is further configured to perform density clustering processing on the second point cloud data to obtain at least one point cloud cluster.

[0140] The pose calculation module 74 is used to determine the point cloud centroid of each target object based on each point cloud cluster.

[0141] The posture calculation module 74 is also used to perform principal component analysis on the point cloud cluster according to the point cloud centroid to obtain the first eigenvector and the second eigenvector of the target object, and determine the posture data of each target object based on the first eigenvector and the second eigenvector of each target object.

[0142] In an optional embodiment of the present application, the point cloud data includes multiple initial discrete points, and the bounding box data includes the bounding box of each target object; accordingly, the data processing module 73 is specifically used to: project each initial discrete point onto the two-dimensional image plane according to a preset intrinsic parameter matrix to obtain a point cloud projection point corresponding to each initial discrete point; traverse each point cloud projection point according to the bounding box of each target object to retain the point cloud projection point located within each bounding box; extract the corresponding initial discrete point from the multiple initial discrete points as the first discrete point according to the point cloud projection point located within each bounding box to obtain the first point cloud data.

[0143] In an optional embodiment of the present application, the first point cloud data includes multiple first discrete points; accordingly, the data processing module 73 is further specifically used to: calculate the point cloud normal vector based on the first discrete points; compare the point cloud normal vector with a preset reference normal vector, so as to eliminate the first discrete point corresponding to the point cloud normal vector from the first point cloud data when a preset point cloud elimination condition is met, to obtain intermediate point cloud data; and downsample the intermediate point cloud data to obtain second point cloud data.

[0144] In an optional embodiment of the present application, the second point cloud data includes multiple second discrete points; accordingly, the data processing module 73 is further specifically used to: divide the multiple second discrete points into core points, boundary points and noise points according to a preset neighborhood radius and a lower limit point number threshold; start from the core point, remove the noise points in the multiple discrete points, and classify the adjacent second discrete points into the same cluster with the core point as the cluster center to form multiple point cloud clusters; each point cloud cluster corresponds to a target object.

[0145] In an optional embodiment of the present application, each point cloud cluster includes multiple second discrete points; accordingly, the posture calculation module 74 is specifically used to: calculate the three-dimensional coordinate mean corresponding to each point cloud cluster based on the second discrete points in each point cloud cluster; and determine the three-dimensional coordinate mean as the coordinates of the point cloud center of mass of the target object corresponding to the point cloud cluster.

[0146] In an optional embodiment of the present application, each point cloud cluster includes multiple second discrete points; accordingly, the posture calculation module 74 is specifically used to: calculate the deviation of each second discrete point in the corresponding point cloud cluster relative to the point cloud center of mass based on the point cloud center of mass, and construct the covariance matrix of the point cloud cluster based on the deviation of each second discrete point relative to the point cloud center of mass; perform eigenvalue decomposition on the covariance matrix of the point cloud cluster to obtain multiple target eigenvalues and multiple corresponding target eigenvectors, wherein the target eigenvalues are proportional to the directional importance corresponding to the target eigenvectors; extract the maximum target eigenvalue and the second largest target eigenvalue from the multiple target eigenvalues, and determine the target eigenvector corresponding to the maximum target eigenvalue as the first eigenvector, and determine the target eigenvector corresponding to the second largest target eigenvalue as the second eigenvector.

[0147] In an optional embodiment of the present application, the posture calculation module 74 is further specifically used to: perform vector product based on the first eigenvector and the second eigenvector of each target object to obtain a third eigenvector; construct a local coordinate system of the target object based on the first eigenvector, the second eigenvector and the third eigenvector, wherein the local coordinate system of each target object represents the posture data of the target object.

[0148] In an optional embodiment of the present application, the recognition and detection module 72 is further used to: input the image data into a pre-trained detection model for attribute extraction processing to obtain attribute data of each target object.

[0149] The posture estimation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0150] Figure 8 This is a schematic diagram of the structure of an electronic device provided by this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus.

[0151] During the specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that the at least one processor 801 performs the above method.

[0152] The specific implementation process of the processor 801 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0153] An embodiment of the present application further provides a humanoid robot, comprising a robot body, a data acquisition device disposed on the robot body, and an electronic device as described in the above embodiment disposed within the robot body.

[0154] In this embodiment, both being arranged on the robot body and being arranged inside the robot body can be realized through hardware connection structures such as bolt connection, thread connection, etc.

[0155] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned pose estimation method is implemented.

[0156] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above pose estimation generation method when executed by a processor.

[0157] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0158] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0159] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0160] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0161] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0162] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0163] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0164] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0165] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0167] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0168] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0169] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A pose estimation method, characterized in that: include: Acquiring image data and point cloud data of at least one target object; Inputting the image data into a pre-trained detection model for feature extraction processing to obtain bounding box data of each target object; Performing point cloud segmentation processing on the point cloud data according to the bounding box data and a preset intrinsic parameter matrix to obtain first point cloud data; Preprocessing the first point cloud data to obtain second point cloud data, wherein the point cloud density of the second point cloud data is lower than that of the first point cloud data; performing density clustering processing on the second point cloud data to obtain at least one point cloud cluster; According to each point cloud cluster, determine the point cloud centroid of each target object; The point cloud cluster is subjected to principal component analysis according to the point cloud centroid to obtain a first eigenvector and a second eigenvector of the target object, and the posture data of each target object is determined according to the first eigenvector and the second eigenvector of each target object.

2. The method according to claim 1, characterized in that The point cloud data includes a plurality of initial discrete points, and the bounding box data includes a bounding box of each target object; Accordingly, performing point cloud segmentation processing on the point cloud data according to the bounding box data and the preset intrinsic parameter matrix to obtain first point cloud data includes: Projecting each initial discrete point onto the two-dimensional image plane according to the preset intrinsic parameter matrix to obtain the point cloud projection point corresponding to each initial discrete point; Traverse each point cloud projection point according to the bounding box of each target object to retain the point cloud projection points located within each bounding box; According to the point cloud projection points located within each bounding box, corresponding initial discrete points are extracted from a plurality of initial discrete points as first discrete points to obtain first point cloud data.

3. The method according to claim 1, characterized in that The first point cloud data includes a plurality of first discrete points; Accordingly, the preprocessing of the first point cloud data to obtain the second point cloud data includes: Calculating a point cloud normal vector according to the first discrete point; Comparing the point cloud normal vector with a preset reference normal vector, and when a preset point cloud rejection condition is met, removing a first discrete point corresponding to the point cloud normal vector from the first point cloud data to obtain intermediate point cloud data; Downsampling is performed on the intermediate point cloud data to obtain second point cloud data.

4. The method according to any one of claims 1 to 3, characterized in that The second point cloud data includes a plurality of second discrete points; Accordingly, the density clustering process is performed on the second point cloud data to obtain at least one point cloud cluster, including: Dividing the plurality of second discrete points into core points, boundary points and noise points according to a preset neighborhood radius and a lower limit point number threshold; Starting from the core point, noise points in the multiple discrete points are removed, and the adjacent second discrete points are classified into the same cluster with the core point as the cluster center to form multiple point cloud clusters; each point cloud cluster corresponds to a target object.

5. The method according to claim 1, wherein Each of the point cloud clusters includes a plurality of second discrete points; Accordingly, determining the point cloud centroid of each target object according to each point cloud cluster includes: The mean of the three-dimensional coordinates corresponding to each point cloud cluster is calculated based on the second discrete point in each point cloud cluster; The three-dimensional coordinate mean is determined as the coordinate of the point cloud centroid of the target object corresponding to the point cloud cluster.

6. The method according to claim 1, characterized in that Each of the point cloud clusters includes a plurality of second discrete points; Accordingly, performing principal component analysis on the point cloud cluster according to the point cloud centroid to obtain a first eigenvector and a second eigenvector of the target object includes: Calculating the deviation of each second discrete point in the corresponding point cloud cluster relative to the point cloud centroid according to the point cloud centroid, and constructing a covariance matrix of the point cloud cluster according to the deviation of each second discrete point relative to the point cloud centroid; Performing eigenvalue decomposition on the covariance matrix of the point cloud cluster to obtain a plurality of target eigenvalues and a plurality of corresponding target eigenvectors, wherein the target eigenvalues are proportional to the importance of directions corresponding to the target eigenvectors; A maximum target eigenvalue and a second maximum target eigenvalue are extracted from multiple target eigenvalues, and a target eigenvector corresponding to the maximum target eigenvalue is determined as a first eigenvector, and a target eigenvector corresponding to the second maximum target eigenvalue is determined as a second eigenvector.

7. The method according to claim 1, characterized in that Determining the posture data of each target object according to the first eigenvector and the second eigenvector of each target object includes: Performing a vector product on the first eigenvector and the second eigenvector of each target object to obtain a third eigenvector; A local coordinate system of the target object is constructed based on the first eigenvector, the second eigenvector, and the third eigenvector, wherein the local coordinate system of each target object represents the posture data of the target object.

8. The method according to claim 1, characterized in that The pre-trained detection model is obtained after model training using training image data collected in a multi-target scene, wherein the training image data collected in the multi-target scene includes images of multiple training targets at multiple angles, multiple lighting conditions, and multiple occlusion conditions.

9. The method according to any one of claims 1 to 3 or 5 to 8, characterized in that Also includes: The image data is input into the pre-trained detection model for attribute extraction processing to obtain attribute data of each target object.

10. An electronic device, characterized in that: It is characterized by comprising: a memory, a processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the pose estimation method according to any one of claims 1 to 9.

Images