6D pose estimation method, grabbing method, electronic equipment and robot

Abstract

The invention provides a 6D pose estimation method. The estimation method comprises the following steps: acquiring an RGB image and depth data; extracting a depth region of interest of the target object; and constructing a first point pair feature descriptor based on the depth region of interest, and matching the first point pair feature descriptor with a pre-acquired second point pair feature descriptor to determine an estimated pose of the target object. According to the pose estimation method provided by the invention, a final result can reach millimeter-level precision, and particularly, stronger robustness can be shown in an object scene with a complex shape or occlusion. In addition, the invention further provides a grabbing method based on the 6D pose estimation method, electronic equipment and a robot.

Description

Technical Field

[0001] The present application relates to the technical field of posture estimation, and in particular, to a 6D posture estimation method, a grasping method, an electronic device and a robot. Background Art

[0002] With the rapid development of industrial automation, intelligent manufacturing and embodied intelligence, the application of robotic grasping technology in production line automation, warehousing logistics, home services and other scenarios is increasing. Existing robotic grasping methods mainly rely on object detection, recognition and pose estimation technology. However, current technical solutions still face many challenges and limitations in practical applications, as follows:

[0003] First, the limitations of traditional geometric methods.

[0004] In robot grasping tasks, pose estimation usually relies on geometric feature matching techniques such as SIFT, SURF, and ORB. These methods calculate the pose by extracting the geometric features of the object surface. However, in complex environments, such methods are easily affected by lighting changes, object occlusion, and lack of texture, resulting in decreased pose estimation accuracy. In addition, geometric methods often require a long calculation time and are difficult to meet the needs of real-time grasping tasks.

[0005] Second, the shortcomings of pose estimation methods based on deep learning.

[0006] With the rapid development of deep learning technology, pose estimation methods based on deep neural networks (such as CNN and GNN) have gradually emerged. These methods regress the pose of objects directly from sensor data (such as RGB images and point clouds) through end-to-end learning, showing good generalization ability. However, such methods usually rely on a large amount of labeled data to train the model, and accurate pose annotation is difficult. At the same time, the high computational complexity of deep learning models also limits their application in resource-constrained scenarios.

[0007] Third, there is a disconnect between pose estimation and path planning in robot grasping.

[0008] In existing robot grasping systems, pose estimation and grasping path planning are usually performed separately. This separation leads to low overall efficiency of grasping tasks, especially when facing dynamic or complex environments, when the robot needs to frequently adjust the pose estimation and path planning results. Existing technologies are difficult to balance the accuracy of pose estimation and the flexibility of path planning, which affects the success rate of grasping.

[0009] Fourth, the shortcomings of 3D object detection methods based on point pair features (PPF).

[0010] Point pair features (PPF) is a 3D object detection and pose estimation method based on geometric features. PPF shows good robustness in handling object matching and pose estimation in complex scenes, especially in the presence of noise and partial occlusion. However, the traditional PPF method relies on a large number of feature calculations and global searches, resulting in low computational efficiency and difficulty in meeting the needs of real-time applications. The PPF method also has certain limitations in its adaptability when facing dynamic scenes.

[0011] Obviously, the existing technologies have shortcomings in pose estimation in robot grasping tasks, the generalization of deep learning, the computational efficiency of PPF, and the combination of pose estimation and path planning. There is an urgent need for a solution that can improve the grasping success rate and enhance real-time and robustness. Summary of the invention

[0012] On the one hand, the present application provides a 6D pose estimation method, including: acquiring RGB images and depth data; extracting a depth region of interest of a target object; constructing a first point pair feature descriptor based on the depth region of interest, and matching it with a pre-acquired second point pair feature descriptor to determine the estimated pose of the target object.

[0013] In one embodiment, the first point pair feature descriptor is fused with point cloud gradient information.

[0014] In one embodiment, the method for extracting the depth area of ​​interest of the target object includes: extracting the first pixel coordinates of the target object based on the RGB image; mapping the first pixel coordinates to the depth data to obtain the second pixel coordinates; extracting the depth value of each pixel point corresponding to the second pixel coordinate from the depth data; and extracting the depth area of ​​interest of the target object by combining the first pixel coordinates and the depth value.

[0015] In one embodiment, the method for constructing the first point pair feature descriptor includes: converting the second pixel coordinates and the depth value into online point cloud data in three-dimensional space, and extracting the online boundary point cloud of the target object from the online point cloud data; constructing the initial point pair feature descriptor based on the online boundary point cloud; fusing the initial point pair feature descriptor with the point cloud gradient information to obtain the first point pair feature descriptor.

[0016] In one embodiment, the pre-acquired second point pair feature descriptor includes acquiring the second point pair feature descriptor in an offline stage, and the method for acquiring the second point pair feature descriptor in the offline stage includes: establishing a 3D model of the target object; extracting boundary information of the 3D model and converting the boundary information into an offline boundary point cloud; and calculating and obtaining the second point pair feature descriptor based on each point in the offline boundary point cloud.

[0017] In one embodiment, the method for determining the estimated pose of the target object includes: using a feature matching algorithm to match the first point pair feature descriptor and the second point pair feature descriptor to obtain a matching pair; using a voting mechanism to assign a weight to each of the matching pairs; and aggregating the voting information of all the matching pairs to determine the estimated pose.

[0018] In one embodiment, the fusion of the initial point pair feature descriptor and the point cloud gradient information includes: calculating the normal vector of each point in the online point cloud data and its neighborhood points; calculating the gradient vector based on the normal vector and the position of the point in the online point cloud data; weighting the relative positions and directions of all the initial point pair feature descriptors according to the strength of the gradient vector to highlight point pairs with large geometric changes, and / or mapping the gradient vector into a feature space and combining it with the feature space of the initial point pair feature descriptor to form a new high-dimensional feature space.

[0019] In one embodiment, the calculation of obtaining the point cloud gradient information includes: Among them, G(p i ) is point p i The gradient vector, N(p i ) is the point containing the point p i A point p surrounded by a set of points i Neighborhood, n i and n j They are point p i and the neighborhood point p j The normal vector, (p i -p j ) is point p i and the neighborhood point p j The coordinate difference between .

[0020] In one embodiment, the method for establishing a 3D model of the target object includes: using a visual part to capture an RGB multi-view image of the target object; obtaining a sparse point cloud reconstructed from the RGB multi-view image through a three-dimensional reconstruction algorithm; providing absolute scale information to the three-dimensional reconstruction algorithm to correct the scale of the sparse point cloud; obtaining a dense point cloud based on the corrected sparse point cloud; using a pre-trained neural network model to process the RGB multi-view image to determine the position of the target object in the RGB multi-view image; combining the posture information of the visual part and the position information of the target object in the RGB multi-view image to calculate the position range of the target object in three-dimensional space; and screening out point clouds within the position range in the dense point cloud to obtain the 3D model.

[0021] In one embodiment, the first point pair feature descriptor is generated in an online stage, and the second point pair feature descriptor is generated in an offline stage, wherein the second point pair feature descriptor generated in the offline stage is directly called in the matching process of the online stage.

[0022] In one embodiment, the depth data is acquired by radar, and before extracting the depth region of interest, the estimation method further includes: geometrically aligning the RGB image and the depth data.

[0023] In one embodiment, the method of acquiring the RGB image and the depth data of the target object through an acquisition module and determining the estimated pose of the acquisition module also includes: adjusting the pose of the target object to align it with the point cloud data of the target object captured in the online stage to output the precise pose of the target object in the coordinate system of the sensor.

[0024] On the other hand, the present application provides a grasping method based on 6D pose estimation, comprising: performing real-time grasping path planning according to the estimated pose of the target object in any of the above-mentioned embodiments; and grasping the target object based on the grasping path.

[0025] On the other hand, the present application provides an electronic device based on 6D pose estimation, comprising: a processing unit and a storage unit connected thereto; the storage unit is used to store a computer program; the processing unit is used to call the computer program stored in the storage unit to execute the estimation method described in any of the above embodiments, or execute the grasping method described in the above embodiments.

[0026] On the other hand, the present application provides a device based on 6D pose estimation, including an electronic device as described in the above embodiment, and at least one sensor, wherein the sensor is used to capture RGB images and depth data of the target object for the electronic device to perform pose estimation and grasping path planning.

[0027] On the other hand, the present application provides a robot based on 6D pose estimation, including: an acquisition module, used to acquire RGB images and depth data; an extraction module, used to extract a depth region of interest of a target object; a pose estimation module, including a first point pair feature descriptor and a second point pair feature descriptor, used to determine the estimated pose of the target object, wherein the first point pair feature descriptor is constructed based on the depth region of interest, the second point pair feature descriptor is pre-acquired, and the estimated pose of the target object is determined by matching the first point pair feature descriptor and the second point pair feature descriptor.

[0028] In one embodiment, the first point pair feature descriptor is fused with point cloud gradient information.

[0029] In one embodiment, the robot further includes: a control module for planning a grasping path according to the determined estimated posture, and / or issuing grasping instructions.

[0030] In one embodiment, the robot further includes: a robotic arm, which receives a grasping instruction sent by the control module and grasps the target object according to the grasping path.

[0031] The 6D pose estimation method provided in this application may have at least one of the following beneficial effects:

[0032] According to the 6D pose estimation method in some embodiments of the present application, the computational burden of online processing is reduced by pre-acquiring the second point pair feature descriptor and calculating the boundary point cloud descriptor of the target object offline, so that the first point pair feature descriptor matching and pose estimation process in the online stage can be accelerated, effectively reducing the time consumption of online calculation, realizing efficient 6D pose estimation, and helping to meet real-time requirements.

[0033] According to the 6D pose estimation method in some embodiments of the present application, the conversion from 2D image to 3D point cloud is realized through RGB image and depth data, and preliminary three-dimensional point cloud data containing the target object is generated through the depth region of interest, and the precise three-dimensional position information of the target object is provided, which helps to construct a more accurate point pair feature descriptor, provides a reliable data basis for subsequent pose estimation, and thus improves the accuracy of matching and recognition.

[0034] According to the 6D pose estimation method in some embodiments of the present application, by extracting the depth region of interest of the target object, it is possible to accurately identify and segment the objects that need pose estimation in complex scenes, thereby ensuring the targetedness of subsequent processing.

[0035] According to the 6D pose estimation method in some embodiments of the present application, a first point pair feature descriptor is constructed based on the depth region of interest, focusing on the depth region of interest, that is, the region of the target object in the depth data, and ignoring the interference of the background and other irrelevant objects, so as to more accurately extract features related to the target object, reduce the amount of data, and improve processing efficiency.

[0036] The grasping method based on 6D pose estimation provided in this application can have at least one of the following beneficial effects:

[0037] According to the grasping method in some embodiments of the present application, real-time grasping path planning is performed through the estimated position and posture of the target object, which can closely combine the position and posture estimation with the path planning, realize the real-time grasping action, and significantly improve the success rate of the grasping task;

[0038] According to the grasping method in some embodiments of the present application, grasping the target object based on the planned grasping path can achieve more accurate and efficient grasping tasks in complex dynamic environments, which helps to solve the problems of low grasping efficiency, insufficient robustness, low grasping success rate, etc. in the prior art.

[0039] The features and advantages of the embodiments of the present invention will be further described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the embodiments of the present invention can be achieved and obtained by the structures and / or functions specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, purposes and advantages of the present application will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0041] Figure 1 is a flowchart of a 6D pose estimation method according to an exemplary embodiment of the present application.

[0042] Figure 2 is a schematic diagram of a system architecture according to an exemplary embodiment of the present application.

[0043] Figure 3 is another schematic diagram of a system architecture according to an exemplary embodiment of the present application.

[0044] Figure 4It is a schematic diagram of point pair features in a 6D pose estimation method according to an exemplary embodiment of the present application.

[0045] Figure 5 It is a schematic diagram of the structure of the robotic arm in an exemplary embodiment of the present application.

[0046] Figure 6 is a flow chart of a crawling method according to an exemplary embodiment of the present application.

[0047] Figure 7 It is another system architecture diagram according to an exemplary embodiment of the present application.

[0048] Figure 8 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.

[0049] Fig. 9 is a schematic structural diagram of a non-volatile computer-readable storage medium according to an exemplary embodiment of the present application.

[0050] Fig.10 is a schematic structural diagram of a device according to an exemplary embodiment of the present application.

[0051] Fig.11 It is a schematic diagram of the structure of a robot according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to better understand the present application, a more detailed description will be made of various aspects of the present application with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application, and are not intended to limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0053] It should be noted that in this specification, expressions such as "first", "second", etc. are only used to distinguish one feature from another feature, and do not represent any limitation on the features, especially do not represent any order of precedence. In the specification, references to "one embodiment", "embodiment", "example embodiment", "some embodiments", etc. indicate that the described embodiment may include a specific feature, structure or characteristic, but each embodiment may not necessarily include the specific feature, structure or characteristic. In addition, these phrases do not necessarily refer to the same embodiment.

[0054] It should be understood that expressions such as “include,” “including,” “have,” “contain” and / or “comprise” in this specification are open rather than closed expressions, which indicate the existence of stated features, elements and / or parts, but do not exclude the existence of one or more other features, elements, parts and / or combinations thereof.

[0055] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments in this application can be combined with each other. In addition, unless explicitly limited or contradictory to the context, the specific steps included in the method recorded in this application are not necessarily limited to the recorded order, but can be performed in any order or in parallel.

[0056] The features, principles and other aspects of the present application will be described in detail below in conjunction with the accompanying drawings.

[0057] The present application provides a 6D pose estimation method, which can at least improve or solve the problems described above. Figure 1 6D pose estimation method according to an exemplary embodiment of the present application. Figure 1 As shown, the present application provides a 6D pose estimation method 1000 including:

[0058] Step S1100, collecting RGB images and depth data;

[0059] Step S1200, extracting a depth region of interest of a target object;

[0060] Step S1300: construct a first point pair feature descriptor based on the depth region of interest, and match it with a pre-acquired second point pair feature descriptor to determine an estimated position and pose of the target object.

[0061] It should be understood that the steps shown in the 6D pose estimation method 1000 are not exclusive, and other steps may be performed before, after, or between any of the steps shown. In addition, some of the steps shown may be performed simultaneously or in different steps. Figure 1 The following is executed in the order shown. Figures 1 to 5 The above steps S1100 to S1300 are further described.

[0062] Step S1100, collecting RGB images and depth data.

[0063] Figure 2 is a schematic diagram of a system architecture according to an exemplary embodiment of the present application. Figure 1 and Figure 2 As shown, the RGB image and depth data of the target object can be collected by the collection module 110. During the collection, the target object and the environmental information around it will be captured.

[0064] In one possible implementation, the acquisition module 110 includes a combination of an RGB (color) sensor and a depth sensor, and the RGB sensor can be used to obtain color data, and the depth sensor can be used to obtain depth data. The RGB image provides color information of the scene, and each pixel has a color value in the RGB image, which is expressed as the intensity of the three channels of red, green, and blue (RGB). The depth data may include a depth image and / or non-image type depth information. The depth information provides distance information from each point in the scene to the depth sensor. For a depth image, each pixel has a depth value in the depth image, which can represent the straight-line distance from the point to the depth sensor.

[0065] As an example, only one pair of RGB images and depth data may be collected each time to provide enough information for immediate pose estimation and decision making.

[0066] As another example, multiple pairs of images may be collected from multiple viewpoints to capture features of an object from different directions to more fully understand the object or scene.

[0067] As another example, a series of images may be continuously collected to track the movement of a target object or scene changes. The number of images collected may be determined based on the temporal resolution and the processing capability of the device.

[0068] The acquisition module 110 can collect RGB images and depth data at the same time. Although these two types of data can be captured at the same time, they come from different sensor modules. Due to the influence of hardware design, accuracy differences between different sensor modules, environmental factors, etc., there may be offsets or angle differences between the depth data and the RGB image, resulting in geometric misalignment. This misalignment will cause each pixel in the RGB image to not completely match its corresponding pixel in the depth data, which will have a greater impact on accurate pose estimation and object recognition.

[0069] In response to the above problems, this application first geometrically aligns the collected RGB images and depth data to ensure the consistency of subsequent processing. Exemplarily, the geometric relationship between the depth sensor module and the RGB sensor module can be determined through a calibration process, including rotation and parameter translation. Specifically, a known calibration object (such as a checkerboard or dot pattern) can be used to determine the geometric relationship between the depth sensor and the RGB sensor. By taking multiple images of the calibration object and recording the position and posture of the calibration object in these images, the transformation matrix between the RGB sensor and the depth sensor is calculated.

[0070] The transformation matrix includes a rotation matrix and a translation vector, which converts each pixel coordinate in the depth data to the corresponding position in the RGB image coordinate system. In this way, each depth value can accurately correspond to the corresponding pixel point in the RGB image.

[0071] By geometrically aligning the RGB image and depth data through a calibration algorithm, the spatial consistency of the two image data can be ensured, which is beneficial to improving the accuracy and reliability of subsequent processing steps.

[0072] As an example, the acquisition module 110 includes an RGB-D camera, which has an RGB sensor and a depth sensor and can output an RGB image and a depth image. In addition, the RGB-D camera can automatically complete the geometric alignment of the RGB image and the depth data.

[0073] As another example, the acquisition module 110 includes a combination of a radar and a camera, and the radar can be used to obtain depth data. Further, the acquisition module 110 may include a combination of LiDAR (Light Detection and Ranging) and an RGB camera, or a combination of LiDAR and an RGB-D camera, or a combination of LiDAR, RGB camera, and RGB-D camera. LiDAR provides non-image type depth information, and the RGB camera provides an RGB image. In this example, an alignment operation such as that described above between the RGB image and the depth information is required.

[0074] Step S1200: extracting a depth region of interest of a target object.

[0075] Figure 3 FIG. 1 is another schematic diagram of a system architecture according to an exemplary embodiment of the present application. Figures 1 to 3 As shown, the first pixel coordinates of the target object can be first extracted based on the RGB image, and then the first pixel coordinates are mapped to the depth data to obtain the second pixel coordinates, and the depth value of each pixel corresponding to the second pixel coordinates is extracted. Finally, the first pixel coordinates and the depth value are combined to extract the depth region of interest of the target object, that is, the depth ROI (Region of Interest).

[0076] In one possible implementation, the collected RGB image is input into a pre-trained segmentation model. The segmentation model can analyze the image content and output a mask that marks the location of the target object with pixel-level accuracy. The goal of the segmentation model is to accurately identify and segment objects that need to be estimated in complex scenes to ensure the targetedness of subsequent processing, which helps to focus on specific areas and ignore irrelevant information. The first pixel coordinates of the target object are extracted through the segmentation model, and then the first pixel coordinates are mapped to the depth data, and these coordinates are used to extract the corresponding part in the depth map to obtain the second pixel coordinates.

[0077] It should be noted that the key to mapping the first pixel coordinates to the depth data is to achieve pixel-level alignment between the RGB image and the depth data. The calibrated camera parameters can be used to ensure that the two types of data can accurately match in the same coordinate system. By converting the first pixel coordinates in the RGB image to the second pixel coordinates in the depth data, it is helpful to accurately locate the position of the target object in three-dimensional space in the subsequent steps.

[0078] For each second pixel coordinate in the depth data, the depth value of each pixel corresponding thereto can be extracted. These depth values ​​provide distance information about the target object from the acquisition module 110. According to the first pixel coordinate obtained from the RGB image and the depth value extracted from the depth data, the depth ROI of the target object can be defined. The depth ROI not only contains the projection information of the target object on the two-dimensional plane, but also comes with the specific depth information of each pixel, which provides a data basis for subsequent posture estimation, three-dimensional reconstruction and other processing processes.

[0079] Exemplarily, the segmentation model is, for example, one or a combination of Mask R-CNN (Mask Region-based Convolutional Neural Networks), DeepLab, and Unet deep learning models. These models can be used to classify each pixel in the image, thereby effectively identifying the target object from a complex background and accurately segmenting its contour. The segmentation processing method of the present application can effectively locate the target object in a complex background, reduce noise interference, and improve the efficiency and accuracy of pose estimation.

[0080] Step S1300: construct a first point pair feature descriptor based on the depth region of interest, and match it with a pre-acquired second point pair feature descriptor to determine an estimated position and pose of the target object.

[0081] The first point pair feature descriptor, i.e., the first PPF (Point Pair Feature) descriptor, can be constructed based on the depth ROI. The second pixel coordinate and the depth value are converted into online point cloud data in three-dimensional space, and the online boundary point cloud of the target object is extracted from the online point cloud data; the initial point pair feature descriptor (i.e., the initial PPF descriptor) is constructed based on the online boundary point cloud; the initial PPF descriptor is fused with the point cloud gradient information to obtain the first PPF descriptor.

[0082] Figure 4 Schematic diagram of point pair features in a 6D pose estimation method according to an exemplary embodiment of the present application. Figure 4As shown in the figure, point a and point b are two reference points selected in the point cloud. Point a and point b form a point pair, which is used to construct the feature descriptor. n1 and n2 represent the surface normal vectors at point a and point b respectively. The normal vector is perpendicular to the surface and can describe the orientation of the surface. The PPF descriptor includes F1, F2, F3 and F4. F1 represents the vector from point a to point b, and F2, F3 and F4 represent the angle relationship between the vector, normal vector n1 and normal vector n2 respectively. These angles are the key components for constructing the PPF descriptor and can describe the orientation of the point pair relative to the surface normal. These PPF descriptors can capture the characteristics of local geometric shapes in the point cloud.

[0083] By analyzing the relative positions and orientations between point pairs, as well as their relationship to surface normals, it is helpful to extract geometric features that are helpful in identifying and matching objects.

[0084] In a possible implementation, after the step of extracting the depth ROI of the target object from the RGB image and the depth data is completed, the information of the second pixel coordinates and the corresponding depth value can be converted into point cloud data in a three-dimensional space. The two-dimensional coordinates and depth values ​​of each pixel are mapped to a three-dimensional coordinate system to form a series of points with (x, y, z) coordinates. These points together constitute the representation of the target object in a three-dimensional space.

[0085] Based on the three-dimensional point cloud data obtained above, a suitable point pair combination is selected to construct the first PPF descriptor. The PPF descriptor is a feature vector used to describe the geometric relationship between point pairs, which includes information such as the distance, direction, and normal angle between the point pairs. The purpose of constructing the PPF descriptor is to efficiently find similar point pair combinations in the subsequent matching process, and then infer the position and posture of the target object.

[0086] In a possible implementation, since point cloud data often contains noise points, these noise points may come from sensor errors or environmental interference. After the point cloud data is generated, it needs to be preprocessed. Noise can be removed by applying statistical filters or local outlier detection methods to improve the quality of the point cloud.

[0087] For example, in order to reduce the amount of calculation and improve the processing speed, the point cloud data can also be downsampled, that is, the number of points in the point cloud can be reduced. For example, by using voxel grid filtering, random sampling and other methods, the density of the point cloud can be effectively reduced while retaining the overall structure of the point cloud.

[0088] For example, in addition to denoising and downsampling, additional filtering processing may be performed on the point cloud data, such as plane fitting filtering, to remove background or unnecessary plane structures and further highlight the features of the target object.

[0089] In a possible implementation, based on the point cloud data preprocessed by denoising, downsampling, filtering, etc., a geometric method (such as convex hull algorithm, edge detection, etc.) is used to extract the boundary point cloud of the target object. The boundary point cloud can reflect the contour information of the target object, which contains the key features of the shape of the target object. The information contained in the boundary point cloud is more critical than that in the complete point cloud, which can effectively improve the accuracy of pose estimation.

[0090] Based on the extracted boundary point cloud, appropriate point pairs are selected to construct the initial PPF descriptor. The initial PPF descriptor is also used to describe the geometric relationship between point pairs in the boundary point cloud, but it focuses more on capturing the characteristics of the object edge, which is particularly important for improving the accuracy of pose estimation.

[0091] It is easy to understand that compared with the traditional point cloud processing methods that cannot effectively distinguish between target objects and background point clouds, affect the subsequent pose estimation accuracy, large amount of calculation for the entire point cloud data and insufficient real-time performance, this application uses boundary point cloud extraction, initial PPF descriptor construction and preprocessing techniques to reduce the amount of data while ensuring the key features of pose estimation, thereby improving the estimation accuracy, data quality and processing efficiency.

[0092] The inventors of this application further discovered that the PPF descriptor is limited in its ability to describe objects of complex shapes during the matching process, resulting in insufficient matching accuracy when estimating the pose of irregular objects. To address this problem, this application innovatively integrates point cloud gradient information into the descriptor to form the first point pair feature descriptor (i.e., the first PPF descriptor), which effectively enhances the sensitivity of the descriptor to changes in the surface shape of the object, improves the robustness and accuracy of the matching, and achieves accurate recognition of objects of complex shapes.

[0093] In one possible implementation, the fusion of the initial PPF descriptor and the point cloud gradient information includes calculating the normal vector of each point in the online point cloud data and its neighborhood points; calculating the gradient vector based on the normal vector and the position of the point in the online point cloud data; weighting the relative positions and directions in all the initial PPF descriptors based on the strength of the gradient vector to highlight point pairs with large geometric changes, and / or mapping the gradient vector into a feature space and combining it with the feature space of the initial PPF descriptor to form a new high-dimensional feature space.

[0094] It should be noted that the point cloud gradient can be understood as the geometric change rate of the point cloud in space, that is, the degree of change of the coordinates of each point relative to its neighborhood. The point cloud gradient is usually expressed in the form of a 3D vector, describing the local changes in the surface geometry of the point cloud.

[0095] For example, for each point in the point cloud, the local neighborhood of the point can be determined first. For example, a nearest neighbor algorithm (including k nearest neighbor or radius search) is used to obtain the neighborhood point set N (p i ).

[0096] For each point p i Neighborhood N(p i ), the normal vector n can be calculated by fitting a local plane or using methods such as principal component analysis (PCA) i The normal vector represents the orientation of the local surface at that point.

[0097] When calculating the point cloud gradient, consider p i The change of a point relative to its neighborhood is calculated using the change of the point and the normal vector to satisfy:

[0098]

[0099] In the formula, G(p i ) is point p i The gradient vector, N(p i ) is the point containing the point p i A point p surrounded by a set of points i Neighborhood, n i and n j They are point p i and the neighborhood point p j The normal vector, (p i -p j ) is point p i and the neighborhood point p j The coordinate difference between .

[0100] Point cloud gradient calculation reflects the changing trend of points in the local neighborhood. By calculating the gradient information for each point, it can be combined with the initial PPF descriptor, making the descriptor more discriminative for objects with complex shapes.

[0101] For example, in order to unify the scale, the calculated gradient vector may be normalized so that the gradient vector of each point has unit length:

[0102]

[0103] In the formula, G′(p i ) is the normalized gradient vector. The strength of the gradient vector can be expressed by calculating its norm, that is, ||G(p i )||. The normalized gradient can be better used to describe the sub-computation and subsequent matching process.

[0104] Through the above steps, the initial PPF descriptor can be effectively fused with the point cloud gradient information, thereby enhancing the ability to describe the geometric features of the target object. This fusion method not only improves the accuracy of pose estimation, but also enhances the robustness to complex shapes and surface textures. Furthermore, the method of weighted processing and feature space combination can highlight point pairs with large geometric changes. This design helps to accurately identify and locate target objects in complex environments.

[0105] As an example, the relative positions and directions in all initial PPF descriptors may be weighted according to the strength of the gradient vector.

[0106] As another example, after weighting the relative positions and directions in all initial PPF descriptors according to the strength of the gradient vector, the gradient vector can also be mapped to a feature space and combined with the feature space of the initial PPF descriptor to form a new high-dimensional feature space. Specifically, the gradient vector is mapped to a feature space, for example, by normalization or other feature transformation methods, and the mapped gradient vector is concatenated with the feature vector of the initial PPF descriptor to form a new high-dimensional feature vector. The new high-dimensional feature vector can be expressed as: New PPF = [InitialPPF, Gradient Feature].

[0107] As another example, the relative position and direction in the initial PPF descriptor may not be weighted, but the gradient vector may be directly mapped to a feature space and combined with the feature space of the initial PPF descriptor to form a new high-dimensional feature space.

[0108] High-dimensional feature space can contain more information, not only the geometric relationship between point pairs, but also gradient information, normal vectors, etc. This additional information helps to more comprehensively describe the geometric features of the target object. The new high-dimensional feature vector New PPF has higher discrimination and can better distinguish different target objects or different postures of the same object, which helps to improve the accuracy of recognition and positioning. In addition, the high-dimensional feature space is not limited to data of a single modality, but can also fuse information from different modalities, such as RGB images, depth data, infrared images, etc. Cross-modal fusion can provide a more comprehensive feature description, make up for the shortcomings of a single modality, and further improve the performance of the system.

[0109] In a possible implementation, the first PPF descriptor may be matched with a pre-acquired second PPF descriptor to determine the estimated position and pose of the target object. The second PPF descriptor may be pre-acquired in an offline stage.

[0110] It should be noted that the technical terms "online stage" and "offline stage" described in the context of this application are used to describe different stages of data processing and calculation. The online stage can refer to the stages of real-time data processing, pose estimation, strategy formulation, action execution, etc. Since the online stage requires fast response, the calculation needs to be continuously optimized to meet the real-time requirements, so it consumes more computing resources.

[0111] The offline phase can refer to the pre-processing phase, where data is pre-processed before actual application, such as calculating the boundary descriptor of a 3D model; pre-calculating data that will be used in multiple scenes or multiple runs, such as the geometric features and physical properties of objects; and storing the pre-calculated results in a database or file system for quick retrieval and use when needed. In the offline phase, algorithms can be optimized, such as selecting appropriate data structures, optimizing search algorithms, etc., to ensure efficient execution in the online phase. Offline computing allows the system to use computing resources during off-peak hours, thereby avoiding competing for resources with user operations during peak hours.

[0112] The online phase corresponds to Figure 3 The light shaded area in the figure (the area within the rectangular dashed box) corresponds to the offline stage. Figure 3 The dark shadow area. Obviously, the light shadow area and the dark shadow area have overlapping and intersecting parts, which refer to the operation steps shared by the online stage and the offline stage. It can be understood that the online stage includes boundary extraction, calculation of PPF and gradient, and PPF matching steps, and the offline stage can also include boundary extraction, calculation of PPF and gradient, and PPF matching steps. The operation steps shared by the online stage and the offline stage may not be shared at the same time. For example, the offline stage first performs boundary extraction, PPF calculation and matching steps, and the online stage performs the above steps later.

[0113] The technical terms "online point cloud data" described in the context of this application refer to the point cloud data generated in the online stage, "online boundary point cloud" refers to the boundary point cloud extracted in the online stage, "offline point cloud data" refers to the point cloud data generated in the offline stage, and "offline boundary point cloud" refers to the boundary point cloud extracted in the offline stage.

[0114] Exemplarily, the method for obtaining the second point pair feature descriptor in the offline stage includes establishing a 3D model of the target object and voxelizing the 3D model. Then, the boundary information of the 3D model is extracted, and the boundary information is converted into an offline boundary point cloud. Based on each point in the offline boundary point cloud, the second PPF descriptor is calculated. The model boundary descriptor (the second PPF descriptor) can be stored in a hash table and applied in the PPF matching step.

[0115] In one possible implementation, an image containing depth information can be obtained by, for example, an RGB-D camera, and a 3D point cloud model can be directly generated based on the depth image combined with the camera intrinsic parameters. For example, the depth images of multiple perspectives can be fused after pose alignment, and then the fused point cloud can be denoised and optimized to form a dense point cloud.

[0116] The inventors of this application have found that the field of view of RGB-D cameras is usually small, requiring multiple scans to cover a large area, and is sensitive to lighting conditions, easily failing in strong light or reflective surfaces, and having difficulty handling occlusion and dynamic scenes. In addition, the accuracy of depth sensors is limited, which will affect the reconstruction quality of 3D models.

[0117] In another possible implementation, the present application uses the vision unit 210 in combination with the three-dimensional reconstruction technology to generate a 3D model to solve the problems of lighting, occlusion, dynamic scenes and camera accuracy that cannot be solved by RGB-D cameras. The inventors of the present application also found that since the reconstruction results of the three-dimensional reconstruction technology will only contain relative scale information but not absolute scale information, the two-dimensional image collected by the vision unit 210 cannot provide the absolute scale information. Therefore, the solution of combining the three-dimensional reconstruction algorithm with the vision unit 210 cannot reflect the true size and / or actual posture of the target object in three-dimensional space.

[0118] To address this problem, the method of establishing a 3D model of a target object in the present application includes using a visual unit 210 to capture an RGB multi-view image of the target object; obtaining a sparse point cloud reconstructed from the RGB multi-view image through a three-dimensional reconstruction algorithm; providing absolute scale information to the three-dimensional reconstruction algorithm to correct the scale of the sparse point cloud; obtaining a dense point cloud based on the corrected sparse point cloud; using a pre-trained neural network model to process the RGB multi-view image to determine the position of the target object in the RGB multi-view image; combining the posture information of the visual unit and the position information of the target object in the RGB multi-view image to calculate the position range of the target object in three-dimensional space; and screening out point clouds within the above-mentioned position range in the dense point cloud to obtain the 3D model.

[0119] Exemplarily, in order to overcome the drawbacks of RGB-D depth cameras in three-dimensional reconstruction and reduce the amount of image data processing in the offline stage, the vision unit 210 may select a sensor that only provides two-dimensional image information, such as a combination of one or more of an RGB camera, a SLR camera, a mirrorless camera, an industrial camera, a webcam, and a mobile phone lens.

[0120] The technical term "scale information" described in the context of this application refers to data that can provide actual physical dimensions or sizes. "Absolute scale information" refers to scale information that directly corresponds to actual dimensions in the real world. "Relative scale information" refers to the proportional relationship between the parts of an object, but does not involve actual physical dimensions.

[0121] In order to further solve the above problems, the present application provides additional absolute scale information in the offline stage to correct the scale of the sparse point cloud.

[0122] Figure 5 Schematic diagram of the structure of the robot arm in the exemplary embodiment of the present application. Figure 5 As shown, a robot arm 120 may be introduced to obtain absolute scale information. The robot arm 120 includes a gripper 121, a base 122 and a control unit 123.

[0123] As an example, the method for providing absolute scale information includes first calibrating the robot 120 to determine the position (including position and posture) of the gripper 121 of the robot 120 in the coordinate system of the base 122 of the robot 120; then controlling the robot 120 to move from the initial position to the reference position, which is a predetermined position of known coordinates. During the movement of the robot 120 to the reference position, the visual unit 210 collects multi-view images, and the control unit 123 of the robot 120 records the position of the gripper 121 during image acquisition; then measuring the actual distance S from the initial position of the robot 120 to the reference position, and establishing a scale reference based on the recorded position of the gripper 121; comparing the above distance S with the distance L estimated from the sparse point cloud by the three-dimensional reconstruction algorithm to obtain a size factor. The absolute scale information provided in this application includes a size reference and a size factor.

[0124] The scale benchmark provides a reference point or object of known size that can be used to determine the actual size of other measurements. In 3D reconstruction and image analysis, the scale benchmark also helps eliminate scale ambiguity and ensures that the reconstructed model and analysis results are accurate in size. The scale factor is used to adjust and convert the relative sizes between different systems and data sets. In 3D model construction, the scale factor adjusts the size of the model to match the actual observed data or expected results. The scale benchmark and scale factor work together to achieve accurate scale correction of the 3D model, so that the 3D point cloud model can truly reflect the target object (including posture and size) in the real world.

[0125] Exemplarily, the first PPF descriptor is generated in the online stage, and the second PPF descriptor is generated in the offline stage. The second PPF descriptor can be directly called during the matching process in the online stage, thereby greatly reducing the burden of online calculation and improving the real-time performance and response speed of the system.

[0126] In the online stage, the boundary point cloud descriptor of the target object collected in real time (which can be fused with point cloud gradient information) can be matched with the model boundary descriptor generated in the offline stage. By calculating the boundary point cloud descriptor of the target object offline, the matching and pose estimation process in the online stage can be accelerated, effectively reducing the time consumption of online calculation, realizing efficient 6D pose estimation, and helping to meet real-time requirements.

[0127] Exemplarily, a feature matching algorithm (PPF matching algorithm) is used to match the first PPF descriptor and the second PPF descriptor to obtain a matching pair; by calculating point pair features and introducing a voting mechanism, a weight is assigned to each matching pair to determine the candidate pose. The voting information of all matching pairs is aggregated, and the voting mechanism can screen out the most likely pose candidate by accumulating support scores to determine the estimated pose.

[0128] It can be understood that the above-mentioned estimated pose can be composed of two parts, namely position (Position) and attitude (Orientation). Position refers to the x, y, and z coordinates of the object in three-dimensional space, that is, the straight-line distance of the object relative to a reference point or a reference coordinate system. The attitude refers to the rotation state of the object in space, which may include three angle descriptions of roll (Roll), pitch (Pitch), and yaw (Yaw). The estimation of position and attitude together constitutes 6D pose estimation. The 6D pose estimation of the present application can provide a complete description, so that the position and orientation of the target object can be accurately reconstructed in space.

[0129] Exemplarily, the method for determining the estimated pose of the target object further includes adjusting the pose of the target object to align it with the point cloud data captured in the online stage, so as to output the precise pose of the target object in the coordinate system of the acquisition module 110. According to the coordinate system of the acquisition module 110, by adjusting the pose online, it is possible to adapt to the dynamically changing environment and the movement of the target object, thereby improving the accuracy and robustness of the pose estimation.

[0130] In some embodiments, after voting to obtain a preliminary estimated pose, the ICP (Iterative Closest Point) algorithm can be used to fine-tune the initial pose of the target object. By minimizing the Euclidean distance between the target object point cloud and the reference point cloud, ICP gradually adjusts the translation and rotation parameters of the target object to achieve the best alignment of the two point clouds. During the optimization process, the dynamic iteration of the closest point matching relationship improves the accuracy and stability of the pose estimation, and finally outputs the precise pose in the optimized camera coordinate system.

[0131] On the other hand, the present application provides a grasping method based on 6D pose estimation. Figure 6 is a flow chart of a method for grasping according to an exemplary embodiment of the present application. Figure 6 As shown, the crawling method 2000 provided by the present application includes:

[0132] Step S2100, performing real-time grasping path planning according to the estimated pose of the target object in any implementation of the above 6D pose estimation method;

[0133] Step S2200: grasping the target object based on the grasping path.

[0134] It should be understood that the steps shown in the crawling method 2000 are not exclusive, and other steps may be performed before, after, or between any of the steps shown. In addition, some of the steps shown may be performed simultaneously or in different steps. Figure 6 The above steps S2100 to S2200 are further described below.

[0135] Step S2100: Perform real-time grasping path planning according to the estimated pose of the target object in any implementation of the above 6D pose estimation method.

[0136] Figure 7 FIG. 1 is another schematic diagram of a system architecture according to an exemplary embodiment of the present application. Figure 7 As shown, after the pose of the target object is successfully estimated by the 6D pose estimation method, real-time grasping path planning is performed next. In one possible implementation, a set of candidate grasping points can be generated based on the shape and surface characteristics of the target object. The candidate grasping points are located in the stable contact area of ​​the object, such as a plane, groove or edge. Each candidate grasping point is then evaluated and a grasping point is selected considering its stability and feasibility. Exemplarily, the evaluation indicators may include grasping force distribution, contact area, friction coefficient, etc.

[0137] Based on the selected grasping point, a path from the current position of the robot 120 to the grasping point can be generated. Path planning takes into account the kinematic constraints and workspace limitations of the robot 120 to ensure the feasibility and safety of the path. During the path planning process, obstacles in the surrounding environment must also be determined so that the robot 120 does not collide during movement. Obstacle avoidance can be achieved using path planning methods such as rapidly expanding random trees.

[0138] Step S2200: grasping the target object based on the grasping path.

[0139] In a possible implementation, during the grasping process, the robot arm 120 grasps the target object in real time based on a pre-planned grasping path.

[0140] During the real-time grasping process, the grasping status and environmental changes can be monitored in real time. When abnormal conditions are detected, such as posture deviation or obstacle movement, adjustments can be made immediately. For example, closed-loop control strategies can be used to adjust the motion parameters of the robot arm based on real-time feedback to ensure the accuracy and stability of the grasping action.

[0141] The robot arm 120 approaches the target object along the planned path until it reaches the predetermined grasping point. When the robot arm 120 reaches the grasping point, it performs a grasping action, which may include steps such as opening the gripper 121, contacting the object, and closing the gripper 121. After the grasping is completed, a sensor (such as a force sensor or a tactile sensor) is used to confirm whether the grasping is successful.

[0142] If the grasping fails, the path can be replanned or other grasping points can be selected according to the grasping situation.

[0143] If the grasping is successful, the robot arm 120 can carry the object to the specified location. During the handling process, real-time monitoring and path optimization are continued to ensure the safety and efficiency of the handling. After reaching the target location, the robot arm 120 releases the target object it grasped, completing the grasping task.

[0144] Through the above steps, pose estimation and path planning are closely integrated to achieve efficient grasping action. This integrated method can reduce the number of times the system frequently adjusts pose estimation and path planning in dynamic or complex environments, which is conducive to taking into account both the accuracy of pose estimation and the flexibility of path planning, thereby achieving a significant improvement in real-time performance and grasping success rate. It effectively solves the low grasping efficiency caused by the separation of pose estimation and grasping path planning in existing robot grasping systems.

[0145] Through the above steps, the whole process from 6D pose estimation to target object grasping can be realized, ensuring the efficiency and reliability of the grasping task. According to the grasping method of this application, the average grasping processing time can be shortened to less than 0.6s, and the response speed of the grasping task is significantly improved.

[0146] On the other hand, the present application provides an electronic device based on 6D pose estimation. Figure 8 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 8 As shown, the electronic device 300 based on 6D pose estimation includes a processing unit 310 and a storage unit 320 connected thereto. The storage unit 320 is used to store computer programs. In one possible implementation, the processing unit 310 is used to call the computer program stored in the storage unit 320 to execute the pose estimation method in any of the above implementations. In another possible implementation, the processing unit 310 is used to call the computer program stored in the storage unit 320 to execute the grasping method in any of the above implementations.

[0147] The processing unit 310 serves as the control center of the electronic device 300. It can be a general-purpose central processing unit (CPU) or other processors, such as a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.; or any conventional general-purpose processor, such as a microprocessor, etc.

[0148] As an example, the processing unit 310 may include one or more CPUs, such as CPU0 and CPU1, and the storage unit 320 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0149] In a possible implementation, the storage unit 320 may exist independently of the processing unit 310. The storage unit 320 may be connected to the processing unit 310 via, for example, a bus 330, and is used to store data, instructions, or program codes. When the processing unit 310 calls and executes the instructions or program codes stored in the storage unit 320, the method provided in the embodiment of the present application can be implemented.

[0150] In another possible implementation, the storage unit 320 may also be integrated with the processing unit 310 .

[0151] Exemplarily, the electronic device 300 can also include a communication interface 340 for connecting to other devices through a communication network, and the communication network can be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 340 can include a receiving unit for receiving data and a sending unit for sending data.

[0152] The bus 330 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0153] Need to explain, Figure 8 The structure shown in the figure does not constitute a limitation on the electronic device 300. Figure 8 In addition to the components shown, the electronic device 300 may include Figure 8 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0154] Optionally, Figure 8 The electronic device 300 shown in FIG. 1 is a chip.

[0155] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the above-mentioned processing unit 310 or the function of the processing unit 310. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the model training device or the natural language processing device in the above-mentioned embodiment according to the program code stored in the external memory.

[0156] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, in the drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0158] In addition, the present invention also proposes the following embodiments:

[0159] A non-volatile computer-readable storage medium. Fig. 9 is a schematic diagram of the structure of a non-volatile computer-readable storage medium according to an exemplary embodiment of the present application. Fig. 9 As shown, the readable storage medium 400 is composed of a semiconductor device 410 and a peripheral circuit 420. The readable storage medium 400 stores a computer program, and when the computer program is executed, it can execute the 6D pose estimation method in any of the above embodiments, and / or execute the grasping method in any of the above embodiments.

[0160] Computer-readable storage medium 400 includes a device capable of temporarily or permanently storing instructions and data for operating in a specific manner. As used herein, the term "computer-readable storage medium" itself does not cover transient electrical or electromagnetic signals (e.g., on a carrier wave propagating through the medium). Therefore, computer-readable storage medium 400 can be considered to be tangible and non-transitory.

[0161] Illustratively, non-limiting examples of non-transitory tangible computer-readable storage media may include, but are not limited to, non-volatile memory, such as flash memory or read-only memory (ROM), volatile memory, such as static random access memory (RAM) or dynamic access memory, buffer memory, cache memory, optical storage media, magnetic storage media and devices, network accessible or cloud storage, and other types of storage and / or any suitable combination thereof.

[0162] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0163] A person of ordinary skill in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium 400. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0164] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium 400. As an example, the computer-readable storage medium 400 may also be a disk, an optical disk, or a solid-state hard disk, etc.

[0165] In another aspect, the present application provides a 6D pose estimation device. 10 is a schematic diagram of the structure of the device according to an exemplary embodiment of the present application. Fig.10 As shown, the device 500 includes an electronic device 300 as in any of the embodiments described above, and at least one sensor 510, wherein the sensor 510 is used to capture RGB images and depth data of the target object for the electronic device 300 to perform pose estimation and grasping path planning.

[0166] Exemplarily, the sensor 510 includes an RGB sensor module 511 and a depth sensor module 512. The RGB sensor module 511 can obtain color data, and the RGB image provides color information of the scene. The depth sensor module 512 can obtain depth data, and the depth data provides distance information from each point in the scene to the depth sensor, and each pixel has a depth value in the depth data.

[0167] Exemplarily, the sensor 510 includes an RGB-D camera, or a combination of LiDAR and an RGB camera, or a combination of LiDAR and an RGB-D camera, or a combination of LiDAR, an RGB camera, and an RGB-D camera.

[0168] On the other hand, the present application provides a robot based on 6D pose estimation. Fig.11 is a schematic diagram of the structure of a robot according to an exemplary embodiment of the present application. Fig.11 As shown, the robot 900 includes a collection module 110, a mechanical arm 120, an extraction module 130 and a posture estimation module 140. It should be noted that, Fig.11 The structure shown in the figure does not constitute a limitation on the robot 900, except Fig.11 In addition to the components shown, the robot 900 may include Figure 8 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0169] In one possible implementation, the robot 900 has learning and adaptive capabilities, and can continuously optimize control strategies based on historical operating data and environmental changes to improve operating efficiency and accuracy.

[0170] Exemplarily, the acquisition module 110 can be used to acquire RGB images and depth data; the extraction module 120 can be used to extract the depth ROI of the target object; the pose estimation module 140 includes a first PPF descriptor and a second PPF descriptor, which can be used to determine the estimated pose of the target object. For example, the first PPF descriptor can be constructed based on the depth ROI, the second PPF descriptor is obtained in advance, and the estimated pose of the target object is determined by matching the first PPF descriptor and the second PPF descriptor.

[0171] For example, the first point PPF descriptor is fused with point cloud gradient information. By introducing the gradient information of the point cloud as a new descriptor and analyzing the surface gradient of the point cloud, the geometric shape details of the object can be better described, the matching accuracy is improved, and the recognition ability of complex shape objects is enhanced.

[0172] Exemplarily, the acquisition module 110 includes an RGB-D camera, or a combination of LiDAR and an RGB camera, or a combination of LiDAR and an RGB-D camera, or a combination of LiDAR, an RGB camera, and an RGB-D camera.

[0173] In a possible real-time manner, the robot 900 further includes a control module 150. The control module 150 can plan the grasping path of the robot arm 120 according to the determined estimated posture, and / or send a grasping instruction to the robot arm 120. The control module 150 can also receive data transmitted by the vision unit 210 and the robot arm 120, and output a point cloud model using multiple algorithms.

[0174] In a possible real-time mode, the robot 900 further includes an offline processing module 160, which is capable of constructing a 3D point cloud of the target object in an offline stage. The offline processing module 160 can pre-calculate the second PPF descriptor to avoid resource competition with online calculations. The offline processing module 160 allows the system to use computing resources during non-peak hours, thereby avoiding competing for resources with user operations during peak hours.

[0175] In a possible real-time mode, the robot 900 further includes a visual unit 210. The visual unit 210 can be used to collect multi-view images for the offline processing module 160 to construct a 3D point cloud model of the target in the offline stage. It should be noted that in order to reduce the amount of image data processing in the offline stage, and to ensure the measurement accuracy of objects at long distances or on smooth and transparent surfaces, the visual unit 210 selects a sensor that only provides two-dimensional image information, such as a combination of one or more of an RGB camera, a SLR camera, a mirrorless camera, an industrial camera, a camera, and a mobile phone lens.

[0176] When the visual unit 210 acquires images from multiple perspectives, the position of the visual unit 210 can be determined by the position of the gripper 121 of the mechanical arm 120. For example, the visual unit 210 is an RGB camera, and the mechanical arm 120 can move according to the instructions issued by the robot 900, and the mechanical arm 120 drives the gripper 121 to grab the RGB camera; or, the RGB camera is pre-installed on the gripper 121 so that the RGB camera can move together with the RGB camera.

[0177] As an example, the position information of the visual unit 210 during image acquisition recorded by the control unit 123 of the robot 120 can be used. The geometric relationship between the image plane and the three-dimensional space is established using the internal and external reference information of the visual unit 210, and the two-dimensional coordinates of the target in the image are converted into three-dimensional rays in the coordinate system of the visual unit 210. The three-dimensional rays of multiple perspectives are combined to calculate the position range of the target in the three-dimensional space using a triangulation method.

[0178] As an example, the target object includes a handle of a kitchen utensil, such as a frying basket handle, a cup handle, a frying pan handle, etc. As another example, the target object may be food or a specific item, such as an apple, milk in a packaging box, French fries, a plate, a seasoning box or a seasoning bottle, etc.

[0179] The robot 900 of the present application can receive the grasping instruction issued by the control module 150 through the mechanical arm 120, and grasp the target object according to the grasping path. The robot 900 closely combines the posture estimation with the path planning, realizes the efficient grasping movement, effectively completes the target grasping task in the complex dynamic scene, and solves the problems of low computational efficiency, insufficient robustness, low grasping success rate, etc. in the prior art.

[0180] The robot 900 of the present application can reduce the interference of noisy point clouds on pose estimation and improve the system's grasping success rate through refined point cloud preprocessing and boundary extraction. Boundary extraction ensures that the system can focus on key features, thereby reducing interference factors in complex scenes and achieving a significant increase in the grasping success rate of the robot 900 in complex environments.

[0181] The above description is only an implementation method of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of protection involved in the present application is not limited to the technical solution formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the technical concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A 6D pose estimation method, characterized in that: The estimation method includes: Collect RGB images and depth data; Extract the depth region of interest of the target object; A first point pair feature descriptor is constructed based on the depth region of interest, and matched with a pre-acquired second point pair feature descriptor to determine an estimated position and pose of the target object.

2. The estimation method according to claim 1, characterized in that The first point pair feature descriptor is fused with point cloud gradient information.

3. The estimation method according to claim 2, characterized in that The method for extracting the depth region of interest of the target object comprises: Extracting a first pixel coordinate of the target object based on the RGB image; Mapping the first pixel coordinate to the depth data to obtain a second pixel coordinate; Extracting a depth value of each pixel corresponding to the second pixel coordinate from the depth data; The first pixel coordinates and the depth value are combined to extract a depth region of interest of the target object.

4. The estimation method according to claim 3, characterized in that: The method for constructing the first point pair feature descriptor includes: Converting the second pixel coordinates and the depth value into online point cloud data in a three-dimensional space, and extracting an online boundary point cloud of the target object from the online point cloud data; Constructing the initial point pair feature descriptor based on the online boundary point cloud; The initial point pair feature descriptor is fused with the point cloud gradient information to obtain the first point pair feature descriptor.

5. The estimation method according to claim 4, characterized in that: The pre-acquired second point pair feature descriptor includes acquiring the second point pair feature descriptor in an offline stage, and the method for acquiring the second point pair feature descriptor in the offline stage includes: Establishing a 3D model of the target object; Extracting boundary information of the 3D model and converting the boundary information into an offline boundary point cloud; Based on each point in the offline boundary point cloud, the second point pair feature descriptor is calculated and obtained.

6. The estimation method according to claim 5, characterized in that: The method for determining the estimated position and pose of the target object comprises: Using a feature matching algorithm to match the first point pair feature descriptor with the second point pair feature descriptor to obtain a matching pair; Assigning a weight to each of the matching pairs through a voting mechanism; The voting information of all the matching pairs is aggregated to determine the estimated pose.

7. The estimation method according to claim 4, characterized in that: The fusion of the initial point pair feature descriptor and the point cloud gradient information includes: Calculate the normal vector of each point and its neighboring points in the online point cloud data; Calculate a gradient vector according to the normal vector and the position of the point in the online point cloud data; weighting the relative positions and directions of all the initial point pairs in the feature descriptors according to the strength of the gradient vector to highlight the point pairs with large geometric changes, and / or The gradient vector is mapped into a feature space and combined with the feature space of the initial point pair feature descriptor to form a new high-dimensional feature space.

8. The estimation method according to claim 2 or 7, characterized in that: The calculation for obtaining the point cloud gradient information includes: Among them, G(p i ) is point p i The gradient vector, N(p i ) is the point containing the point p i A point p surrounded by a set of points i Neighborhood, n i and n j They are point p i and the neighborhood point p j The normal vector, (p i -p j ) is point p i and the neighborhood point p j The coordinate difference between .

9. The estimation method according to claim 5, characterized in that: The method for establishing the 3D model of the target object includes: Using a visual unit to capture an RGB multi-view image of the target object; Obtaining a sparse point cloud reconstructed from the RGB multi-view image by a three-dimensional reconstruction algorithm; Providing absolute scale information to the 3D reconstruction algorithm to correct the scale of the sparse point cloud; Obtain a dense point cloud based on the corrected sparse point cloud; Processing the RGB multi-view image using a pre-trained neural network model to determine a position of the target object in the RGB multi-view image; Calculating the position range of the target object in three-dimensional space by combining the posture information of the visual unit and the position information of the target object in the RGB multi-view image; and Point clouds located within the position range are screened out from the dense point cloud to obtain the 3D model.

10. The estimation method according to claim 1, characterized in that: The first point pair feature descriptor is generated in an online stage, and the second point pair feature descriptor is generated in an offline stage, wherein the second point pair feature descriptor generated in the offline stage is directly called in the matching process of the online stage.

11. The estimation method according to claim 1, characterized in that: The depth data is acquired by radar, and before extracting the depth region of interest, the estimation method further includes: The RGB image and the depth data are geometrically aligned.

12. The estimation method according to claim 10, characterized in that: The method of acquiring the RGB image and the depth data of the target object through the acquisition module and determining the estimated pose of the acquisition module also includes: The pose of the target object is adjusted to align with the point cloud data of the target object captured in the online stage, so as to output the precise pose of the target object in the coordinate system of the sensor.

13. A grasping method based on 6D pose estimation, characterized in that: The crawling method includes: Performing real-time grasping path planning according to the estimated pose of the target object according to any one of claims 1 to 12; The target object is grasped based on the grasping path.

14. An electronic device based on 6D pose estimation, characterized in that: The electronic device comprises: a processing unit and its connected storage unit; The storage unit is used to store computer programs; The processing unit is used to call the computer program stored in the storage unit to execute the estimation method according to any one of claims 1 to 12, or to execute the grasping method according to claim 13.

15. A device based on 6D pose estimation, comprising the electronic device as claimed in claim 14, and at least one sensor, wherein the sensor is used to capture RGB images and depth data of a target object for the electronic device to perform pose estimation and grasping path planning.

16. A robot based on 6D pose estimation, characterized in that: The robot comprises: Acquisition module, used to collect RGB images and depth data; An extraction module is used to extract the depth region of interest of the target object; A pose estimation module, comprising a first point pair feature descriptor and a second point pair feature descriptor, for determining an estimated pose of the target object, The first point pair feature descriptor is constructed based on the depth region of interest, the second point pair feature descriptor is pre-acquired, and the estimated position and posture of the target object is determined by matching the first point pair feature descriptor with the second point pair feature descriptor.

17. The robot according to claim 16, characterized in that: The first point pair feature descriptor is fused with point cloud gradient information.

18. The robot according to claim 16, characterized in that: Also includes: The control module plans a grasping path according to the determined estimated posture and / or issues a grasping instruction.

19. The robot according to claim 18, characterized in that Also includes: The robotic arm receives the grasping instruction sent by the control module and grasps the target object according to the grasping path.

Images