Article grabbing pose determination method, device and equipment based on multi-view point cloud fusion

Abstract

The invention provides an object grabbing pose determination method, device and equipment based on multi-view point cloud fusion, and relates to the technical field of robots. The method comprises the steps that point cloud data of a plurality of articles at different visual angles are acquired, and the point cloud data comprise three-dimensional coordinates of a plurality of points and are used for reflecting geometrical shapes and spatial distribution of the articles; fusing the point cloud data of the plurality of articles at different visual angles to obtain fused point cloud data of the plurality of articles; performing feature fusion on the point cloud data of the plurality of articles at different visual angles through a point cloud fusion network model to obtain visual angle fusion point cloud features of the plurality of articles; performing point cloud instance segmentation on the fused point cloud data of the plurality of articles through an instance segmentation network model according to the view angle fused point cloud features of the plurality of articles to obtain instance point cloud data corresponding to each article; and according to the instance point cloud data corresponding to each article, determining the grabbing pose of each article. According to the method, the accuracy of the determined grabbing pose can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method, device and equipment for determining the position and posture of an object grasped by fusion of multi-view point clouds. Background Art

[0002] With the development of robot technology, when a product is assembled by a robot, the robot can automatically grab the various parts and items that make up the product. Before the robot grabs the parts and items, it is necessary to first determine the grabbing posture of the parts and items.

[0003] In the prior art, first, point cloud data of a single perspective is obtained; then, the point cloud data of a single perspective is input into the grasping detection model, and the grasping posture of each object is determined by the grasping detection model. Point cloud data refers to a set of discrete points on the surface of an object obtained by a three-dimensional scanning device (such as a lidar, a 3D scanner, a depth camera, etc.). Point cloud data is a direct representation of the surface of an object in three-dimensional space. Each point in the point cloud data contains three-dimensional coordinates (X, Y, Z), which can accurately reflect the geometric shape and spatial distribution of the object.

[0004] However, the input of the above grasping detection model is all single-view point cloud data. Due to the limitations of single-view point cloud data, it is inevitable that there will be problems such as object occlusion, object stacking, and incomplete display of edge objects. This may cause some smaller objects to be ignored due to occlusion when determining the grasping posture of the object. Therefore, the grasping posture of smaller objects may be missing in the final grasping scene, affecting the overall grasping accuracy. Summary of the invention

[0005] The present application provides a method, device and equipment for determining the grasping posture of an object by fusion of multi-view point clouds, so as to solve the problem in the prior art of poor overall grasping accuracy due to the lack of grasping postures for smaller objects.

[0006] In a first aspect, the present application provides a method for determining the position and posture of an object grasped by fusion of multi-view point clouds, comprising:

[0007] Acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects;

[0008] fusing the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and fusing features of the point cloud data of the multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects;

[0009] fusing point cloud features from the perspectives of the multiple objects through an instance segmentation network model, performing point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtaining instance point cloud data corresponding to each object;

[0010] The grasping posture of each object is determined according to the instance point cloud data corresponding to each object, wherein the grasping posture includes a grasping direction, a grasping position, a grasping angle and a grasping width.

[0011] In an optional embodiment, the point cloud data includes three-dimensional coordinates of multiple points in a camera coordinate system. Accordingly, the point cloud data of the multiple objects at different perspectives are fused to obtain fused point cloud data of the multiple objects, including: according to camera parameters and coordinate transformation matrices of different perspectives, the three-dimensional coordinates of each point in the point cloud data corresponding to each perspective in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system to obtain point cloud data of each perspective after coordinate conversion; the point cloud data of each perspective after coordinate conversion are fused, and the fused point cloud data of the multiple objects are obtained by downsampling through the dynamic voxel method.

[0012] In an optional embodiment, the point cloud data of the multiple objects at different perspectives are subjected to feature fusion through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects, including: extracting single-perspective high-dimensional features corresponding to the point cloud data of each perspective, and extracting perspective fused high-dimensional features corresponding to the fused point cloud data; and performing feature fusion on the single-perspective high-dimensional features corresponding to each of the multiple perspectives with the perspective fused high-dimensional features to obtain perspective fused point cloud features of the multiple objects.

[0013] In an optional embodiment, the extracting of single-perspective high-dimensional features corresponding to the point cloud data of each perspective, and the extracting of perspective fusion high-dimensional features corresponding to the fused point cloud data, include: using a lightweight feature extraction network to effectively extract high-dimensional features through a multi-scale grouping strategy to obtain single-perspective high-dimensional features corresponding to the point cloud data of each perspective; and using a deep feature extraction network to capture the global geometric structure in multi-perspective information to obtain fusion perspective high-dimensional features corresponding to the fused point cloud data.

[0014] In an optional embodiment, the single-view high-dimensional features corresponding to each of the multiple viewpoints are fused with the view fusion high-dimensional features to obtain the view fusion point cloud features of the multiple objects, including: point-by-point aligning the single-view point cloud features corresponding to each viewpoint with the view fusion high-dimensional features; propagating the single-view high-dimensional features corresponding to each viewpoint to the view fusion high-dimensional features through an attention feature propagation model to obtain the view fusion point cloud features of the multiple objects; wherein the attention feature propagation model is used to autonomously select specific features in each viewpoint based on the feature propagation mechanism, enhance the expression of the specific features, and ignore the overlapping parts in different viewpoints, so as to realize the feature fusion of the single-view high-dimensional features extracted from different viewpoints and the view fusion high-dimensional features.

[0015] In an optional embodiment, the instance segmentation network model is used to perform point cloud instance segmentation on the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects to obtain instance point cloud data corresponding to each object, including: determining the foreground point cloud data, background point cloud data and noise point cloud data in the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects; removing the background point cloud data and noise point cloud data in the fused point cloud data to obtain pure foreground point cloud data corresponding to the multiple objects; performing point cloud instance segmentation on the pure foreground point cloud corresponding to the multiple objects in combination with the position information of each object through the instance segmentation network model to obtain instance point cloud data corresponding to each object.

[0016] In an optional embodiment, the grasping posture of each object is determined according to the instance point cloud data corresponding to each object, including: for the instance point cloud data corresponding to each object, performing feature extraction on the instance point cloud data through a global feature extraction network in a grasping detection network model to obtain global high-dimensional features corresponding to the object; determining multiple candidate grasping points corresponding to the object and a grasping direction corresponding to each candidate grasping point according to the global high-dimensional features corresponding to the object through the grasping detection network model, determining the cylindrical grasping area corresponding to the candidate grasping point from the instance point cloud data according to the grasping direction corresponding to each candidate grasping point, determining the grasping position, grasping angle and grasping width corresponding to each cylindrical grasping area, and obtaining multiple candidate grasping postures corresponding to the object; scoring the multiple candidate grasping postures through a grasping quality evaluation network, and screening out the candidate grasping posture with the highest score to determine as the grasping posture of the object.

[0017] In a second aspect, the present application provides a device for determining object grasping posture by fusion of multi-view point clouds, comprising:

[0018] An acquisition module, used to acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects;

[0019] A feature fusion module is used to fuse the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and to perform feature fusion on the point cloud data of the multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects;

[0020] A point cloud instance segmentation module is used to fuse point cloud features according to the perspectives of the multiple objects through an instance segmentation network model, perform point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtain instance point cloud data corresponding to each object;

[0021] The determination module is used to determine the grasping posture of each object according to the instance point cloud data corresponding to each object, wherein the grasping posture includes grasping direction, grasping position, grasping angle and grasping width.

[0022] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0023] The memory stores computer-executable instructions;

[0024] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0026] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in any one of the first aspects when executed by a processor.

[0027] The present application provides a method, device and equipment for determining the grasping posture of an object by multi-view point cloud fusion, the method comprising: acquiring point cloud data of multiple objects at different viewpoints, the point cloud data comprising three-dimensional coordinates of multiple points, for reflecting the geometric shape and spatial distribution of the objects; fusing the point cloud data of multiple objects at different viewpoints to obtain fused point cloud data of multiple objects; and, performing feature fusion on the point cloud data of multiple objects at different viewpoints through a point cloud fusion network model to obtain the viewpoint fusion point cloud features of multiple objects; performing point cloud instance segmentation on the fused point cloud data of multiple objects according to the viewpoint fusion point cloud features of multiple objects through an instance segmentation network model to obtain instance point cloud data corresponding to each object; determining the grasping posture of each object according to the instance point cloud data corresponding to each object, wherein the grasping posture comprises grasping direction, grasping position, grasping angle and grasping width. In the embodiment of the present application, since point cloud data of multiple objects at different perspectives are obtained, perspective fused point cloud features are obtained by fusing the point cloud data at different perspectives, and point cloud instance segmentation is performed on the fused point cloud data of multiple objects to obtain instance point cloud data corresponding to each object; and the point cloud data of multiple objects at different perspectives can avoid problems such as object occlusion, object stacking, and incomplete display of edge objects, and thus can improve the accuracy of the determined grasping posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0029] Figure 1 An application scenario diagram of the method for determining the object grasping posture by fusion of multi-view point clouds provided in this application;

[0030] Figure 2 A flowchart of a method for determining the position and posture of an object grasped by fusion of multi-view point clouds provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of a process of multi-view point cloud feature extraction and point cloud instance segmentation provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a scene-level point cloud and an object-level point cloud provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of a process flow of a grabbing model provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of the structure of a device for determining the position and posture of an object grasped by fusion of multi-view point clouds provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0036] Figure 8 A block diagram of an electronic device shown in an embodiment of the present application.

[0037] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0038] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] With the development of robot technology, when a product is assembled by a robot, the robot can automatically grab the various parts and items that make up the product. Before the robot grabs the parts and items, it is necessary to first determine the grabbing posture of the parts and items.

[0041] In the prior art, first, point cloud data of a single perspective is obtained; then, the point cloud data of a single perspective is input into the grasping detection model, and the grasping posture of each object is determined by the grasping detection model. Point cloud data refers to a set of discrete points on the surface of an object obtained by a three-dimensional scanning device (such as a lidar, a 3D scanner, a depth camera, etc.). Point cloud data is a direct representation of the surface of an object in three-dimensional space. Each point in the point cloud data contains three-dimensional coordinates (X, Y, Z), which can accurately reflect the geometric shape and spatial distribution of the object.

[0042] However, the input of the above grasping detection model is all single-view point cloud data. Due to the limitations of single-view point cloud data, it is inevitable that there will be problems such as object occlusion, object stacking, and incomplete display of edge objects. This may cause some smaller objects to be ignored due to occlusion when determining the grasping posture of the object, resulting in low grasping quality or even no grasping of small objects. Therefore, the grasping posture of small objects may be missing in the final grasping scene, affecting the overall grasping accuracy.

[0043] In addition, due to the incomplete display of some objects, the object position information is insufficient, which leads to the fact that the features extracted during the grasping process are not enough to fully express the position relationship of the entire scene, affecting the success rate of the robot grasping. At the same time, since some objects are not visible, there may be dangerous phenomena such as collision between the robot's gripper and the object during the robot's grasping process.

[0044] In view of the technical problems in the above-mentioned prior art, the inventor proposes the following technical concept: the present invention proposes a segmentation and capture framework for multi-view point cloud fusion. Figure 1 As shown in the figure, the framework includes a multi-view point cloud fusion module, a point cloud instance segmentation module and a multi-scale grasping and detection module. Using the information of point clouds from multiple different viewpoints, the multi-view point cloud fusion module will use feature interpolation and feature propagation to fuse the features of multi-view point clouds to obtain a complete point cloud with comprehensive location information, solving the problem of incomplete location information under a single viewpoint. Using the point cloud with complete location information, the point cloud instance segmentation module is used to perform point cloud instance segmentation. First, the foreground points of the point cloud are distinguished from the background points and the noise points are filtered out to obtain a pure point cloud. Secondly, the foreground point cloud is further instance segmented to obtain the point cloud corresponding to each object, solving the problem that the background points and noise points cause the computational load and affect the success of grasping. At the same time, through instance segmentation, the scene-level point cloud can be converted to the object-level point cloud. Using the object-level point cloud converted from the scene-level point cloud for grasping and detection can solve the problem that some small objects with few points are ignored during grasping and detection, and can ensure that each object can be detected. At the same time, in order to further enrich the features extracted in the feature extraction stage, a multi-scale feature extraction module is designed for feature extraction and fusion to ensure that the point cloud feature information is rich during the segmentation and grasping detection process.

[0045] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0046] Figure 2A flowchart of a method for determining the position and posture of an object grasped by multi-view point cloud fusion provided in an embodiment of the present application. The method can be executed by a device for determining the position and posture of an object grasped by multi-view point cloud fusion, which can be a server or an electronic device. The following description takes an electronic device as an example. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware, such as Figure 2 As shown, the method comprises the following steps:

[0047] S201. Obtain point cloud data of multiple objects at different viewing angles, where the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects.

[0048] In an embodiment of the present application, it is first necessary to obtain a basic data set, which may be point cloud data of multiple objects at multiple different viewing angles. The basic data set may also be depth maps of multiple objects at multiple different viewing angles, and then the point cloud data is constructed from the depth maps. It should be noted that steps S201 to S204 determine the method of grasping the posture of each object in a scene of multiple objects. The grasping posture determination method is also suitable for determining the grasping posture of a single object in a scene of a single object.

[0049] The different viewing angles may be a plurality of preset viewing angles. The number of the plurality of preset viewing angles is not specifically limited in the present application and can be set and modified as required.

[0050] S202, fusing the point cloud data of multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and, fusing the point cloud data of multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects.

[0051] In the embodiment of the present application, the single-view point cloud in the basic data set can be used to construct multi-view fusion point cloud data. The point cloud data of each current view is a point cloud in the camera coordinate system. The coordinate transformation matrix between the camera coordinate system and the point clouds of different views can be used to obtain the coordinate transformation matrix from the camera coordinate system to the world coordinate system. That is, through the transformation of the camera coordinate system and the world coordinate system, the multi-view point clouds can be unified into the same coordinate system through coordinate transformation, thereby realizing the fusion of multi-view point clouds.

[0052] In some embodiments, the point cloud data includes three-dimensional coordinates of multiple points in a camera coordinate system. Accordingly, the point cloud data of multiple objects at different perspectives are fused to obtain fused point cloud data of the multiple objects, including: according to the camera parameters and the coordinate transformation matrix of different perspectives, the three-dimensional coordinates of each point in the point cloud data corresponding to each perspective in the camera coordinate system are converted to three-dimensional coordinates in the world coordinate system to obtain the point cloud data of each perspective after the coordinate conversion; the point cloud data of each perspective after the coordinate conversion are fused, and the fused point cloud data of the multiple objects are obtained by downsampling through the dynamic voxel method.

[0053] The coordinate transformation formula of the point cloud transformation is as follows. First, the point clouds of each view are unified from the camera view to the camera 0 view, and then unified from the camera 0 view to the world coordinate system, which can be expressed by the following formula:

[0054]

[0055] in, Represents the point cloud coordinates of each viewpoint, Indicates the coordinate transformation of the point cloud coordinates of each view to the camera 0 view. Represents the coordinate transformation from the camera 0 perspective to the world coordinate system.

[0056] In the disclosed embodiment, the number of points in the point cloud data after multi-perspective fusion will double. Therefore, after the point cloud fusion, the fused point cloud is further downsampled using the dynamic voxel method to obtain fused point cloud data with an appropriate amount of data. Each single-perspective point cloud corresponds to a segmentation and capture label, so in the process of multi-perspective point cloud fusion, the label data is corresponded to the points in the point cloud through dimensional expansion. When the point cloud is fused from multiple perspectives, the segmentation and capture labels are also fused synchronously, and there will be no confusion in their respective labels.

[0057] In the embodiments of the present application, in order to solve the problems of occlusion, stacking and incomplete parts encountered when grasping objects under single-view conditions, a multi-view fusion data set is constructed to realize multi-view point cloud information fusion. This method uses multi-view fusion data to extract local features of a single view and global features of a fused view from point cloud data. In combination with the extracted multi-scale features, multi-view point cloud feature fusion and point cloud feature propagation methods are used to fuse multi-view point cloud data and multi-view point cloud features to complete the single-view point cloud information and obtain complete point cloud feature information. This part of the content can be divided into two parts: point cloud feature extraction and point cloud feature fusion.

[0058] Accordingly, the point cloud data of multiple objects at different perspectives are subjected to feature fusion through the point cloud fusion network model to obtain the perspective fusion point cloud features of multiple objects, including the following steps (1) to (2):

[0059] (1) Extracting the single-view high-dimensional features corresponding to the point cloud data of each view, and extracting the view fusion high-dimensional features corresponding to the fused point cloud data.

[0060] In this step, based on the uniqueness of point cloud features, differentiated strategies are proposed for feature extraction of single-view point clouds and multi-view fusion point clouds. For single-view point clouds, we focus on capturing local geometric details through lightweight feature extraction networks to cope with the challenges of occlusion and local noise. For feature extraction of multi-view fusion point clouds, we turn to more complex deep feature extraction networks to fully capture the global geometric structure and semantic features in multi-view information and enhance the understanding of the overall shape and position of objects.

[0061] Optionally, single-view high-dimensional features corresponding to the point cloud data of each view are extracted, and perspective fusion high-dimensional features corresponding to the fused point cloud data are extracted, including: using a lightweight feature extraction network to effectively extract high-dimensional features through a multi-scale grouping strategy to obtain single-view high-dimensional features corresponding to the point cloud data of each view; and using a deep feature extraction network to capture the global geometric structure in multi-view information to obtain fused perspective high-dimensional features corresponding to the fused point cloud data.

[0062] For example, Figure 3 As shown, the lightweight feature extraction network can be a PointNet++ network model or a PointCNN network model. The lightweight network structure model can effectively extract local features through a multi-scale grouping strategy, thereby providing robustness to noise and occlusion.

[0063] For example, Figure 3 As shown in the figure, the deep feature extraction network can be a multi-scale Point Transformer (point cloud feature fusion model) or PV-RCNN (Point-Voxel Feature Set Abstraction for 3D Object Detection‌, a point voxel feature set abstract network model for three-dimensional target detection), among which the deep feature extraction network can better handle the complexity and richness of multi-view data.

[0064] (2) The single-view high-dimensional features corresponding to multiple viewpoints are fused with the view fusion high-dimensional features to obtain the view fusion point cloud features of multiple objects.

[0065] In this step, after completing the single-view point cloud feature extraction and fused point cloud feature extraction, the key to this module is to use the point cloud feature propagation mechanism to achieve the fusion of multi-view point cloud features. Accordingly, the single-view high-dimensional features corresponding to each of the multiple viewpoints are fused with the view fusion high-dimensional features to obtain the view fusion point cloud features of multiple objects, including: aligning the single-view point cloud features corresponding to each viewpoint with the view fusion high-dimensional features point by point. Through the attention feature propagation model, the single-view high-dimensional features corresponding to each viewpoint are propagated to the view fusion high-dimensional features to obtain the view fusion point cloud features of multiple objects; wherein, the attention feature propagation model is used to autonomously select specific features in each viewpoint based on the feature propagation mechanism, enhance the expression of specific features, and ignore the overlapping parts in different viewpoints, so as to achieve the feature fusion of the single-view high-dimensional features extracted from different viewpoints and the view fusion high-dimensional features.

[0066] It should be noted that to achieve the fusion of multi-view point cloud features, the point cloud features must be aligned point by point. If the point cloud cannot be aligned point by point during the point cloud feature fusion process, the fused point cloud features will be in a chaotic state, making the subsequent processing process impossible. Among them, point-by-point feature alignment can be achieved through methods such as ICP (iterative closest point) algorithm, point cloud coordinate position transformation, and feature matching-based alignment methods.

[0067] Optionally, the feature propagation mechanism in the PointNet++ network model can propagate the features of one point cloud to another point cloud and fuse the features of the two point clouds to achieve point-by-point feature alignment of point clouds and feature propagation of point clouds.

[0068] It should be noted that the number of single-view high-dimensional features corresponding to multiple viewpoints is multiple, and the weight of each single-view point cloud feature can be configured after the model is trained. Figure 3 As shown, the number of single-view high-dimensional features corresponding to multiple viewpoints is 3, and the weight of each single-view point cloud feature can be 0.5, 0.2, and 0.3.

[0069] Understandably, Figure 3 The number of features in is only an illustrative description of the feature extraction process and the feature fusion process. In actual application scenarios, the number of features in the feature extraction process and the number of features in the feature fusion process should be consistent.

[0070] S203, using an instance segmentation network model to fuse point cloud features from the perspectives of multiple objects, performing point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtaining instance point cloud data corresponding to each object.

[0071] In the embodiment of the present application, after successfully achieving the fusion and alignment of multi-view point cloud features, the fused feature segmentation stage uses the fused features to perform accurate instance segmentation so that different objects can be accurately identified and distinguished in the grasping scene. In order to ensure the accuracy of segmentation, the segmentation network based on the fused features is combined with the point cloud neighborhood features and point cloud positions to control the consistency of point cloud segmentation. At the same time, combined with classification, boundary and instance center loss, and enhanced instance segmentation effects, it is ensured that instances of different objects can be accurately segmented to assist the subsequent grasping process.

[0072] Optionally, point cloud instance segmentation is performed on the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects through an instance segmentation network model to obtain instance point cloud data corresponding to each object, including: determining foreground point cloud data, background point cloud data and noise point cloud data in the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects; removing the background point cloud data and noise point cloud data in the fused point cloud data to obtain pure foreground point cloud data corresponding to the multiple objects; performing point cloud instance segmentation on the pure foreground point cloud corresponding to the multiple objects in combination with the position information of each object through the instance segmentation network model to obtain instance point cloud data corresponding to each object.

[0073] In an embodiment of the present application, first, using the complete point cloud feature information, instance segmentation is performed by distinguishing the point cloud background and foreground through point cloud segmentation, thereby reducing the interference of background noise on subsequent grasping tasks. In this way, a pure foreground point cloud with complete perspective information and not interfered by background noise is obtained. Then, the pure point cloud can continue to be segmented according to the object instance to obtain the instance point cloud corresponding to each object, and the instance point cloud corresponding to each object can be fused to obtain a pure foreground point cloud. The fusion of pure instance point clouds and further segmentation of foreground point clouds realize arbitrary transformation of scene-level point clouds and object-level point clouds. Among them, using object-level point clouds for grasping detection can make up for the problem that small-volume objects are easily ignored when performing grasping detection on scene-level point clouds. Exemplarily, scene-level point clouds and object-level point clouds are as follows: Figure 4 shown.

[0074] S204. Determine a grasping posture of each object according to the instance point cloud data corresponding to each object, wherein the grasping posture includes a grasping direction, a grasping position, a grasping angle, and a grasping width.

[0075] In the process of grasping and detection, the surface features of some objects are complex and the volume of some objects is small. Even if the complete spatial position information of parts is obtained through multi-view fusion, the feature extraction of only a single scale will affect the extraction of local detail features of parts, and then affect the detection of grasping posture. In order to enhance the extraction of local detail features, a multi-scale feature extraction method is necessary. Considering that the point cloud data input to the grasping network is pure multi-view point cloud data, the cylinder grouping method is used in the grasping prediction stage to realize the extraction of multi-scale features. Based on this, a multi-scale cylinder grouping, cylinder feature extraction and feature fusion module is constructed. This module has multiple cylinder groups of different sizes. These groups will extract grasping features at different scales, and then use the attention mechanism to make the grasping network pay more attention to the grasping information of key positions. Finally, the multi-scale information is organically combined by designing a feature fusion module to obtain comprehensive grasping feature information.

[0076] In some embodiments, in order to further improve the extraction of small object features and enhance the grasping accuracy, a feature extraction method of multi-sampling quantity fusion / multi-radius fusion is designed in the global feature extraction stage, using multi-level sampling quantity point clouds and multiple sampling ball radii to enrich the collected global features, and all collected features are fused to ensure that the collected point cloud features are more comprehensive.

[0077] Accordingly, the grasping posture of each object is determined according to the instance point cloud data corresponding to each object, including:

[0078] (1) For each instance point cloud data corresponding to an object, the global feature extraction network in the grasping and detection network model is used to extract features from the instance point cloud data to obtain the global high-dimensional features corresponding to the object.

[0079] In some embodiments, during the grasping posture detection process, feature extraction is performed on the fused point cloud based on a multi-scale Point Transformer global feature extraction network to generate multiple candidate grasping points, and the three-dimensional position of each grasping point and the rotation angle of the grasper are generated through a 6D grasping posture prediction module.

[0080] (2) The grasping detection network model is used to determine multiple candidate grasping points corresponding to the object and the grasping direction corresponding to each candidate grasping point based on the global high-dimensional features of the object. According to the grasping direction corresponding to each candidate grasping point, the cylindrical grasping area corresponding to the candidate grasping point is determined from the instance point cloud data. The grasping position, grasping angle and grasping width corresponding to each cylindrical grasping area are determined to obtain multiple candidate grasping postures corresponding to the object.

[0081] In some embodiments, Figure 5As shown in the figure, in the feature extraction stage, Point Transformer (point cloud feature fusion model) can be used to extract multi-scale global features of fused point clouds, using multi-level local geometric information for extraction while maintaining global information to ensure that the overall information is not lost. Figure 5 After the global features in the point cloud are obtained, Approach Net (policy network model) predicts the approach vector of the robot grasping through the input global features, that is, predicts from which direction the robot approaches the object. Using the approach vector predicted from Approach Net, the system groups the point cloud into cylindrical regions. Each group corresponds to a candidate grasping area, which represents part of the object surface that the gripper can cover. The grouped point cloud is used to further analyze the detailed information of the grasping and ensure that the grasping area is stable enough and will not be disturbed. Cylindrical grouping ensures that the gripper can find a suitable grasping position on complex geometric shapes.

[0082] (3) The grasping quality evaluation network is used to score multiple candidate grasping postures, and the candidate grasping posture with the highest score is selected as the grasping posture of the object.

[0083] In some embodiments, Figure 5 As shown in the figure, after multi-scale feature extraction and multi-scale feature fusion are performed using the cylinder grouping model, Operation Net is used to predict grasping posture parameters such as grasping position, grasping width, grasping angle, etc. Among them, the cylinder grouping model can be Group Attention.

[0084] Among them, grasping posture evaluation is a key step in the robot grasping task, which determines whether the grasping action can be successful. After obtaining the grasping direction, grasping position, grasping width and grasping angle predicted by the grasping network, the grasping quality evaluation network is used to score and screen the candidate grasping postures to select the optimal grasping posture.

[0085] The present application provides a method for determining the grasping posture of an object by multi-view point cloud fusion, the method comprising: acquiring point cloud data of multiple objects at different viewpoints, the point cloud data comprising three-dimensional coordinates of multiple points, for reflecting the geometric shape and spatial distribution of the objects; fusing the point cloud data of multiple objects at different viewpoints to obtain fused point cloud data of multiple objects; and, performing feature fusion on the point cloud data of multiple objects at different viewpoints through a point cloud fusion network model to obtain viewpoint fused point cloud features of multiple objects; performing point cloud instance segmentation on the fused point cloud data of multiple objects according to the viewpoint fusion point cloud features of multiple objects through an instance segmentation network model to obtain instance point cloud data corresponding to each object; determining the grasping posture of each object according to the instance point cloud data corresponding to each object, wherein the grasping posture comprises a grasping direction, a grasping position, a grasping angle and a grasping width. In the embodiment of the present application, since point cloud data of multiple objects at different perspectives are obtained, perspective fused point cloud features are obtained by fusing the point cloud data at different perspectives, and point cloud instance segmentation is performed on the fused point cloud data of multiple objects to obtain instance point cloud data corresponding to each object; and the point cloud data of multiple objects at different perspectives can avoid problems such as object occlusion, object stacking, and incomplete display of edge objects, and thus can improve the accuracy of the determined grasping posture.

[0086] It should be noted that in the embodiments of the present application, (1) the problem of occlusion, stacking and incomplete display of edge objects under a single perspective is solved by fusion of multi-view point cloud data, and the object position information is supplemented. (2) The object points of the point cloud are extracted by point cloud instance segmentation to filter out background points and noise points, which solves the problem that background points and noise points exist in the grasping scene, introducing unnecessary calculations and affecting the grasping detection efficiency. (3) Point cloud instance segmentation can obtain pure instance point clouds, thereby realizing the conversion between scene-level grasping and object-level grasping through the point clouds of each instance. Through object-level grasping, small objects are focused on to avoid small objects being ignored. (4) The multi-scale grasping detection module can use different radii to perform multi-scale extraction and fusion of global features and grasped local features, enrich the grasping information extracted by the network, and ensure that the grasping information is rich and complete.

[0087] In the multi-view point cloud fusion segmentation and grasping framework of the embodiment of the present application, segmentation loss and grasping loss are key components used to improve segmentation accuracy and grasping effect. The following is a detailed design and function description of various loss functions. Among them, various loss functions include basic loss function, segmentation loss function and grasping loss function. Basic loss functions include: cross entropy loss function, smooth L1 loss function and mean square error loss function.

[0088] (I) Introduction to basic loss function:

[0089] 1. Cross entropy loss (CE Loss), which is used to measure the difference between the predicted probability distribution and the true label, can be expressed by the following formula:

[0090]

[0091] Among them, y ic is the true label, is the predicted probability, N is the number of points in the point cloud, and C is the number of object categories.

[0092] 2. Smooth L1 loss, suitable for regression tasks, is robust to outliers and can be expressed by the following formula:

[0093]

[0094] Huber Loss is a generalized form of smooth L1 loss:

[0095]

[0096] Among them, x n and n Represent the predicted coordinates and the actual coordinates respectively.

[0097] 3. Mean square error loss:

[0098]

[0099] Among them, x n and n Represent the predicted coordinates and the actual coordinates respectively.

[0100] (II) Introduction to segmentation loss function:

[0101] The segmentation loss function is used to measure the difference between the predicted segmentation result and the true label. The commonly used loss function is the cross entropy loss. In order to improve the stability and accuracy of the segmented area, after introducing the cross entropy loss on the segmentation label, the center offset loss and spatial consistency loss are also introduced. These three loss functions are combined to improve the accuracy of segmentation.

[0102] Optionally, spatial consistency loss is used to ensure that adjacent points have similar labels. This loss function is implemented by calculating the probability difference between each point and its neighboring points, which can be expressed by the following formula:

[0103]

[0104] Among them, N represents the total number of points in the point cloud, K represents the neighboring points considered when calculating the loss, is the predicted probability of the i-th point, is the predicted probability of the jth neighboring point of the ith point.

[0105] Segmentation label loss:

[0106] Center shift loss:

[0107] The final segmentation loss can be expressed as:

[0108] in, is the balance coefficient, which is used to balance the importance between different losses.

[0109] (III) Introduction to crawling loss function:

[0110] The grasping loss function is used to measure the difference between the predicted grasping posture and the actual grasping posture. The points obtained after instance segmentation can be used as grasping points. First, the grasping direction loss is calculated for each grasping point:

[0111]

[0112] Among them, V ij represents the jth predicted grasping direction vector of the i-th graspable point. ij : The jth predicted confidence score of the i-th graspable point. Regression loss (L reg ): The mean square error loss function is used to calculate the loss between the predicted grasping direction vector and the true grasping direction vector, but only the prediction results within 5 degrees of the true grasping direction vector are considered.

[0113] The crawling parameter loss can be expressed as:

[0114]

[0115]

[0116]

[0117] Among them, L angle represents the grasping rotation angle loss, L width Grasp width loss, L score Represents the grasp confidence loss.

[0118] Full crawl loss:

[0119]

[0120] Combining segmentation loss and grasping loss, a complete loss function is designed:

[0121]

[0122] in, is the balance coefficient, which is used to balance the importance between different losses.

[0123] In the disclosed embodiment, by optimizing the loss function, the multi-view point cloud fusion segmentation and grasping framework can be effectively trained to improve the accuracy of segmentation and grasping.

[0124] Figure 6 This is a schematic diagram of the structure of a device for determining the object grasping posture by fusion of multi-view point clouds provided in an embodiment of the present application. Figure 6 The device for determining the object grasping posture by multi-view point cloud fusion includes: an acquisition module 601, a feature fusion module 602, a point cloud instance segmentation module 603 and a determination module 604;

[0125] The acquisition module 601 is used to acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects;

[0126] The feature fusion module 602 is used to fuse the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and to perform feature fusion on the point cloud data of the multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects;

[0127] The point cloud instance segmentation module 603 is used to perform point cloud instance segmentation on the fused point cloud data of the multiple objects by fusing point cloud features according to the perspectives of the multiple objects through an instance segmentation network model, so as to obtain instance point cloud data corresponding to each object;

[0128] The determination module 604 is used to determine the grasping posture of each object according to the instance point cloud data corresponding to each object, wherein the grasping posture includes grasping direction, grasping position, grasping angle and grasping width.

[0129] In an optional embodiment, the point cloud data includes three-dimensional coordinates of multiple points in a camera coordinate system. Accordingly, the feature fusion module 602 fuses the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects, including: according to camera parameters and coordinate transformation matrices of different perspectives, converting the three-dimensional coordinates of each point in the point cloud data corresponding to each perspective in the camera coordinate system into three-dimensional coordinates in the world coordinate system to obtain point cloud data of each perspective after coordinate conversion; fusing the point cloud data of each perspective after coordinate conversion, and obtaining the fused point cloud data of the multiple objects by downsampling through the dynamic voxel method.

[0130] In an optional embodiment, the feature fusion module 602 performs feature fusion on the point cloud data of the multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects, including: extracting single-perspective high-dimensional features corresponding to the point cloud data of each perspective, and extracting perspective fused high-dimensional features corresponding to the fused point cloud data; and performing feature fusion on the single-perspective high-dimensional features corresponding to each of the multiple perspectives with the perspective fused high-dimensional features to obtain perspective fused point cloud features of the multiple objects.

[0131] In an optional embodiment, the feature fusion module 602 extracts single-view high-dimensional features corresponding to the point cloud data of each view, and extracts perspective fusion high-dimensional features corresponding to the fused point cloud data, including: using a lightweight feature extraction network to effectively extract high-dimensional features through a multi-scale grouping strategy to obtain single-view high-dimensional features corresponding to the point cloud data of each view; and using a deep feature extraction network to capture the global geometric structure in multi-view information to obtain fused perspective high-dimensional features corresponding to the fused point cloud data.

[0132] In an optional embodiment, the feature fusion module 602 performs feature fusion on the single-view high-dimensional features corresponding to each of the multiple viewpoints with the view-fusion high-dimensional features to obtain the view-fusion point cloud features of the multiple objects, specifically including: point-by-point aligning the single-view point cloud features corresponding to each viewpoint with the view-fusion high-dimensional features; propagating the single-view high-dimensional features corresponding to each viewpoint to the view-fusion high-dimensional features through an attention feature propagation model to obtain the view-fusion point cloud features of the multiple objects; wherein the attention feature propagation model is used to autonomously select specific features in each viewpoint based on the feature propagation mechanism, enhance the expression of the specific features, and ignore the overlapping parts in different viewpoints, so as to realize the feature fusion of the single-view high-dimensional features extracted from different viewpoints and the view-fusion high-dimensional features.

[0133] In an optional embodiment, the point cloud instance segmentation module 603 performs point cloud instance segmentation on the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects through an instance segmentation network model to obtain instance point cloud data corresponding to each object, including: determining the foreground point cloud data, background point cloud data and noise point cloud data in the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects; removing the background point cloud data and noise point cloud data in the fused point cloud data to obtain pure foreground point cloud data corresponding to the multiple objects; performing point cloud instance segmentation on the pure foreground point cloud corresponding to the multiple objects in combination with the position information of each object through an instance segmentation network model to obtain instance point cloud data corresponding to each object.

[0134] In an optional embodiment, the determination module 604 determines the grasping posture of each object according to the instance point cloud data corresponding to each object, including: for the instance point cloud data corresponding to each object, performing feature extraction on the instance point cloud data through a global feature extraction network in a grasping detection network model to obtain global high-dimensional features corresponding to the object; determining multiple candidate grasping points corresponding to the object and a grasping direction corresponding to each candidate grasping point according to the global high-dimensional features corresponding to the object through the grasping detection network model, determining the cylindrical grasping area corresponding to the candidate grasping point from the instance point cloud data according to the grasping direction corresponding to each candidate grasping point, determining the grasping position, grasping angle and grasping width corresponding to each cylindrical grasping area, and obtaining multiple candidate grasping postures corresponding to the object; scoring the multiple candidate grasping postures through a grasping quality evaluation network, and screening out the candidate grasping posture with the highest score to determine as the grasping posture of the object.

[0135] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device of this embodiment may include:

[0136] at least one processor 710; and

[0137] A memory 720 communicatively coupled to at least one processor;

[0138] The memory 720 stores instructions that can be executed by at least one processor 710, and the instructions are executed by at least one processor 710 to enable the server to execute a method as described in any of the above embodiments.

[0139] Optionally, the memory 720 may be independent or integrated with the processor 710 .

[0140] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0141] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0142] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0144] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0145] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The memory may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0146] The above storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium may be any available medium that can be accessed by a general or special purpose computer.

[0147] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in a server or a main control device.

[0148] Figure 8 This is a block diagram of an electronic device shown in an embodiment of the present application. The device may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0149] See also Figure 8 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0150] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0151] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0152] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0153] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0154] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0155] The input / output interface 812 provides an interface between the processing component 802 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0156] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge-coupled device (CCD), for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0157] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0158] In an exemplary embodiment, the device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0159] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a read-only memory (ROM), a high-speed random access memory (RAM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0160] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned electronic device method.

[0161] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0162] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for determining the object grasping posture by multi-view point cloud fusion, characterized in that: The method comprises: Acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects; fusing the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and fusing features of the point cloud data of the multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects; fusing point cloud features from the perspectives of the multiple objects through an instance segmentation network model, performing point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtaining instance point cloud data corresponding to each object; The grasping posture of each object is determined according to the instance point cloud data corresponding to each object, wherein the grasping posture includes a grasping direction, a grasping position, a grasping angle and a grasping width.

2. The method for determining the object grasping posture according to claim 1, characterized in that: The point cloud data includes three-dimensional coordinates of multiple points in a camera coordinate system. Accordingly, the point cloud data of the multiple objects at different viewing angles are fused to obtain fused point cloud data of the multiple objects, including: According to the camera parameters and the coordinate transformation matrix of different viewing angles, the three-dimensional coordinates of each point in the point cloud data corresponding to each viewing angle in the camera coordinate system are converted to the three-dimensional coordinates in the world coordinate system, and the point cloud data of each viewing angle after the coordinate conversion is obtained; The point cloud data of each perspective after coordinate transformation are fused, and the fused point cloud data of the multiple objects are obtained by downsampling through the dynamic voxel method.

3. The method for determining the object grasping posture according to claim 1, characterized in that: The step of fusing the point cloud data of the multiple objects at different perspectives using a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects includes: Extracting single-view high-dimensional features corresponding to the point cloud data of each view, and extracting view fusion high-dimensional features corresponding to the fused point cloud data; The single-view high-dimensional features corresponding to each of the multiple viewpoints are fused with the view fusion high-dimensional features to obtain view fusion point cloud features of the multiple objects.

4. The method for determining the object grasping posture according to claim 3, characterized in that: The extracting of single-viewpoint high-dimensional features corresponding to the point cloud data of each viewpoint, and the extracting of viewpoint fusion high-dimensional features corresponding to the fused point cloud data, include: Through a lightweight feature extraction network, high-dimensional features are effectively extracted through a multi-scale grouping strategy to obtain single-view high-dimensional features corresponding to the point cloud data of each view; and through a deep feature extraction network, the global geometric structure in multi-view information is captured to obtain the fused view high-dimensional features corresponding to the fused point cloud data.

5. The method for determining the object grasping posture according to claim 3, characterized in that: The step of fusing the single-view high-dimensional features corresponding to each of the multiple viewpoints with the view-fusion high-dimensional features to obtain the view-fusion point cloud features of the multiple objects includes: Aligning the single-view point cloud features corresponding to each view with the view fusion high-dimensional features point by point; The single-view high-dimensional features corresponding to each view are propagated to the view fusion high-dimensional features through the attention feature propagation model to obtain the view fusion point cloud features of the multiple objects; wherein the attention feature propagation model is used to autonomously select the specific features in each view on the basis of the feature propagation mechanism, enhance the expression of the specific features, and ignore the overlapping parts in different view angles, so as to realize the feature fusion of the single-view high-dimensional features extracted from different view angles and the view fusion high-dimensional features.

6. The method for determining the object grasping posture according to claim 1, characterized in that: The example segmentation network model is used to fuse point cloud features from the perspectives of the multiple objects, and performs point cloud instance segmentation on the fused point cloud data of the multiple objects to obtain example point cloud data corresponding to each object, including: Determine, according to the perspective fusion point cloud features of the multiple objects, foreground point cloud data, background point cloud data, and noise point cloud data in the fusion point cloud data of the multiple objects; Removing background point cloud data and noise point cloud data from the fused point cloud data to obtain pure foreground point cloud data corresponding to the multiple objects; The instance segmentation network model is combined with the location information of each object to perform point cloud instance segmentation on the pure foreground point clouds corresponding to the multiple objects to obtain instance point cloud data corresponding to each object.

7. The method for determining the object grasping posture according to claim 1, characterized in that: Determining the grasping posture of each object according to the instance point cloud data corresponding to each object includes: For instance point cloud data corresponding to each object, feature extraction is performed on the instance point cloud data through a global feature extraction network in a crawling detection network model to obtain global high-dimensional features corresponding to the object; Determine multiple candidate grasping points corresponding to the object and the grasping direction corresponding to each candidate grasping point through the grasping detection network model according to the global high-dimensional features corresponding to the object, determine the cylindrical grasping area corresponding to the candidate grasping point from the instance point cloud data according to the grasping direction corresponding to each candidate grasping point, determine the grasping position, grasping angle and grasping width corresponding to each cylindrical grasping area, and obtain multiple candidate grasping postures corresponding to the object; The multiple candidate grasping postures are scored through a grasping quality evaluation network, and the candidate grasping posture with the highest score is screened out to be determined as the grasping posture of the object.

8. A device for determining object grasping posture by multi-view point cloud fusion, characterized in that: The device comprises: An acquisition module, used to acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects; A feature fusion module is used to fuse the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and to perform feature fusion on the point cloud data of the multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects; A point cloud instance segmentation module is used to fuse point cloud features according to the perspectives of the multiple objects through an instance segmentation network model, perform point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtain instance point cloud data corresponding to each object; The determination module is used to determine the grasping posture of each object according to the instance point cloud data corresponding to each object, wherein the grasping posture includes grasping direction, grasping position, grasping angle and grasping width.

9. An electronic device, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; When executing the computer-executable instructions, the processor is used to implement the object grasping posture determination method of multi-view point cloud fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for determining the object grasping posture by multi-view point cloud fusion as described in any one of claims 1 to 7.

Images