Method, device and equipment for determining object grasping posture based on multi-view point cloud fusion

Abstract

The present application provides a method, device, and apparatus for determining the grasping posture of an object using multi-viewpoint point cloud fusion, relating to the field of robotics. The method comprises: acquiring point cloud data of multiple objects at different viewpoints, the point cloud data comprising the three-dimensional coordinates of multiple points used to reflect the geometric shape and spatial distribution of the objects; fusing the point cloud data of the multiple objects at different viewpoints to obtain fused point cloud data of the multiple objects; and, using a point cloud fusion network model, performing feature fusion on the point cloud data of the multiple objects at different viewpoints to obtain perspective-fused point cloud features of the multiple objects; using an instance segmentation network model to perform point cloud instance segmentation on the fused point cloud data of the multiple objects based on the perspective-fused point cloud features of the multiple objects to obtain instance point cloud data corresponding to each object; and determining the grasping posture of each object based on the instance point cloud data corresponding to each object. This method can improve the accuracy of the determined grasping posture.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method, device, and equipment for determining the position and posture of an object grasped by fusion of multi-view point clouds. Background Art

[0002] With the development of robotics, when assembling a product through a robot, the robot can automatically grasp the various parts and objects that make up the product. Before the robot grasps the parts and objects, it needs to first determine the grasping posture of the parts and objects.

[0003] In existing technologies, point cloud data from a single perspective is first acquired. This point cloud data is then fed into a grasping detection model, which then determines the grasping pose of each object. Point cloud data refers to a collection of discrete points on an object's surface acquired using 3D scanning equipment (such as LiDAR, 3D scanners, and depth cameras). Point cloud data is a direct representation of the object's surface in three-dimensional space. Each point in the point cloud data contains three-dimensional coordinates (X, Y, Z), accurately reflecting the object's geometric shape and spatial distribution.

[0004] However, the input to these grasp detection models is single-view point cloud data. Due to the limitations of single-view point cloud data, issues such as object occlusion, object stacking, and incomplete representation of edge objects are inevitable. This can cause some smaller objects to be overlooked when determining the grasp pose. Consequently, the grasp poses of smaller objects may be missing from the final grasping scene, affecting overall grasping accuracy. Summary of the Invention

[0005] The present application provides a method, device and equipment for determining the grasping posture of an object by fusion of multi-view point clouds, which is used to solve the problem in the prior art of poor overall grasping accuracy due to the lack of grasping postures for smaller objects.

[0006] In a first aspect, the present application provides a method for determining the object grasping posture by fusion of multi-view point clouds, comprising:

[0007] Acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects;

[0008] fusing the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and performing feature fusion on the point cloud data of the multiple objects at different perspectives using a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects;

[0009] fusing point cloud features of the multiple objects according to their perspectives using an instance segmentation network model, performing point cloud instance segmentation on the fused point cloud data of the multiple objects to obtain instance point cloud data corresponding to each object;

[0010] The grasping posture of each object is determined according to the instance point cloud data corresponding to each object, wherein the grasping posture includes a grasping direction, a grasping position, a grasping angle, and a grasping width.

[0011] In an optional embodiment, the point cloud data includes three-dimensional coordinates of multiple points in a camera coordinate system. Accordingly, the point cloud data of the multiple objects at different perspectives are fused to obtain fused point cloud data of the multiple objects, including: according to camera parameters and coordinate transformation matrices of different perspectives, the three-dimensional coordinates of each point in the point cloud data corresponding to each perspective in the camera coordinate system are converted into three-dimensional coordinates in the world coordinate system to obtain point cloud data of each perspective after coordinate conversion; the point cloud data of each perspective after coordinate conversion are fused, and the fused point cloud data of the multiple objects are obtained by downsampling through the dynamic voxel method.

[0012] In an optional embodiment, the point cloud data of the multiple objects at different perspectives are subjected to feature fusion through a point cloud fusion network model to obtain perspective fusion point cloud features of the multiple objects, including: extracting single-perspective high-dimensional features corresponding to the point cloud data of each perspective, and extracting perspective fusion high-dimensional features corresponding to the fused point cloud data; and fusing the single-perspective high-dimensional features corresponding to each of the multiple perspectives with the perspective fusion high-dimensional features to obtain perspective fusion point cloud features of the multiple objects.

[0013] In an optional embodiment, the extracting of single-perspective high-dimensional features corresponding to the point cloud data of each perspective, and the extracting of perspective fusion high-dimensional features corresponding to the fused point cloud data, include: using a lightweight feature extraction network to effectively extract high-dimensional features through a multi-scale grouping strategy to obtain single-perspective high-dimensional features corresponding to the point cloud data of each perspective; and using a deep feature extraction network to capture the global geometric structure in multi-perspective information to obtain fusion perspective high-dimensional features corresponding to the fused point cloud data.

[0014] In an optional embodiment, the single-view high-dimensional features corresponding to each of the multiple viewpoints are fused with the view-fusion high-dimensional features to obtain the view-fusion point cloud features of the multiple objects, including: point-by-point aligning the single-view point cloud features corresponding to each viewpoint with the view-fusion high-dimensional features; propagating the single-view high-dimensional features corresponding to each viewpoint into the view-fusion high-dimensional features through an attention feature propagation model to obtain the view-fusion point cloud features of the multiple objects; wherein the attention feature propagation model is used to autonomously select specific features in each viewpoint based on the feature propagation mechanism, enhance the expression of the specific features, and ignore overlapping parts in different viewpoints, so as to achieve feature fusion of single-view high-dimensional features extracted from different viewpoints and view-fusion high-dimensional features.

[0015] In an optional embodiment, the instance segmentation network model is used to perform point cloud instance segmentation on the fused point cloud data of the multiple items according to the perspective fusion point cloud features of the multiple items to obtain instance point cloud data corresponding to each item, including: determining the foreground point cloud data, background point cloud data and noise point cloud data in the fused point cloud data of the multiple items according to the perspective fusion point cloud features of the multiple items; removing the background point cloud data and noise point cloud data in the fused point cloud data to obtain pure foreground point cloud data corresponding to the multiple items; and performing point cloud instance segmentation on the pure foreground point cloud corresponding to the multiple items in combination with the position information of each item through the instance segmentation network model to obtain instance point cloud data corresponding to each item.

[0016] In an optional embodiment, the grasping posture of each item is determined based on the instance point cloud data corresponding to each item, including: for the instance point cloud data corresponding to each item, performing feature extraction on the instance point cloud data through a global feature extraction network in a grasping detection network model to obtain global high-dimensional features corresponding to the item; determining multiple candidate grasping points corresponding to the item and a grasping direction corresponding to each candidate grasping point through the grasping detection network model based on the global high-dimensional features corresponding to the item, determining the cylindrical grasping area corresponding to the candidate grasping point from the instance point cloud data based on the grasping direction corresponding to each candidate grasping point, determining the grasping position, grasping angle and grasping width corresponding to each cylindrical grasping area, and obtaining multiple candidate grasping postures corresponding to the item; scoring the multiple candidate grasping postures through a grasping quality evaluation network, screening out the candidate grasping posture with the highest score and determining it as the grasping posture of the item.

[0017] In a second aspect, the present application provides a device for determining object grasping posture by fusion of multi-view point clouds, comprising:

[0018] An acquisition module is used to acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects;

[0019] a feature fusion module, configured to fuse the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and perform feature fusion on the point cloud data of the multiple objects at different perspectives using a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects;

[0020] a point cloud instance segmentation module, configured to fuse point cloud features of the multiple objects according to the perspectives of the multiple objects using an instance segmentation network model, perform point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtain instance point cloud data corresponding to each object;

[0021] The determination module is used to determine the grasping posture of each object based on the instance point cloud data corresponding to each object, wherein the grasping posture includes grasping direction, grasping position, grasping angle and grasping width.

[0022] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0023] The memory stores computer-executable instructions;

[0024] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0026] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the method described in any one of the first aspects when executed by a processor.

[0027] The present application provides a method, device and equipment for determining the grasping posture of an object by multi-perspective point cloud fusion, the method comprising: obtaining point cloud data of multiple objects at different perspectives, the point cloud data comprising three-dimensional coordinates of multiple points, for reflecting the geometric shape and spatial distribution of the objects; fusing the point cloud data of multiple objects at different perspectives to obtain fused point cloud data of multiple objects; and performing feature fusion on the point cloud data of multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of multiple objects; performing point cloud instance segmentation on the fused point cloud data of multiple objects according to the perspective fusion point cloud features of multiple objects through an instance segmentation network model to obtain instance point cloud data corresponding to each object; determining the grasping posture of each object according to the instance point cloud data corresponding to each object, wherein the grasping posture comprises grasping direction, grasping position, grasping angle and grasping width. In an embodiment of the present application, since point cloud data of multiple objects at different perspectives are obtained, perspective fused point cloud features are obtained by fusing the point cloud data at different perspectives, and point cloud instance segmentation is performed on the fused point cloud data of multiple objects to obtain instance point cloud data corresponding to each object; and the point cloud data of multiple objects at different perspectives can avoid problems such as object occlusion, object stacking, and incomplete display of edge objects, thereby improving the accuracy of the determined grasping posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0029] Figure 1 An application scenario diagram of the object grasping posture determination method based on multi-view point cloud fusion provided in this application;

[0030] Figure 2 A flowchart of a method for determining the object grasping posture by multi-view point cloud fusion provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of a process for multi-view point cloud feature extraction and point cloud instance segmentation provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a scene-level point cloud and an object-level point cloud provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of a process flow of a grasping model provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of the structure of a device for determining the object grasping posture by multi-view point cloud fusion provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0036] Figure 8 This is a block diagram of an electronic device according to an embodiment of the present application.

[0037] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0038] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances must be provided for users to choose to authorize or refuse.

[0040] With the development of robotics, when assembling a product through a robot, the robot can automatically grasp the various parts and objects that make up the product. Before the robot grasps the parts and objects, it needs to first determine the grasping posture of the parts and objects.

[0041] In existing technologies, point cloud data from a single perspective is first acquired. This point cloud data is then fed into a grasping detection model, which then determines the grasping pose of each object. Point cloud data refers to a collection of discrete points on an object's surface acquired using 3D scanning equipment (such as LiDAR, 3D scanners, and depth cameras). Point cloud data is a direct representation of the object's surface in three-dimensional space. Each point in the point cloud data contains three-dimensional coordinates (X, Y, Z), accurately reflecting the object's geometric shape and spatial distribution.

[0042] However, the input to the aforementioned grasp detection models is single-view point cloud data. Due to the limitations of single-view point cloud data, object occlusion, object stacking, and incomplete representation of edge objects are inevitable. This can cause some smaller objects to be overlooked when determining the grasp pose, resulting in poor grasp quality or even no grasp of small objects. Consequently, the grasp poses of smaller objects may be missing in the final grasping scene, affecting overall grasping accuracy.

[0043] Furthermore, due to incomplete display of some objects, insufficient positional information is available. Consequently, the features extracted during the grasping process are insufficient to fully represent the positional relationships of the entire scene, affecting the robot's grasping success rate. Furthermore, since some objects are invisible, the robot's gripper may collide with the object during the grasping process, potentially causing dangerous situations.

[0044] In response to the technical problems in the above-mentioned prior art, the inventors propose the following technical concept: This invention proposes a multi-view point cloud fusion segmentation and capture framework. Figure 1 As shown in Figure 1, the framework includes a multi-view point cloud fusion module, a point cloud instance segmentation module, and a multi-scale grasp detection module. Leveraging information from point clouds from multiple different viewpoints, the multi-view point cloud fusion module utilizes feature interpolation and feature propagation to fuse features from multiple viewpoints, resulting in a complete point cloud with comprehensive location information. This solves the problem of incomplete location information from a single viewpoint. The point cloud with complete location information is then used for instance segmentation by the point cloud instance segmentation module. First, the foreground and background points of the point cloud are separated and noise points are filtered out to obtain a pure point cloud. The foreground point cloud is then further instance segmented to obtain the point cloud corresponding to each object, resolving the computational overhead caused by background and noise points, which can affect grasp success. Simultaneously, through instance segmentation, scene-level point clouds can be converted to object-level point clouds. Using the object-level point cloud converted from the scene-level point cloud for grasp detection solves the problem of small objects with few points being overlooked during grasp detection, ensuring that every object is detected. At the same time, in order to further enrich the features extracted in the feature extraction stage, a multi-scale feature extraction module is designed for feature extraction and fusion to ensure that the point cloud feature information is rich during the segmentation and grasping detection process.

[0045] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0046] Figure 2This is a flow chart of a method for determining the position and posture of an object grasped by multi-view point cloud fusion provided in an embodiment of the present application. This method can be executed by a device for determining the position and posture of an object grasped by multi-view point cloud fusion. The device can be a server or an electronic device. The following description will take an electronic device as an example. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware, such as Figure 2 As shown, the method includes the following steps:

[0047] S201. Obtain point cloud data of multiple objects at different viewing angles. The point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects.

[0048] In the embodiment of the present application, it is first necessary to obtain a basic data set, which can be point cloud data of multiple objects at multiple different perspectives. The basic data set can also be depth maps of multiple objects at multiple different perspectives, and then the point cloud data is constructed from the depth maps. It should be noted that steps S201 to S204 determine the grasping posture of each object in a scene with multiple objects. This grasping posture determination method is also suitable for determining the grasping posture of a single object in a scene with a single object.

[0049] The different viewing angles may be a plurality of preset viewing angles. The number of the plurality of preset viewing angles is not specifically limited in this application and can be set and modified as needed.

[0050] S202. Fusing point cloud data of multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and performing feature fusion on the point cloud data of multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects.

[0051] In the embodiments of this application, multi-view fused point cloud data can be constructed using single-view point clouds from a basic dataset. The point cloud data for each current view is in the camera coordinate system. Using camera parameters and the coordinate transformation matrix between point clouds from different viewpoints, the coordinate transformation matrix from the camera coordinate system to the world coordinate system can be derived. In other words, by transforming the camera coordinate system to the world coordinate system, the multi-view point clouds can be unified into the same coordinate system through coordinate transformation, achieving multi-view point cloud fusion.

[0052] In some embodiments, the point cloud data includes the three-dimensional coordinates of multiple points in the camera coordinate system. Accordingly, the point cloud data of multiple objects at different perspectives are fused to obtain fused point cloud data of the multiple objects, including: according to the camera parameters and the coordinate transformation matrix of different perspectives, the three-dimensional coordinates of each point in the point cloud data corresponding to each perspective in the camera coordinate system are converted to the three-dimensional coordinates in the world coordinate system to obtain the point cloud data of each perspective after the coordinate conversion; the point cloud data of each perspective after the coordinate conversion are fused, and the fused point cloud data of the multiple objects are obtained by downsampling through the dynamic voxel method.

[0053] The coordinate transformation formula for point cloud transformation is as follows. First, the point clouds of each view are unified from the camera view to the view of camera 0, and then unified from the view of camera 0 to the world coordinate system, which can be expressed by the following formula:

[0054]

[0055] in, Represents the point cloud coordinates of each viewpoint, Indicates the coordinate transformation of the point cloud coordinates of each view to the camera 0 view, Represents the coordinate transformation from the camera 0 perspective to the world coordinate system.

[0056] In the disclosed embodiment, the point cloud data after multi-view fusion will have the number of points doubled. Therefore, after point cloud fusion, the fused point cloud is further downsampled using the dynamic voxel method to obtain fused point cloud data with an appropriate amount of data. Each single-view point cloud corresponds to a segmentation and capture label, so during the multi-view point cloud fusion process, the label data is matched with the points in the point cloud through dimensional expansion. When the point cloud is fused from multiple perspectives, the segmentation and capture labels are also fused synchronously, and there will be no confusion in the respective labels.

[0057] In the embodiment of the present application, in order to solve the problems of occlusion, stacking and incomplete parts encountered when grasping objects under single-view conditions, a multi-view fusion dataset is constructed to realize multi-view point cloud information fusion. This method uses multi-view fusion data to extract local features of a single view and global features of a fused view from point cloud data. Combined with the extracted multi-scale features, the multi-view point cloud feature fusion and point cloud feature propagation methods are used to fuse the multi-view point cloud data and the multi-view point cloud features to complete the single-view point cloud information and obtain complete point cloud feature information. This part can be divided into two parts: point cloud feature extraction and point cloud feature fusion.

[0058] Accordingly, the point cloud data of multiple objects at different perspectives are fused using a point cloud fusion network model to obtain the perspective fused point cloud features of multiple objects, including the following steps (1) to (2):

[0059] (1) Extract the single-view high-dimensional features corresponding to the point cloud data of each view, and extract the perspective fusion high-dimensional features corresponding to the fused point cloud data.

[0060] In this step, based on the unique characteristics of point cloud features, differentiated feature extraction strategies are adopted for single-view point clouds and multi-view fused point clouds. For single-view point clouds, a lightweight feature extraction network is used to capture local geometric details to address challenges such as occlusion and local noise. For multi-view fused point clouds, a more complex deep feature extraction network is used to comprehensively capture the global geometric structure and semantic features in the multi-view information, enhancing the understanding of the overall shape and position of the object.

[0061] Optionally, single-perspective high-dimensional features corresponding to the point cloud data of each perspective are extracted, and perspective fusion high-dimensional features corresponding to the fused point cloud data are extracted, including: using a lightweight feature extraction network to effectively extract high-dimensional features through a multi-scale grouping strategy to obtain single-perspective high-dimensional features corresponding to the point cloud data of each perspective; and using a deep feature extraction network to capture the global geometric structure in multi-perspective information to obtain fusion perspective high-dimensional features corresponding to the fused point cloud data.

[0062] For example, Figure 3 As shown, the lightweight feature extraction network can be a PointNet++ network model or a PointCNN network model. The lightweight network structure model can effectively extract local features through a multi-scale grouping strategy, thereby providing robustness to noise and occlusion.

[0063] For example, Figure 3 As shown in the figure, the deep feature extraction network can be a multi-scale Point Transformer (point cloud feature fusion model) or PV-RCNN (Point-Voxel Feature Set Abstraction for 3D Object Detection‌, a point voxel feature set abstraction network model for three-dimensional target detection). Among them, the deep feature extraction network can better handle the complexity and richness of multi-view data.

[0064] (2) The single-view high-dimensional features corresponding to multiple viewpoints are fused with the view fusion high-dimensional features to obtain the view fusion point cloud features of multiple objects.

[0065] In this step, after completing the single-view point cloud feature extraction and fused point cloud feature extraction, the key to this module is to use the point cloud feature propagation mechanism to achieve the fusion of multi-view point cloud features. Accordingly, the single-view high-dimensional features corresponding to each of the multiple viewpoints are fused with the view fusion high-dimensional features to obtain the view fusion point cloud features of multiple objects, including: point-by-point alignment of the single-view point cloud features corresponding to each viewpoint with the view fusion high-dimensional features. Through the attention feature propagation model, the single-view high-dimensional features corresponding to each viewpoint are propagated into the view fusion high-dimensional features to obtain the view fusion point cloud features of multiple objects; wherein, the attention feature propagation model is used to autonomously select the specific features in each viewpoint based on the feature propagation mechanism, enhance the expression of specific features, and ignore the overlapping parts in different viewpoints, so as to achieve feature fusion of the single-view high-dimensional features extracted from different viewpoints and the view fusion high-dimensional features.

[0066] It's important to note that achieving multi-viewpoint point cloud feature fusion requires point-by-point alignment. Failure to achieve point-by-point alignment during the point cloud feature fusion process will result in a chaotic state of the fused point cloud features, rendering subsequent processing impossible. Point-by-point feature alignment can be achieved using methods such as the ICP (Iterative Closest Point) algorithm, point cloud coordinate position transformation, and feature matching-based alignment.

[0067] Optionally, the feature propagation mechanism in the PointNet++ network model can propagate the features of one point cloud to another point cloud and fuse the features of the two point clouds to achieve point-by-point feature alignment and feature propagation of point clouds.

[0068] It should be noted that the number of single-view high-dimensional features corresponding to multiple viewpoints is multiple, and the weight of each single-view point cloud feature can be configured after the model is trained. Figure 3 As shown in FIG, the number of single-view high-dimensional features corresponding to each of the multiple viewpoints is 3, and the weight of each single-view point cloud feature can be 0.5, 0.2, and 0.3.

[0069] It is understandable that Figure 3 The number of features in the figure is only an example of the feature extraction process and the feature fusion process. In actual application scenarios, the number of features in the feature extraction process and the number of features in the feature fusion process should be consistent.

[0070] S203 , using an instance segmentation network model to fuse point cloud features according to the perspectives of multiple objects, performing point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtaining instance point cloud data corresponding to each object.

[0071] In the embodiment of the present application, after successfully achieving the fusion and alignment of multi-view point cloud features, the fused feature segmentation stage uses the fused features to perform accurate instance segmentation so that different objects can be accurately identified and distinguished in the grasping scene. To ensure the accuracy of segmentation, the segmentation network based on the fused features is combined with the point cloud neighborhood features and point cloud position to control the consistency of point cloud segmentation. At the same time, the classification, boundary and instance center loss are combined, and the instance segmentation effect is enhanced to ensure that instances of different objects can be accurately segmented to assist in the subsequent grasping process.

[0072] Optionally, point cloud instance segmentation is performed on the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects through an instance segmentation network model to obtain instance point cloud data corresponding to each object, including: determining the foreground point cloud data, background point cloud data and noise point cloud data in the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects; removing the background point cloud data and noise point cloud data in the fused point cloud data to obtain pure foreground point cloud data corresponding to the multiple objects; performing point cloud instance segmentation on the pure foreground point cloud corresponding to the multiple objects in combination with the position information of each object through the instance segmentation network model to obtain instance point cloud data corresponding to each object.

[0073] In an embodiment of the present application, first, using the complete point cloud feature information, instance segmentation is performed to distinguish the point cloud background and foreground through point cloud segmentation, thereby reducing the interference of background noise on subsequent grasping tasks. In this way, a pure foreground point cloud with complete perspective information and not interfered by background noise is obtained. Then, the pure point cloud can continue to be segmented according to the object instance to obtain the instance point cloud corresponding to each object, and the instance point cloud corresponding to each object can be fused to obtain a pure foreground point cloud. The fusion of pure instance point clouds and the further segmentation of foreground point clouds realize arbitrary transformation of scene-level point clouds and object-level point clouds. Among them, the use of object-level point clouds for grasping detection can make up for the problem that small-volume objects are easily ignored when performing grasping detection on scene-level point clouds. For example, scene-level point clouds and object-level point clouds are as follows: Figure 4 shown.

[0074] S204: Determine a grasping posture of each object based on the instance point cloud data corresponding to each object, wherein the grasping posture includes a grasping direction, a grasping position, a grasping angle, and a grasping width.

[0075] During the grasp detection process, some objects have complex surface features, and some are small in size. Even if comprehensive spatial position information of components is obtained through multi-view fusion, extracting features at only a single scale will affect the extraction of local detail features, and thus the detection of the grasp pose. To enhance the extraction of local detail features, a multi-scale feature extraction method is necessary. Considering that the point cloud data input to the grasping network is pure multi-view point cloud data, cylinder grouping is used to extract multi-scale features during the grasp prediction stage. Based on this, a multi-scale cylinder grouping, cylinder feature extraction, and feature fusion module are constructed. This module comprises multiple cylinder groups of different sizes, which extract grasp features at different scales. An attention mechanism is then used to focus the grasping network's attention on grasp information at key locations. Finally, a feature fusion module is designed to organically combine this multi-scale information to obtain comprehensive grasp feature information.

[0076] In some embodiments, to further improve the extraction of small object features and enhance grasping accuracy, a multi-sampling number fusion / multi-radius fusion feature extraction method is designed in the global feature extraction stage. This method uses point clouds with multiple levels of sampling numbers and multiple sampling sphere radii to enrich the collected global features and fuses all collected features to ensure a more comprehensive point cloud feature set.

[0077] Accordingly, the grasping posture of each object is determined based on the instance point cloud data corresponding to each object, including:

[0078] (1) For the instance point cloud data corresponding to each object, the global feature extraction network in the crawling detection network model is used to extract features from the instance point cloud data to obtain the global high-dimensional features corresponding to the object.

[0079] In some embodiments, during the grasping pose detection process, feature extraction is performed on the fused point cloud based on a multi-scale Point Transformer global feature extraction network to generate multiple candidate grasping points, and the three-dimensional position of each grasping point and the rotation angle of the grasper are generated through a 6D grasping pose prediction module.

[0080] (2) The grasping detection network model is used to determine multiple candidate grasping points corresponding to the object and the grasping direction corresponding to each candidate grasping point based on the global high-dimensional features corresponding to the object. According to the grasping direction corresponding to each candidate grasping point, the cylindrical grasping area corresponding to the candidate grasping point is determined from the instance point cloud data. The grasping position, grasping angle and grasping width corresponding to each cylindrical grasping area are determined to obtain multiple candidate grasping postures corresponding to the object.

[0081] In some embodiments, as Figure 5As shown in the figure, in the feature extraction stage, Point Transformer (point cloud feature fusion model) can be used to extract multi-scale global features of fused point clouds, using multi-level local geometric information to extract while maintaining global information to ensure that the overall information is not lost. Figure 5 After receiving the global features from the input, Approach Net (a policy network model) uses the input global features to predict the approach vector of the robot arm's grasp, that is, the direction from which the robot arm will approach the object. Using the approach vectors predicted by Approach Net, the system groups the point cloud into cylindrical regions. Each group corresponds to a candidate grasping region, which represents a portion of the object surface that the gripper can cover. The grouped point cloud is used to further analyze the grasping details and ensure that the grasping region is sufficiently stable and free from interference. Cylindrical grouping ensures that the gripper can find a suitable grasping position on complex geometries.

[0082] (3) The grasping quality evaluation network is used to score multiple candidate grasping postures, and the candidate grasping posture with the highest score is selected and determined as the grasping posture of the object.

[0083] In some embodiments, as Figure 5 As shown in the figure, after using the cylinder grouping model to extract and fuse multi-scale features, the Operation Net is used to predict grasping pose parameters such as grasping position, grasping width, and grasping angle. The cylinder grouping model can be a Group Attention model.

[0084] Grasp pose evaluation is a key step in robotic grasping tasks, determining the success of a grasp. After obtaining the grasp direction, grasp position, grasp width, and grasp angle predicted by the grasp network, the grasp quality assessment network scores and screens candidate grasp poses to select the optimal one.

[0085] The present application provides a method for determining the grasping posture of an object by multi-perspective point cloud fusion, the method comprising: acquiring point cloud data of multiple objects at different perspectives, the point cloud data comprising three-dimensional coordinates of multiple points, for reflecting the geometric shape and spatial distribution of the objects; fusing the point cloud data of multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and, performing feature fusion on the point cloud data of multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects; performing point cloud instance segmentation on the fused point cloud data of the multiple objects according to the perspective fusion point cloud features of the multiple objects through an instance segmentation network model to obtain instance point cloud data corresponding to each object; and determining the grasping posture of each object according to the instance point cloud data corresponding to each object, wherein the grasping posture comprises grasping direction, grasping position, grasping angle and grasping width. In an embodiment of the present application, since point cloud data of multiple objects at different perspectives are obtained, perspective fused point cloud features are obtained by fusing the point cloud data at different perspectives, and point cloud instance segmentation is performed on the fused point cloud data of multiple objects to obtain instance point cloud data corresponding to each object; and the point cloud data of multiple objects at different perspectives can avoid problems such as object occlusion, object stacking, and incomplete display of edge objects, thereby improving the accuracy of the determined grasping posture.

[0086] It should be noted that in the embodiments of the present application, (1) the problem of occlusion, stacking and incomplete display of edge objects under a single perspective is solved by fusion of multi-view point cloud data, and the object position information is supplemented. (2) The object points of the point cloud are extracted by point cloud instance segmentation to filter out background points and noise points, which solves the problem that background points and noise points exist in the grasping scene, introducing unnecessary calculations and affecting the grasping detection efficiency. (3) Point cloud instance segmentation can obtain pure instance point clouds, thereby realizing the conversion between scene-level grasping and object-level grasping through the point clouds of each instance, and focusing on smaller objects through object-level grasping to avoid small objects being ignored. (4) The multi-scale grasping detection module can use different radii to perform multi-scale extraction and fusion of global features and grasped local features, enrich the grasping information extracted by the network, and ensure that the grasping information is rich and complete.

[0087] In the multi-view point cloud fusion segmentation and grasping framework of the embodiment of this application, segmentation loss and grasping loss are key components used to improve segmentation accuracy and grasping effect. The following is a detailed description of the design and function of various loss functions. Among them, various loss functions include basic loss functions, segmentation loss functions, and grasping loss functions. Basic loss functions include: cross entropy loss function, smoothed L1 loss function, and mean square error loss function.

[0088] (1) Introduction to basic loss function:

[0089] 1. Cross entropy loss (CE Loss), which is used to measure the difference between the predicted probability distribution and the true label, can be expressed by the following formula:

[0090]

[0091] Among them, y ic is the true label, is the predicted probability, N is the number of points in the point cloud, and C is the number of object categories.

[0092] 2. Smooth L1 loss, suitable for regression tasks, is robust to outliers and can be expressed by the following formula:

[0093]

[0094] Huber Loss is a generalized form of smooth L1 loss:

[0095]

[0096] Among them, x n and y n Represent the predicted coordinates and actual coordinates respectively.

[0097] 3. Mean Square Error Loss:

[0098]

[0099] Among them, x n and y n Represent the predicted coordinates and actual coordinates respectively.

[0100] (2) Introduction to segmentation loss function:

[0101] The segmentation loss function measures the difference between the predicted segmentation result and the true label. A commonly used loss function is cross entropy loss. To improve the stability and accuracy of the segmented region, after introducing cross entropy loss on the segmentation label, center offset loss and spatial consistency loss are also introduced. Combining these three loss functions improves segmentation accuracy.

[0102] Optionally, spatial consistency loss is used to ensure that adjacent points have similar labels. This loss function is implemented by calculating the probability difference between each point and its neighboring points, which can be expressed by the following formula:

[0103]

[0104] Among them, N represents the total number of points in the point cloud, K represents the neighboring points considered when calculating the loss, is the predicted probability of the i-th point, is the predicted probability of the jth neighboring point of the i-th point.

[0105] Segmentation label loss:

[0106] Center shift loss:

[0107] The final segmentation loss can be expressed as:

[0108] in, is the balance coefficient, which is used to balance the importance of different losses.

[0109] (3) Introduction to crawling loss function:

[0110] The grasping loss function is used to measure the difference between the predicted grasping pose and the actual grasping pose. The points obtained by instance segmentation can be used as grasping points. First, the grasping direction loss is calculated for each grasping point:

[0111]

[0112] Among them, V ij represents the jth predicted grasping direction vector of the i-th graspable point. ij : The j-th prediction confidence score of the i-th graspable point. Regression loss (L reg ): The mean square error loss function is used to calculate the loss between the predicted grasp direction vector and the true grasp direction vector, but only the prediction results within 5 degrees of the true grasp direction vector are considered.

[0113] The crawling parameter loss can be expressed as:

[0114]

[0115]

[0116]

[0117] Among them, L angle represents the grasping rotation angle loss, L width Crawl width loss, L score represents the grasp confidence loss.

[0118] Complete crawl loss:

[0119]

[0120] Combining segmentation loss and grasping loss, we design a complete loss function:

[0121]

[0122] in, is the balance coefficient, which is used to balance the importance of different losses.

[0123] In the disclosed embodiment, by optimizing the loss function, the multi-view point cloud fusion segmentation and grasping framework can be effectively trained to improve the accuracy of segmentation and grasping.

[0124] Figure 6 This is a schematic diagram of the structure of a device for determining the object grasping posture by fusion of multi-view point clouds provided in an embodiment of the present application. Figure 6 The device for determining the object grasping posture by multi-view point cloud fusion includes: an acquisition module 601, a feature fusion module 602, a point cloud instance segmentation module 603 and a determination module 604;

[0125] The acquisition module 601 is used to acquire point cloud data of multiple objects at different viewing angles. The point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects.

[0126] a feature fusion module 602 for fusing the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and performing feature fusion on the point cloud data of the multiple objects at different perspectives using a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects;

[0127] The point cloud instance segmentation module 603 is configured to fuse point cloud features of the multiple objects according to the perspectives of the multiple objects using an instance segmentation network model, perform point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtain instance point cloud data corresponding to each object;

[0128] The determination module 604 is configured to determine a grasping posture of each object based on the instance point cloud data corresponding to each object, wherein the grasping posture includes a grasping direction, a grasping position, a grasping angle, and a grasping width.

[0129] In an optional embodiment, the point cloud data includes three-dimensional coordinates of multiple points in a camera coordinate system. Accordingly, the feature fusion module 602 fuses the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects, including: converting the three-dimensional coordinates of each point in the point cloud data corresponding to each perspective in the camera coordinate system into three-dimensional coordinates in the world coordinate system according to camera parameters and coordinate transformation matrices of different perspectives, to obtain point cloud data of each perspective after coordinate conversion; fusing the point cloud data of each perspective after coordinate conversion, and obtaining fused point cloud data of the multiple objects by downsampling through the dynamic voxel method.

[0130] In an optional embodiment, the feature fusion module 602 performs feature fusion on the point cloud data of the multiple objects at different perspectives through a point cloud fusion network model to obtain perspective fusion point cloud features of the multiple objects, including: extracting single-perspective high-dimensional features corresponding to the point cloud data of each perspective, and extracting perspective fusion high-dimensional features corresponding to the fused point cloud data; and performing feature fusion on the single-perspective high-dimensional features corresponding to each of the multiple perspectives with the perspective fusion high-dimensional features to obtain perspective fusion point cloud features of the multiple objects.

[0131] In an optional embodiment, the feature fusion module 602 extracts single-perspective high-dimensional features corresponding to the point cloud data of each perspective, and extracts perspective fusion high-dimensional features corresponding to the fused point cloud data, including: using a lightweight feature extraction network to effectively extract high-dimensional features through a multi-scale grouping strategy to obtain single-perspective high-dimensional features corresponding to the point cloud data of each perspective; and using a deep feature extraction network to capture the global geometric structure in multi-perspective information to obtain fusion perspective high-dimensional features corresponding to the fused point cloud data.

[0132] In an optional embodiment, the feature fusion module 602 performs feature fusion on the single-view high-dimensional features corresponding to each of the multiple viewpoints with the view-fusion high-dimensional features to obtain the view-fusion point cloud features of the multiple objects, specifically including: point-by-point aligning the single-view point cloud features corresponding to each viewpoint with the view-fusion high-dimensional features; propagating the single-view high-dimensional features corresponding to each viewpoint into the view-fusion high-dimensional features through an attention feature propagation model to obtain the view-fusion point cloud features of the multiple objects; wherein the attention feature propagation model is used to autonomously select specific features in each viewpoint based on the feature propagation mechanism, enhance the expression of the specific features, and ignore the overlapping parts in different viewpoints, so as to realize the feature fusion of the single-view high-dimensional features extracted from different viewpoints and the view-fusion high-dimensional features.

[0133] In an optional embodiment, the point cloud instance segmentation module 603 performs point cloud instance segmentation on the fused point cloud data of the multiple items according to the perspective fusion point cloud features of the multiple items through an instance segmentation network model to obtain instance point cloud data corresponding to each item, including: determining the foreground point cloud data, background point cloud data and noise point cloud data in the fused point cloud data of the multiple items according to the perspective fusion point cloud features of the multiple items; removing the background point cloud data and noise point cloud data in the fused point cloud data to obtain pure foreground point cloud data corresponding to the multiple items; performing point cloud instance segmentation on the pure foreground point cloud corresponding to the multiple items in combination with the position information of each item through an instance segmentation network model to obtain instance point cloud data corresponding to each item.

[0134] In an optional embodiment, the determination module 604 determines the grasping posture of each item based on the instance point cloud data corresponding to each item, including: for the instance point cloud data corresponding to each item, performing feature extraction on the instance point cloud data through the global feature extraction network in the grasping detection network model to obtain the global high-dimensional features corresponding to the item; determining multiple candidate grasping points corresponding to the item and the grasping direction corresponding to each candidate grasping point through the grasping detection network model according to the global high-dimensional features corresponding to the item, determining the cylindrical grasping area corresponding to the candidate grasping point from the instance point cloud data according to the grasping direction corresponding to each candidate grasping point, determining the grasping position, grasping angle and grasping width corresponding to each cylindrical grasping area, and obtaining multiple candidate grasping postures corresponding to the item; scoring the multiple candidate grasping postures through the grasping quality evaluation network, screening out the candidate grasping posture with the highest score and determining it as the grasping posture of the item.

[0135] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device of this embodiment may include:

[0136] at least one processor 710; and

[0137] a memory 720 communicatively coupled to the at least one processor;

[0138] The memory 720 stores instructions that can be executed by at least one processor 710 , and the instructions are executed by at least one processor 710 to enable the server to execute a method as described in any of the above embodiments.

[0139] Optionally, the memory 720 may be independent or integrated with the processor 710 .

[0140] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0141] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0142] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is only a logical function division. In actual implementation, other division methods may be used. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not implemented.

[0144] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The software function modules stored in a storage medium include a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some of the steps of the methods of various embodiments of the present application.

[0145] It should be understood that the processor described above may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or as being executed by a combination of hardware and software modules within the processor. The memory may include high-speed random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, or may be a USB flash drive, a mobile hard drive, a read-only memory, a magnetic disk, or an optical disk.

[0146] The aforementioned storage medium may be implemented by any type of volatile or nonvolatile memory device, or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0147] An exemplary storage medium is coupled to a processor, such that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor and storage medium can also exist as discrete components in a server or a host device.

[0148] Figure 8 This is a block diagram of an electronic device shown in an embodiment of the present application. The device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0149] See also Figure 8 , device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0150] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0151] The memory 804 is configured to store various types of data to support the operations of the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0152] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 800.

[0153] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, it may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can detect not only the boundaries of a touch or slide action, but also the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have variable focal length and optical zoom capabilities.

[0154] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0155] The input / output interface 812 provides an interface between the processing component 802 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0156] Sensor assembly 814 includes one or more sensors for providing various status assessments of device 800. For example, sensor assembly 814 can detect the open / closed state of device 800, the relative positioning of components, such as the display and keypad of device 800, and changes in the position of device 800 or a component thereof, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and changes in the temperature of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD), for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0157] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0158] In an exemplary embodiment, the device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0159] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions. The instructions are executable by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a high-speed random access memory (RAM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0160] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-mentioned electronic device method.

[0161] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0162] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for determining the object grasping posture by multi-view point cloud fusion, characterized in that: The method comprises: Acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects; fusing the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and performing feature fusion on the point cloud data of the multiple objects at different perspectives using a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects; fusing point cloud features of the multiple objects according to their perspectives using an instance segmentation network model, performing point cloud instance segmentation on the fused point cloud data of the multiple objects to obtain instance point cloud data corresponding to each object; Determine the grasping posture of each object based on the instance point cloud data corresponding to each object, wherein the grasping posture includes grasping direction, grasping position, grasping angle, and grasping width; The step of fusing the point cloud data of the multiple objects at different perspectives using a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects includes: Extracting single-viewpoint high-dimensional features corresponding to the point cloud data of each viewpoint, and extracting perspective fusion high-dimensional features corresponding to the fused point cloud data; Aligning the single-view point cloud features corresponding to each view with the view fusion high-dimensional features point by point; Through the attention feature propagation model, the single-view high-dimensional features corresponding to each view are propagated to the view fusion high-dimensional features to obtain the view fusion point cloud features of the multiple objects; wherein, the attention feature propagation model is used to autonomously select the specific features in each view on the basis of the feature propagation mechanism, enhance the expression of the specific features, and ignore the overlapping parts in different viewpoints, so as to realize the feature fusion of the single-view high-dimensional features extracted from different viewpoints and the view fusion high-dimensional features.

2. The method for determining the object grasping posture according to claim 1, characterized in that: The point cloud data includes three-dimensional coordinates of a plurality of points in a camera coordinate system. Accordingly, fusing the point cloud data of the plurality of objects at different viewing angles to obtain fused point cloud data of the plurality of objects includes: According to the camera parameters and the coordinate transformation matrix of different viewing angles, the three-dimensional coordinates of each point in the point cloud data corresponding to each viewing angle in the camera coordinate system are converted to the three-dimensional coordinates in the world coordinate system, and the point cloud data of each viewing angle after the coordinate conversion is obtained; The point cloud data of each perspective after coordinate transformation are fused, and the fused point cloud data of the multiple objects are obtained by downsampling using a dynamic voxel method.

3. The method for determining the object grasping posture according to claim 1, characterized in that: The extracting of single-viewpoint high-dimensional features corresponding to the point cloud data of each viewpoint, and the extracting of perspective fusion high-dimensional features corresponding to the fused point cloud data, include: Through a lightweight feature extraction network, high-dimensional features are effectively extracted through a multi-scale grouping strategy to obtain single-view high-dimensional features corresponding to the point cloud data of each view; and through a deep feature extraction network, the global geometric structure in the multi-view information is captured to obtain the fused view high-dimensional features corresponding to the fused point cloud data.

4. The method for determining the object grasping posture according to claim 1, characterized in that: The example segmentation network model is used to fuse point cloud features according to the perspectives of the multiple objects, and performs point cloud instance segmentation on the fused point cloud data of the multiple objects to obtain instance point cloud data corresponding to each object, including: Determining foreground point cloud data, background point cloud data, and noise point cloud data in the fused point cloud data of the multiple objects according to the perspective fused point cloud features of the multiple objects; removing background point cloud data and noise point cloud data from the fused point cloud data to obtain pure foreground point cloud data corresponding to the plurality of objects; The instance segmentation network model is combined with the location information of each object to perform point cloud instance segmentation on the pure foreground point clouds corresponding to the multiple objects to obtain instance point cloud data corresponding to each object.

5. The method for determining the object grasping posture according to claim 1, characterized in that: Determining the grasping posture of each object based on the instance point cloud data corresponding to each object includes: For each instance point cloud data corresponding to an object, feature extraction is performed on the instance point cloud data through the global feature extraction network in the crawling detection network model to obtain the global high-dimensional features corresponding to the object; Determine, through the grasp detection network model, multiple candidate grasping points corresponding to the object and the grasping direction corresponding to each candidate grasping point based on the global high-dimensional features corresponding to the object; determine, based on the grasping direction corresponding to each candidate grasping point, the cylindrical grasping area corresponding to the candidate grasping point from the instance point cloud data; determine the grasping position, grasping angle, and grasping width corresponding to each cylindrical grasping area, and obtain multiple candidate grasping postures corresponding to the object; The plurality of candidate grasping postures are scored using a grasping quality evaluation network, and the candidate grasping posture with the highest score is selected and determined as the grasping posture of the object.

6. A device for determining object grasping posture based on multi-view point cloud fusion, characterized in that: The device comprises: An acquisition module is used to acquire point cloud data of multiple objects at different viewing angles, wherein the point cloud data includes three-dimensional coordinates of multiple points, which are used to reflect the geometric shape and spatial distribution of the objects; a feature fusion module, configured to fuse the point cloud data of the multiple objects at different perspectives to obtain fused point cloud data of the multiple objects; and perform feature fusion on the point cloud data of the multiple objects at different perspectives using a point cloud fusion network model to obtain perspective fused point cloud features of the multiple objects; a point cloud instance segmentation module, configured to fuse point cloud features of the multiple objects according to the perspectives of the multiple objects using an instance segmentation network model, perform point cloud instance segmentation on the fused point cloud data of the multiple objects, and obtain instance point cloud data corresponding to each object; a determination module, configured to determine a grasping posture of each object based on instance point cloud data corresponding to each object, wherein the grasping posture includes a grasping direction, a grasping position, a grasping angle, and a grasping width; The feature fusion module is specifically used to extract single-viewpoint high-dimensional features corresponding to the point cloud data of each viewpoint, and to extract perspective fusion high-dimensional features corresponding to the fused point cloud data; Aligning the single-view point cloud features corresponding to each view with the view fusion high-dimensional features point by point; Through the attention feature propagation model, the single-view high-dimensional features corresponding to each view are propagated to the view fusion high-dimensional features to obtain the view fusion point cloud features of the multiple objects; wherein, the attention feature propagation model is used to autonomously select the specific features in each view on the basis of the feature propagation mechanism, enhance the expression of the specific features, and ignore the overlapping parts in different viewpoints, so as to realize the feature fusion of the single-view high-dimensional features extracted from different viewpoints and the view fusion high-dimensional features.

7. An electronic device, characterized in that: include: a processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; When executing the computer-executable instructions, the processor is used to implement the method for determining the object grasping posture by multi-view point cloud fusion as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for determining the object grasping posture by multi-view point cloud fusion as described in any one of claims 1 to 4.

Images