Method and apparatus for analyzing grasping pose of robotic arm, and electronic device

Abstract

A method and apparatus for analyzing the grasping pose of a robotic arm, and an electronic device. The method for analyzing the grasping pose of a robotic arm comprises: acquiring a target object point cloud image; filtering the target object point cloud image to obtain a preprocessed target object point cloud image; inputting the preprocessed target object point cloud image into an object detection model for analysis, so as to obtain grasping poses of a robotic arm; and filtering the grasping poses of the robotic arm to obtain a target grasping pose of the robotic arm.

Description

A robot arm grasping posture analysis method and device and electronic equipment TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a robot arm grasping posture analysis method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of artificial intelligence and robotics, robots are increasingly widely used in industrial manufacturing, daily life, and dangerous environment operations. In these applications, the precise operation of the robot arm, especially the grasping action of the object, is the key to achieving efficient automation. Traditional robot arm grasping methods mostly rely on pre-set programs or fixed patterns, which are not suitable for handling complex or variable objects. Therefore, developing an intelligent robot arm grasping posture analysis system that can analyze the shape, size, position and optimal grasping point of an object in real time is of great significance to improve the flexibility and accuracy of robot operation.

[0003] Currently, most grasping detection algorithms such as Generative Grasping Convolutional Neural Network (GGCNN) in the field of robot grasping posture analysis mainly focus on the detection of planar grasping postures. This kind of algorithm identifies and executes the grasping task through a top-down grasping method, but this method does not fully comply with the habits of humans in daily life when grasping objects. Humans often consider multiple angles and directions when grasping objects and adjust the grasping method according to the specific characteristics of the object.

[0004] In addition, the detection algorithm of planar grasping posture often cannot provide the optimal solution when dealing with objects with complex shapes or requiring side grasping and inclined grasping. At the same time, these algorithms usually only rely on input depth images or point cloud data to generate possible optimal grasping postures through models, but due to the lack of recognition of the specific type or attribute of the object, the generated grasping posture is often random and cannot generate a special grasping strategy for a specific object.

[0005] Therefore, a robot arm grasping posture analysis method, device and electronic equipment are proposed.

[0006] SUMMARY

[0007] The present application provides a robot arm grasping posture analysis method, device and electronic equipment, which enables the robot to adapt to different grasping tasks in multiple scenarios, greatly enhancing the flexibility of the robot and the environment interaction, and effectively handling objects with complex shapes and postures.

[0008] The present application provides a robot arm grasping posture analysis method, comprising:

[0009] obtaining a target object point cloud image;

[0010] filtering the target object point cloud image to obtain a preprocessed target object point cloud image;

[0011] inputting the preprocessed target object point cloud image into an object detection model to analyze a robot arm grasping pose;

[0012] filtering the robot arm grasping pose to obtain a robot arm target grasping pose.

[0013] Optionally, the obtaining a target object point cloud image comprises:

[0014] obtaining a scene RGB image through a robot camera;

[0015] analyzing the scene RGB image through an object recognition detection model to obtain a target object RGB image;

[0016] converting the target object RGB image based on an intrinsic matrix of the robot camera to obtain a target object point cloud image.

[0017] Optionally, the converting the target object RGB image based on the intrinsic matrix of the robot camera to obtain the target object point cloud image comprises:

[0018] P(x,y)=K -1 ·[x,y,1] T ·D(x,y)

[0019] wherein P(x,y) is a point in a three-dimensional space of the target object point cloud image, K is the intrinsic matrix of the robot camera, [x,y,1] T is a normalized pixel coordinate of the target object, and D(x,y) is a depth value of the target object.

[0020] Optionally, the filtering the target object point cloud image to obtain the preprocessed target object point cloud image comprises:

[0021] assuming that the size of a voxel is l×w×h, converting a point P i (x i ,y i ,z i ) in the target object point cloud image to a voxel grid coordinate V i comprises:

[0022] replacing each of the voxel grid coordinates with an average position, thereby reducing the total number of points of the target object point cloud image to obtain the preprocessed target object point cloud image.

[0023] Optionally, the average position P avg , comprising:

[0024] wherein the average position P avg N is the total number of voxel grid points.

[0025] Optionally, the pre-processed target object point cloud image is input into an object detection model, and a robot arm grasping pose is obtained through analysis, and the method comprises:

[0026] The pre-processed target object point cloud image is subjected to feature extraction through an encoder, and a coded feature matrix is obtained;

[0027] The coded feature matrix is input into a proximity network sub-model, and a grasping score is output;

[0028] K grasping clusters are generated based on the coded feature matrix, and operation parameters of each grasping cluster are determined through an operation network sub-model, so as to obtain a robot arm grasping pose.

[0029] Optionally, the robot arm grasping pose is filtered to obtain a target robot arm grasping pose, and the method comprises:

[0030] The robot arm grasping pose with a grasping score lower than a first preset value is filtered to obtain a target robot arm grasping pose after preliminary filtering;

[0031] The tolerance of the grasping cluster of the target robot arm grasping pose after preliminary filtering is determined through a tolerance network sub-model, and the method comprises:

[0032] τ k =ToleranceNet(C k )

[0033] wherein τ k is the tolerance of the grasping cluster, and C k is the grasping cluster.

[0034] The target robot arm grasping pose is obtained through secondary filtering based on the target robot arm grasping pose after preliminary filtering, and the method comprises:

[0035] G=Filter({θ1,τ1},{θ2,τ2},…,{θ k ,τ k})

[0036] wherein G is a set of target robot arm grasping poses, and each target robot arm grasping pose comprises a six-degree-of-freedom grasping pose of (x, y, z, rx, rp, py).

[0037] The present specification provides a mechanical arm grasping posture analysis device, comprising:

[0038] An acquisition module 10 is configured to acquire a target object point cloud image.

[0039] A filtering module 20 is configured to perform filtering processing on the target object point cloud image to obtain a preprocessed target object point cloud image.

[0040] A detection module 30 is configured to input the preprocessed target object point cloud image into an object detection model to analyze a mechanical arm grasping posture.

[0041] A filtering module 40 is configured to filter the mechanical arm grasping posture to obtain a target mechanical arm grasping posture.

[0042] Optionally, the acquisition module 10 comprises:

[0043] A scene RGB image is acquired by a robot camera.

[0044] A target object RGB image is obtained by analyzing the scene RGB image by an object recognition detection model.

[0045] The target object RGB image is converted based on an intrinsic matrix of the robot camera to obtain a target object point cloud image.

[0046] Optionally, the conversion of the target object RGB image based on the intrinsic matrix of the robot camera to obtain the target object point cloud image comprises:

[0047] P(x, y) = K -1 ·[x, y, 1] T ·D(x, y)

[0048] wherein P(x, y) is a point in a three-dimensional space of the target object point cloud image, K is an intrinsic matrix of the robot camera, [x, y, 1] T is a normalized pixel coordinate of the target object, and D(x, y) is a depth value of the target object.

[0049] Optionally, the filtering module 20 comprises:

[0050] A voxel size is set as l x w x h, and a point P i (x i ,y i ,z i ) in the target object point cloud image is converted to a voxel grid coordinate V i , comprising:

[0051] Replace each of the voxel grid coordinates with an average position, thereby reducing the total number of points of the target object point cloud image, to obtain a preprocessed target object point cloud image.

[0052] Optionally, the replacing each of the voxel grid coordinates with an average position P avg , comprising:

[0053] wherein the average position P avg , N is the total number of voxel grid points.

[0054] Optionally, the detection module 30, comprising:

[0055] extracting features of the preprocessed target object point cloud image through an encoder to obtain an encoded feature matrix;

[0056] inputting the encoded feature matrix into a proximity network submodel to output a grasping score;

[0057] generating K grasping clusters based on the encoded feature matrix, and determining operation parameters of each grasping cluster through an operation network submodel, thereby obtaining a robot grasping pose.

[0058] Optionally, the filtering module 40, comprising:

[0059] filtering the robot grasping pose with a grasping score lower than a first preset value to obtain a robot preliminarily filtered target grasping pose;

[0060] determining a tolerance of the grasping cluster of the robot preliminarily filtered target grasping pose through a tolerance network submodel, comprising:

[0061] τ k =ToleranceNet(C k )

[0062] wherein τ k is the tolerance of the grasping cluster, and C k is the grasping cluster.

[0063] based on the robot preliminarily filtered target grasping pose, to obtain a robot target grasping pose, comprising:

[0064] G=Filter({θ1,τ1},{θ2,τ2},…,{θ k ,τ k})

[0065] wherein G is a set of robot target grasping poses, each robot target grasping pose comprising a six-degree-of-freedom grasping pose of (x, y, z, rx, rp, py).

[0066] The specification also provides an electronic device, wherein the electronic device includes:

[0067] a processor; and

[0068] a memory storing computer-executable instructions that, when executed, cause the processor to perform any of the above methods.

[0069] The specification also provides a computer-readable storage medium storing one or more programs, which when executed by a processor, implement any of the above methods.

[0070] In the present application, the six-degree-of-freedom grasping posture enables the robot to adapt to different grasping tasks in multiple scenarios, greatly enhancing the flexibility of the robot in interacting with the environment, and effectively handling objects of complex shape and posture. Through advanced deep learning technology, the point cloud image of the specified object can be accurately recognized and segmented, realizing accurate grasping of the specific object and ensuring the accuracy of the grasping action and the recognition rate of the target object. Through the above content, the mobile robot can automatically perform complex grasping tasks without direct human intervention, and has wide application prospects in the fields of automated logistics, home service robots, remote operation and the like. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0072] FIG. 1 is a schematic diagram of a mechanical arm grasping posture analysis method according to an embodiment of the present specification;

[0073] FIG. 2 is a structural schematic diagram of a mechanical arm grasping posture analysis device according to an embodiment of the present specification;

[0074] FIG. 3 is a structural schematic diagram of an electronic device according to an embodiment of the present specification;

[0075] FIG. 4 is a schematic diagram of a computer-readable medium according to an embodiment of the present specification. DETAILED DESCRIPTION

[0076] The following description is provided so as to enable any person skilled in the art to practice the application. The preferred embodiments described herein are only examples of the application and the application is not limited to these embodiments. Various modifications to these embodiments can be made by those skilled in the art without departing from the spirit and scope of the application. Those skilled in the art will further appreciate that the application can be used in other embodiments and equivalents thereof without departing from the spirit and scope of the application.

[0077] Exemplary embodiments of the present application are described herein below with reference to Figs. 1-4. However, exemplary embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the inventive concept to those skilled in the art. Like reference numerals refer to like elements throughout the specification. Repetitive descriptions of like elements will be omitted for sake of brevity.

[0078] In the case of a certain specific embodiment, the features, structures, characteristics or other details described do not exclude the possibility of being combined in one or more other embodiments in a suitable manner, in accordance with the technical concept of the present application.

[0079] In the description of specific embodiments, the features, structures, characteristics or other details described are intended to enable a person skilled in the art to fully understand the embodiments. However, it does not exclude the possibility that one or more of the specific features, structures, characteristics or other details can not be practiced by a person skilled in the art without the specific feature, structure, characteristic or other detail.

[0080] The flowcharts shown in the drawings are only exemplary illustrations and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further broken down, while some operations / steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0081] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0082] The term "and / or" or "and / or" includes all combinations of one or more of the associated listed items.

[0083] Fig. 1 is a schematic diagram of a mechanical arm grasping posture analysis method provided by the embodiments of the present specification. The method can include:

[0084] S110: Obtain a target object point cloud image;

[0085] Optionally, the S110 comprises:

[0086] acquire a scene RGB image through a robot camera;

[0087] analyze the scene RGB image through an object recognition detection model to obtain a target object RGB image;

[0088] convert the target object RGB image based on an intrinsic matrix of the robot camera to obtain a target object point cloud image.

[0089] In the specific embodiments of the present specification, the object recognition detection model comprises a pre-trained YOLOv8, SSD, Faster R-CNN, and the YOLOv8 model can quickly locate the top object in the image and generate a corresponding bounding box.

[0090] YOLOv8 is a model that supports all-around visual AI tasks, including detection, segmentation, pose estimation, tracking, and classification. YOLOv8 inherits the advantages of the YOLO series, and through a series of innovations and improvements, it provides higher performance and a wider range of applications.

[0091] Optionally, the conversion of the target object RGB image based on the intrinsic matrix of the robot camera to obtain the target object point cloud image comprises:

[0092] P(x,y)=K -1 ·[x,y,1] T ·D(x,y)

[0093] wherein P(x,y) is a point in the three-dimensional space of the target object point cloud image, K is the intrinsic matrix of the robot camera, [x,y,1] T is the normalized pixel coordinates of the target object, and D(x,y) is the depth value of the target object.

[0094] In the specific embodiments of the present specification, the RGB image contains the depth value corresponding to each pixel point, specifically, the depth value is the distance from the camera as the origin to the actual object surface corresponding to the pixel point.

[0095] S120: filter the target object point cloud image to obtain a preprocessed target object point cloud image;

[0096] In the specific implementation of this specification, the purpose of filtering is to remove noise and clutter, which may be introduced due to environmental factors or sensor errors. By removing isolated points that are too far from their surroundings—points that may also be introduced due to environmental factors or sensor errors—voxel mesh filtering divides the point cloud space into a fixed-size voxel mesh. Points within each voxel are replaced with their average positions, effectively reducing the number of points while preserving structural features.

[0097] Optionally, S120 includes:

[0098] Let the size of the voxel be l×w×h, and let the point P in the point cloud image of the target object be... i (x i ,y i ,z i Transform to voxel grid coordinates V i ,include:

[0099] Each voxel grid coordinate is replaced with the average position, thereby reducing the total number of points in the target object point cloud image, resulting in a preprocessed target object point cloud image.

[0100] Optionally, the step of replacing each voxel grid coordinate with the average position P avg ,include:

[0101] Wherein, the average position P avg N is the total number of voxel grid points.

[0102] S130: Input the preprocessed target object point cloud image into the object detection model and analyze it to obtain the robotic arm grasping posture;

[0103] Optionally, S130 includes:

[0104] The preprocessed target object point cloud image is used to extract features by an encoder to obtain an encoded feature matrix.

[0105] The encoded feature matrix is ​​input into the proximity network sub-model, and the capture score is output.

[0106] K grasping clusters are generated based on the encoded feature matrix, and the operation parameters of each grasping cluster are determined by the operation network sub-model, thereby obtaining the grasping posture of the robotic arm.

[0107] In the specific embodiments of the present specification, the object detection model includes GraspnetCNN, which is a high-efficiency convolutional neural network for real-time grasp detection of low-power devices. Through the design of a double-branch structure and a DDF module, it realizes high grasp detection accuracy and real-time performance under a compact model.

[0108] The input pre-processed target object point cloud image Where each row corresponds to the XYZ coordinates of a point in the point cloud, and the features are extracted by the encoder:

[0109] F = Encoder (P)

[0110] Where, F is the encoded feature matrix, and C is the additional learned features other than spatial coordinates.

[0111] The approach network sub-model includes ApproachNet, which is a network model for predicting the proximity of the grasp point. It extracts features from point cloud data through a series of convolution and pooling operations and outputs a proximity score to filter out effective grasp candidate points. It receives the encoded feature matrix F and outputs the grasp score:

[0112] S = ApproachNet (F)

[0113] Where, S contains the grasp score of each sample.

[0114] Align and cluster the feature samples to generate K clusters:

[0115] {C1, C2, … C K} = Cluster (F)

[0116] Where each C K contains the features of a grasp cluster.

[0117] The operation network sub-model includes OperationNet, which is a network model that is crucial in the robot grasping task. It processes point cloud data to predict the specific parameters of the grasp operation, providing accurate grasp instructions for the robot. The operation parameters of each cluster are determined by OperationNet:

[0118] θ k = OperationNet (C k )

[0119] S140: filtering the robot grasp pose to obtain a target robot grasp pose.

[0120] Optionally, the S140 includes:

[0121] filtering the robot preliminary filtered target grasping pose to obtain a robot target grasping pose, comprising:

[0122] determining the tolerance of the grasping cluster of the robot preliminary filtered target grasping pose through the tolerance network sub-model, comprising:

[0123] τ k =ToleranceNet(C k )

[0124] wherein τ k is the tolerance of the grasping cluster, and C k is the grasping cluster.

[0125] filtering the robot preliminary filtered target grasping pose to obtain a robot target grasping pose, comprising:

[0126] G=Filter({θ1,τ1},{θ2,τ2},…,{θ k ,τ k})

[0127] wherein G is a set of robot target grasping poses, each robot target grasping pose comprising a six-degree-of-freedom grasping pose of (x, y, z, rx, rp, py).

[0128] In the specific embodiments of the present specification, the tolerance network sub-model comprises ToleranceNet, which is a network model for evaluating the tolerance of a robot grasping point. It provides information about the stability of the grasping point to the robot by processing point cloud data and outputting a tolerance score.

[0129] In the present application, the six-degree-of-freedom grasping pose enables the robot to adapt to different grasping tasks in multiple scenarios, greatly enhancing the flexibility of the robot in interacting with the environment, and effectively handling objects of complex shape and posture. Through advanced deep learning technology, the point cloud image of the specified object can be accurately recognized and segmented, enabling precise grasping of the specific object and ensuring the accuracy of the grasping action and the recognition rate of the target object. Through the above content, the mobile robot can automatically perform complex grasping tasks without direct human intervention, and has wide application prospects in the fields of automated logistics, home service robots, and remote operation.

[0130] Fig. 2 is a schematic diagram of a robot grasping pose analysis device provided by an embodiment of the present specification, which can comprise:

[0131] The acquisition module 10 is configured to acquire a target object point cloud image.

[0132] a filtering module 20, configured to perform filtering processing on the target object point cloud image to obtain a preprocessed target object point cloud image;

[0133] a detection module 30, configured to input the preprocessed target object point cloud image into an object detection model to analyze a robot arm grasping pose;

[0134] a filtering module 40, configured to filter the robot arm grasping pose to obtain a target robot arm grasping pose.

[0135] Optionally, the acquisition module 10 comprises:

[0136] acquiring a scene RGB image through a robot camera;

[0137] analyzing the scene RGB image through an object recognition detection model to obtain a target object RGB image;

[0138] converting the target object RGB image based on an intrinsic matrix of the robot camera to obtain a target object point cloud image.

[0139] Optionally, the conversion of the target object RGB image based on the intrinsic matrix of the robot camera to obtain the target object point cloud image comprises:

[0140] P(x, y) = K -1 · [x, y, 1] T · D(x, y)

[0141] wherein P(x, y) is a point in a three-dimensional space of the target object point cloud image, K is the intrinsic matrix of the robot camera, [x, y, 1] T is a normalized pixel coordinate of the target object, and D(x, y) is a depth value of the target object.

[0142] Optionally, the filtering module 20 comprises:

[0143] assuming that the size of a voxel is l x w x h, converting a point P i (x i , y i , z i ) in the target object point cloud image to a voxel grid coordinate V i comprises:

[0144] replacing each voxel grid coordinate with an average position, thereby reducing the total number of points of the target object point cloud image to obtain a preprocessed target object point cloud image.

[0145] Optionally, the replacement of each voxel grid coordinate with an average position Pavg , comprising:

[0146] wherein the average position P avg , N is the total number of voxel grid points.

[0147] Optionally, the detection module 30 comprises:

[0148] The pre-processed target object point cloud image is subjected to feature extraction by an encoder to obtain an encoded feature matrix;

[0149] The encoded feature matrix is input into a proximity network submodel to output a grasping score;

[0150] Based on the encoded feature matrix, K grasping clusters are generated, and an operation network submodel is used to determine the operation parameters of each grasping cluster, thereby obtaining a robot grasping pose.

[0151] Optionally, the filtering module 40 comprises:

[0152] The robot grasping pose with a grasping score lower than a first preset value is filtered to obtain a robot preliminarily filtered target grasping pose;

[0153] The tolerance of the grasping cluster of the robot preliminarily filtered target grasping pose is determined by a tolerance network submodel, comprising:

[0154] τ k =ToleranceNet(C k )

[0155] wherein τ k is the tolerance of the grasping cluster, and C k is the grasping cluster.

[0156] The robot preliminarily filtered target grasping pose is subjected to secondary filtering to obtain a robot target grasping pose, comprising:

[0157] G=Filter({θ1,τ1},{θ2,τ2},…,{θ k ,τ k})

[0158] wherein G is a set of robot target grasping poses, each robot target grasping pose comprising a six-degree-of-freedom grasping pose of (x, y, z, rx, rp, py).

[0159] The functions of the device of the embodiment of the application have been described in the above-mentioned method embodiment, and therefore, the description of the present embodiment will not be described in detail, and the relevant description in the foregoing embodiments can be referred to, which will not be described herein.

[0160] Based on the same inventive concept, the embodiments of the present specification also provide an electronic device.

[0161] The embodiments of the electronic device of the present application are described below, which can be regarded as a specific physical implementation of the above-mentioned method and device embodiments of the present application. For the details described in the electronic device embodiments of the present application, it should be regarded as a supplement to the above-mentioned method or device embodiments; for the details not disclosed in the electronic device embodiments of the present application, it can be realized by referring to the above-mentioned method or device embodiments.

[0162] FIG. 3 is a structural schematic diagram of an electronic device provided by an embodiment of the present specification. The electronic device 300 according to the embodiment of the present application is described below with reference to FIG. 3. The electronic device 300 shown in FIG. 3 is only an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0163] As shown in FIG. 3, the electronic device 300 is in the form of a general computing device. The components of the electronic device 300 can include but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components including the storage unit 320 and the processing unit 310, a display unit 340, etc.

[0164] The storage unit stores program codes which can be executed by the processing unit 310, so that the processing unit 310 executes the steps according to various exemplary embodiments of the present application described in the above processing method part of the present specification. For example, the processing unit 310 can execute the steps as shown in FIG. 1.

[0165] The storage unit 320 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache memory unit 3202, and can further include a read-only memory (ROM) 3203.

[0166] The storage unit 320 can also include program / utilities 3204 having a set of (at least one) program modules 3205, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include implementation of a network environment.

[0167] The bus 330 can be one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0168] Electronic device 300 can also communicate with one or more external devices 400 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable viewers to interact with electronic device 300, and / or with any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. Network adapter 360 can communicate with other modules of electronic device 300 via bus 330. It should be understood that, although not shown in Figure 3, other hardware and / or software modules can be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0169] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product. This software product can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method described above according to this invention. When the computer program is executed by a data processing device, it enables the computer-readable medium to implement the method described above, i.e., the method shown in FIG1.

[0170] Figure 4 is a schematic diagram of a computer-readable medium provided in an embodiment of this specification.

[0171] A computer program implementing the method shown in Figure 1 can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0172] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0173] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the audience's computing device, partially on the audience's device, as a standalone software package, partially on the audience's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the audience's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0174] In summary, this invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the invention. The invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the invention can be stored on a computer-readable medium or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0175] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0176] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0177] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for analyzing the grasping posture of a robotic arm, characterized in that, The method comprises: obtaining a target object point cloud image; filtering the target object point cloud image to obtain a preprocessed target object point cloud image; inputting the preprocessed target object point cloud image into an object detection model to analyze and obtain a robot arm grasping pose; filtering the robot arm grasping pose to obtain a robot arm target grasping pose.

2. The method of claim 1, wherein, The method of obtaining a target object point cloud image comprises: obtaining a scene RGB image through a robot camera; analyzing the scene RGB image through an object recognition detection model to obtain a target object RGB image; converting the target object RGB image based on an intrinsic matrix of the robot camera to obtain a target object point cloud image.

3. The method of claim 2, wherein, The robot camera-based intrinsic matrix converts the target object RGB image to obtain a target object point cloud image, including: P(x,y)=K -1 ·[x,y,1] T ·D(x,y) Wherein, P(x, y) is a point in the three-dimensional space of the target object point cloud image, K is an intrinsic matrix of the robot camera, [x, y, 1] T is a normalized pixel coordinate of the target object, and D(x, y) is a depth value of the target object.

4. The method of claim 3, wherein, The method of filtering the target object point cloud image to obtain a preprocessed target object point cloud image comprises: Suppose the size of the voxel is l x w x h, convert the point P i (x i ,y i ,z i ) in the point cloud image of the target object to the voxel grid coordinate V i , including: replacing each voxel grid coordinate with an average position to reduce the total number of points in the target object point cloud image to obtain a preprocessed target object point cloud image.

5. The method of claim 4, wherein, said replacing each of the voxel grid coordinates by the average position P avg comprising: where the average position P avg N is the total number of voxel grid points.

6. The method of claim 5, wherein, The method of inputting the preprocessed target object point cloud image into an object detection model to analyze and obtain a robot arm grasping pose comprises: extracting features of the preprocessed target object point cloud image through an encoder to obtain an encoded feature matrix; inputting the encoded feature matrix into a proximity network submodel to output a grasping score; generating K grasping clusters based on the encoded feature matrix and determining operation parameters of each grasping cluster through an operation network submodel to obtain a robot arm grasping pose.

7. The method of claim 6, wherein, The method of filtering the robot arm grasping pose to obtain a robot arm target grasping pose comprises: filtering robot arm grasping poses with a grasping score lower than a first preset value to obtain robot arm target grasping poses after preliminary filtering; determining a tolerance of a grasping cluster of the robot arm target grasping pose after preliminary filtering through a tolerance network submodel, comprising: τ k = ToleranceNet(C k ) where τ k is the tolerance of the cluster of grabs, C k is the cluster of grabs; secondarily filtering the robot arm target grasping pose after preliminary filtering to obtain a robot arm target grasping pose, comprising: wherein G is a set of robot arm target grasping poses, and each robot arm target grasping pose comprises a six-degree-of-freedom grasping pose of (x, y, z, rx, rp, py). G = Filter({θ1,τ1}, {θ2,τ2},…,{θ k ,τ k}) The method comprises:

8. A robot arm grasp pose analysis apparatus, characterized by, an acquisition module configured to obtain a target object point cloud image; a filtering module configured to filter the target object point cloud image to obtain a preprocessed target object point cloud image; a detection module configured to input the preprocessed target object point cloud image into an object detection model to analyze and obtain a robot arm grasping pose; a filtering module configured to filter the robot arm grasping pose to obtain a robot arm target grasping pose. The electronic device comprises:

9. An electronic device, comprising: a processor; and a memory storing computer-executable instructions that, when executed, cause the processor to perform the method of any one of claims 1-7. The computer-readable storage medium stores one or more programs that, when executed by a processor, implement the method of any one of claims 1-7.

10. A computer readable storage medium, wherein, ​

Images