Robot grabbing pose generation method, system and equipment and storage medium
By constructing a 3D model of the object, a two-stream network, and dynamic characteristic evaluation, combined with rule base optimization, the problem of inaccurate robot gripping point recognition was solved, achieving a gripping task with high reliability and high success rate.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NINGBO FOTILE KITCHEN WARE CO LTD
- Filing Date
- 2026-01-08
- Publication Date
- 2026-04-10
AI Technical Summary
In existing technologies, robots have difficulty accurately identifying gripping points when the object has an irregular shape or a limited field of vision, and the surface features of the object are easily affected by the environment, which leads to a decrease in recognition accuracy.
A deep neural network is used to construct a 3D model of the object. A two-stream network is used to generate an initial set of candidate gripping poses. The stability risk is evaluated by combining the object's physical property database and the gripper's dynamic model. The gripping scheme is optimized by a hierarchical gripping rule base and the Rete algorithm to ensure the accuracy and stability of the gripping pose.
It improves the accuracy and robustness of grip point recognition, enhances the success rate and adaptability of gripping tasks, and ensures high reliability of gripping pose at the geometric, physical, and semantic levels.
Smart Images
Figure 1
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent robot technology, specifically to a method, system, device, and storage medium for generating robot grasping poses. Background Technology
[0002] There are two main methods for robot gripping point recognition technology: one is to identify the type of object to be grasped through image information and match a preset gripping strategy based on the object's shape and geometric features, such as selecting the handle when grasping a cup and selecting the edge when grasping a bowl; the other is to use an end-to-end deep neural network, directly inputting image and depth information into the network to automatically output the gripping point position.
[0003] Existing gripping point recognition technologies suffer from several problems: when the object has an irregular shape or the robot's field of vision is limited, it is difficult to accurately identify the gripping point; and the material, texture, and color of the object's surface are easily affected by the environment, reducing recognition accuracy. Summary of the Invention
[0004] To address the problems in existing technologies where it is difficult to accurately identify gripping points when the object is irregularly shaped or the robot's field of vision is limited, and where the recognition accuracy is reduced due to the susceptibility of the object's surface material, texture, and color to environmental influences.
[0005] This application provides a method for generating robot grasping pose, the method comprising: The preset angle image information of the object is input into a deep neural network to construct a three-dimensional model of the object; The three-dimensional model is processed using a two-stream network to generate an initial gripping pose candidate set, which includes gripping point coordinates, pose quaternions, and gripping type confidence scores. Based on a pre-built database of object physical properties and a gripper dynamics model, the object's inertia tensor is calculated and the stability risk during the gripping process is predicted. The initial gripping pose candidate set is matched in real time using a hierarchical gripping rule base and the Rete algorithm. The gripping schemes that do not conform to prior knowledge are weighted and decayed to obtain the sorted initial gripping pose candidate set. The final grip pose is determined based on the sorted initial grip pose candidate set, and the grip task is executed.
[0006] Furthermore, the step of inputting the preset angle image information of the object into a deep neural network to construct a three-dimensional model of the object includes: A two-dimensional RGB color image of an object at a preset angle is acquired using an image capture device; Feature extraction is performed on the two-dimensional RGB color image using a convolutional neural network; Feature mapping is performed using fully connected layers or Transformer structures, and an attention mechanism is applied. Generate and optimize 3D shapes using 3D convolutional networks or graph convolutional networks, and output 3D data in point cloud format; Based on the 3D data, output a three-dimensional model of the object.
[0007] Furthermore, the dual-stream network includes an attention mechanism to improve the accuracy of grip point recognition.
[0008] Furthermore, the two streams of the dual-stream network are responsible for processing appearance features and motion features respectively, and performing feature fusion.
[0009] Furthermore, the step of calculating the object's inertia tensor and predicting stability risks during the grasping process based on a pre-built object physical property database and a grasper dynamics model includes: Retrieve physical attribute parameters that match the 3D model from the object's physical attribute database; The retrieved physical property parameters of the object are input into the dynamic model of the grabber; Calculate the inertia tensor of an object; By combining the dynamic model of the gripper and the inertia tensor of the object, the risk of the object sliding during the gripping process is assessed.
[0010] Furthermore, the object physical property database includes the object's material density and coefficient of friction.
[0011] Furthermore, the initial grasping pose candidate set is subjected to real-time rule matching using a hierarchical grasping rule base and the Rete algorithm. Grasping schemes that do not conform to prior knowledge are subject to weight decay to obtain a sorted initial grasping pose candidate set, including: Build a hierarchical rule base for grasping; The Rete algorithm is used to compare the features of each initial grasp pose candidate set with the rules in the hierarchical grasp rule base. Weight reduction is applied to grasping schemes that do not conform to the rules; Output the sorted initial grip pose candidate set.
[0012] This application also provides a robot grasping pose generation system, including: The 3D model building module is used to input the preset angle image information of the object into the deep neural network to build a 3D model of the object. The candidate grasp pose generation module is used to process the three-dimensional model using a two-stream network to generate an initial grasp pose candidate set, which includes grasp point coordinates, pose quaternions, and grasp type confidence scores. The dynamic characteristics assessment module is used to calculate the object's inertia tensor and predict stability risks during the grasping process based on a pre-built object physical property database and grasper dynamics model. The rule matching module is used to perform real-time rule matching on the initial grasp pose candidate set using a hierarchical grasp rule base and the Rete algorithm, and to perform weight decay on grasp schemes that do not conform to prior knowledge, so as to obtain a sorted initial grasp pose candidate set. The execution module is used to determine the final grip pose based on the sorted initial grip pose candidate set and to execute the grip task.
[0013] This application also provides a robot device, including a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the robot grasping pose generation method.
[0014] This application also provides a storage medium storing at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the robot grasping pose generation method.
[0015] Implementing the embodiments of this application has the following beneficial effects: This application effectively solves the problem of inaccurate grasping point recognition caused by limited field of vision or complex object shape in robots by introducing a multi-stage collaborative processing mechanism, including 3D reconstruction technology, a two-stream network with attention mechanism, dynamic characteristic evaluation and knowledge-driven optimization. It improves the accuracy and robustness of grasping point recognition, ensures that the final output grasping pose has high reliability and interpretability at the geometric, physical and semantic levels, and improves the success rate and adaptability of grasping tasks. Attached Figure Description
[0016] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0017] Figure 1 This is a flowchart of the robot grasping pose generation method according to an embodiment of this application; Figure 2 This is a flowchart illustrating the construction of a 3D model according to an embodiment of this application; Figure 3 This is a flowchart of the dynamic characteristic evaluation of an embodiment of this application; Figure 4 This is a flowchart illustrating the rule matching process in an embodiment of this application; Figure 5 This is a hardware structure block diagram of the server for the robot grasping pose generation method according to an embodiment of this application. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0019] For the terms defined below, unless a different definition is given elsewhere in the claims or this specification, these definitions shall apply. All numerical values, whether explicitly indicated or not, are defined herein as being modified by the term "about." The term "about" generally refers to a range of numerical values that a person skilled in the art would consider equivalent to the stated values to produce substantially the same properties, functions, results, etc. A range of numerical values indicated by a low value and a high value is defined as including all numerical values included within that range and all subranges included within that range.
[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.
[0021] The following combination Figures 1-5 This application introduces a robot grasping pose generation method, system, device, and storage medium provided by embodiments of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual execution of the preparation method, it can be executed in the order shown in the embodiments or accompanying drawings or in parallel.
[0022] Figure 1 This is a flowchart of the robot grasping pose generation method according to an embodiment of this application; Figure 2 This is a flowchart illustrating the construction of a 3D model according to an embodiment of this application; Figure 3This is a flowchart of the dynamic characteristic evaluation of an embodiment of this application; Figure 4 This is a flowchart illustrating the rule matching process in an embodiment of this application; Figure 5 This is a hardware structure block diagram of the server for the robot grasping pose generation method according to an embodiment of this application.
[0023] like Figure 1 As shown in the figure, this application provides a method for generating robot grasping pose, the method including: S100 inputs the preset angle image information of the object into the deep neural network to construct a three-dimensional model of the object.
[0024] Further, in step S100, inputting the preset angle image information of the object into a deep neural network to construct a three-dimensional model of the object includes: S101 acquires a two-dimensional RGB color image of an object at a preset angle using an image capture device.
[0025] S102, use a convolutional neural network to extract features from the two-dimensional RGB color image.
[0026] S103 uses fully connected layers or Transformer structures for feature mapping and applies an attention mechanism.
[0027] S104 generates and optimizes 3D shapes through a 3D convolutional network or a graph convolutional network, and outputs 3D data in point cloud format.
[0028] S105, Based on the 3D data, output a three-dimensional model of the object.
[0029] Specifically, a two-dimensional RGB color image of the object can be acquired from a preset angle using an image capture device, such as an RGB camera or a depth camera. The image capture device can be mounted on the robot's end effector or fixed above the workspace to obtain a clear and complete image of the object. The acquired two-dimensional RGB color image is then input into a convolutional neural network (CNN) for feature extraction. The CNN network may include multiple convolutional layers, pooling layers, and activation function layers to extract spatial features such as edges, textures, and shapes from the image, forming a high-dimensional feature map. The extracted feature map is then input into a feature mapping network, which can employ fully connected layers (FC) or a Transformer structure to achieve spatial transformation and semantic enhancement of the features. The mapped features are then input into a 3D decoding network, which can employ a 3D convolutional network (Conv3D) or a graph convolutional network (GCN), to generate the initial three-dimensional shape of the object. In some possible implementations, the initial 3D shape can also be refined and optimized, such as through denoising, completion, and smoothing, to improve model accuracy. The final output is three-dimensional data in point cloud format. Finally, based on the 3D data in point cloud format, a complete 3D model of the object is constructed for subsequent steps such as gripping point recognition, pose estimation, and stability analysis.
[0030] The above steps enable the automatic construction of a complete 3D model from a single 2D image, effectively solving the problem of difficulty in recognizing grasping points caused by limited field of vision or complex object shapes. Compared with traditional 2D recognition methods, it has stronger spatial perception capabilities and higher recognition accuracy, which is beneficial to improving the robot's grasping success rate in complex environments.
[0031] S200, the three-dimensional model is processed using a two-stream network to generate an initial gripping pose candidate set, which includes gripping point coordinates, pose quaternions, and gripping type confidence scores.
[0032] In this embodiment, the dual-stream network includes an attention mechanism to improve the accuracy of grip point recognition. The two streams of the dual-stream network are responsible for processing appearance features and motion features respectively, and performing feature fusion.
[0033] Specifically, the constructed 3D model (point cloud format) of the object is input into the improved GripNet two-stream network. The improved GripNet network employs a two-stream architecture with an attention mechanism, including: a first stream (appearance stream): responsible for processing the appearance features of the 3D model, extracting geometric information related to grip point recognition, such as edges, corners, and planes on the object's surface. A second stream (motion stream): responsible for processing the dynamic features of the 3D model, extracting motion information related to grip stability, such as the object's center of gravity and inertia tensor. Attention mechanism: An attention mechanism is introduced into the two-stream network to dynamically adjust the network's attention to different features, highlighting the most critical feature regions for grip point recognition, thereby improving the accuracy of grip point recognition. The features extracted from the appearance stream and motion stream are fused to obtain a feature representation that integrates geometric and dynamic information. Feature fusion can be achieved in various ways, such as weighted summation, concatenation, or using a specific fusion layer. The fused feature representation can more comprehensively describe the characteristics of the grip point and environmental information.
[0034] Based on the fused feature representation, the network's output layer predicts the coordinates (x, y, z) of the grasping points. The output layer can be a fully connected layer with the number of neurons corresponding to the number of predicted grasping points. In this way, the network can output the coordinates of multiple grasping points, providing various options for subsequent grasping scheme evaluation. Simultaneously, the two-stream network also predicts pose quaternions (qx, qy, qz, qw) associated with the grasping points to describe the rotational attitude of the grasper relative to the grasping points. The prediction of the pose quaternions can be achieved through an additional output layer dedicated to predicting the four components of the quaternions. Using the pose quaternions, the orientation of the grasper in three-dimensional space can be accurately determined, thus achieving stable grasping.
[0035] In addition to the grip point coordinates and pose quaternions, the two-stream network also outputs a grip type confidence score, ranging from 0 to 1. This score is used to evaluate the reliability of the predicted grip point, helping the system select the optimal gripping scheme from multiple candidate grip points. In one possible implementation, the confidence score can be implemented using a sigmoid activation function, mapping the output value to the interval [0, 1], where a score closer to 1 indicates higher confidence.
[0036] The obtained grip point coordinates, pose quaternions, and confidence scores are integrated into an initial grip pose candidate set. Each candidate set contains multiple sets of grip point information for subsequent dynamic characteristic evaluation and knowledge-driven optimization, providing a variety of possible gripping schemes and offering rich choices for the final gripping decision.
[0037] This application utilizes an improved GripNet two-stream network to process 3D models, obtaining an initial candidate set of gripping poses containing gripping point coordinates, pose quaternions, and confidence scores. This two-stream network structure, combined with an attention mechanism, not only fully utilizes the geometric and dynamic information of the object but also dynamically adjusts the focus on key features, thereby significantly improving the accuracy and reliability of gripping point recognition. The introduction of confidence scores provides an important reference for subsequent gripping scheme evaluation and optimization.
[0038] S300 calculates the object's inertia tensor and predicts stability risks during the grasping process based on a pre-built database of object physical properties and a grasper dynamics model.
[0039] Further, S300, the step of calculating the object's inertia tensor and predicting stability risks during the grasping process based on a pre-built object physical property database and a grasper dynamics model includes: S301, retrieve physical attribute parameters that match the three-dimensional model from the object's physical attribute database; S302, input the retrieved physical property parameters of the object into the dynamic model of the grabber; S303, calculates the inertia tensor of an object; S304, combining the dynamics model of the gripper and the inertia tensor of the object, assesses the risk of the object sliding during the gripping process.
[0040] Specifically, based on the constructed 3D model, the object's type or geometric features are identified, and matching physical property parameters are retrieved from a pre-built object physical property database. These physical property parameters include, but are not limited to, the object's material density, surface friction coefficient, mass distribution, and center of gravity location. This database can be built using experimental measurements, material handbooks, or simulation data, and supports indexing and querying by object category, shape, or material.
[0041] The retrieved physical property parameters of the object are input into the dynamic model of the gripper. This dynamic model describes the mechanical interaction between the gripper and the object, including but not limited to gripping force, contact force, friction, and inertial response. The gripper's dynamic model can be a mathematical model based on rigid body dynamics or a data-driven machine learning model, depending on the specific application scenario.
[0042] Based on the object's mass distribution and three-dimensional geometry, its inertia tensor is calculated. Combining the gripper's dynamic model with the object's inertia tensor, the potential risk of slippage during the gripping process is assessed. Specifically, by analyzing factors such as the contact force distribution, friction magnitude, and inertial torque at the gripping point, it can be determined whether the current gripping scheme possesses sufficient stability. For gripping schemes with a high risk of slippage, screening or optimization can be performed by setting a risk threshold, such as adjusting the gripping posture, increasing the number of contact points, or changing the gripping force.
[0043] Through the above steps, this application achieves the evaluation of the stability of the grasping scheme. Compared with traditional grasping recognition methods that rely solely on geometric features, this method introduces the physical properties and dynamic characteristics of the object, enabling more accurate prediction of potential risks during the grasping process. This effectively improves the success rate and safety of grasping tasks, and is applicable to robot grasping tasks in various complex scenarios.
[0044] S400, using a hierarchical gripping rule base and the Rete algorithm, performs real-time rule matching on the initial gripping pose candidate set, and performs weight decay on gripping schemes that do not conform to prior knowledge, to obtain a sorted initial gripping pose candidate set.
[0045] Further, in step S400, the initial grasping pose candidate set is subjected to real-time rule matching using a hierarchical grasping rule base and the Rete algorithm, and the grasping schemes that do not conform to prior knowledge are subject to weight decay to obtain a sorted initial grasping pose candidate set, including: S401, build a hierarchical grasp rule base.
[0046] S402, the Rete algorithm is used to compare the features of each of the initial grip pose candidate sets with the rules in the hierarchical grip rule base.
[0047] S403 applies weight decay to grasping schemes that do not conform to the rules; it reduces their priority or feasibility by weight decay.
[0048] S404, outputs the sorted initial grip pose candidate set.
[0049] Specifically, based on actual application scenarios and prior knowledge, a hierarchical grasping rule base is constructed, including multiple levels of grasping rules, with each rule corresponding to different grasping conditions and constraints.
[0050] In some possible implementations, rules may include: the grip point should be located in a rigid area of the object; the grip posture should avoid excessive deviation from the normal direction of the object's surface; the grip type should match the object's purpose, such as prioritizing the handle when gripping a cup; grip stability should meet the minimum contact area requirement; and the grip should avoid obstructing key functional areas of the object, such as the bottle opening or buttons. The rule base adopts a hierarchical structure for easy and rapid retrieval and matching, and can be stored and managed in the form of logical expressions, decision trees, or rule engines.
[0051] The Rete algorithm is used to perform real-time rule matching on each candidate solution in the initial grasp pose candidate set. Features of each candidate grasp pose (such as coordinates, attitude, confidence level, object attributes, etc.) are used as fact input; rules from a hierarchical grasp rule base are used as patterns; the Rete algorithm quickly identifies candidate solutions that satisfy or do not satisfy the rules by constructing a matching network; candidate solutions that satisfy the rules are marked as "compliant"; candidate solutions that do not satisfy the rules are marked as "non-compliant," and the type and degree of rule violation are recorded.
[0052] Grasp solutions marked as "non-compliant" undergo weight decay. Weight decay methods include: reducing their grasp type confidence score; assigning them lower priority during ranking; and directly eliminating candidate solutions that seriously violate key rules.
[0053] In some possible implementations, the degree of weight decay can be dynamically adjusted based on the type and number of rule violations. For example, candidate schemes that violate stability rules will be assigned a higher decay weight.
[0054] Based on the results after weight decay, the initial set of candidate grip poses is reordered. Ranking criteria may include: higher grip type confidence scores; more rules met; and smaller weight decay.
[0055] The knowledge-driven optimization steps in this application introduce a hierarchical grasping rule base and the Rete algorithm to evaluate and filter grasping schemes. This effectively eliminates grasping schemes that do not meet the actual application requirements, improving the success rate and security of grasping tasks. Furthermore, the scalability of the rule base allows the system to adapt to different application scenarios, exhibiting good versatility and practicality.
[0056] S500, determine the final gripping pose based on the sorted initial gripping pose candidate set, and execute the gripping task.
[0057] Specifically, based on the sorted initial candidate grasping poses, the highest priority grasping pose is selected as the final grasping pose. Based on the final determined grasping pose, robot control commands are generated. Upon receiving the grasping commands, the robot controller controls the robotic arm and end effector to move along the planned path to the target grasping pose and execute the grasping action.
[0058] In some possible implementations, robot gripping commands include: gripping point coordinates; target posture of the robot end effector, i.e., posture quaternion; gripping type, such as clamping, adsorption, hooking, etc.; gripping force or contact force control parameters; gripping path planning information, such as motion trajectory, speed, acceleration, etc.
[0059] In some possible implementations, during the robot's grasping action, force and visual information can be fed back in real time for closed-loop control, ensuring the stability and safety of the grasp. If the grasp fails or an anomaly is detected (such as object slippage or excessive posture deviation), the system can trigger a retry mechanism to reselect a suboptimal grasping posture or report the anomaly for manual intervention.
[0060] The robot grasping pose generation method in this application implements a complete closed-loop control process from grasping pose generation, evaluation, sorting to final execution. Through multi-stage evaluation and sorting, it ensures that the selected grasping pose is feasible at the geometric, physical, and semantic levels; it can dynamically respond to environmental changes and task requirements, supports various object shapes and materials, and is easy to debug and optimize.
[0061] The robot grasping pose generation method of this application effectively solves the problem of inaccurate grasping point recognition caused by limited field of vision or complex object shape by introducing a multi-stage collaborative processing mechanism, including 3D reconstruction technology, a two-stream network with attention mechanism, dynamic characteristic evaluation and knowledge-driven optimization. This improves the accuracy and robustness of grasping point recognition, and ensures that the final output grasping pose has high reliability and interpretability at the geometric, physical and semantic levels, thereby improving the success rate and adaptability of grasping tasks.
[0062] This embodiment also provides a robot grasping pose generation system, including: The 3D model building module is used to input the preset angle image information of an object into a deep neural network to build a 3D model of the object.
[0063] The 3D model building module includes: The image acquisition unit is used to acquire a two-dimensional RGB color image of an object at a preset angle through an image capture device.
[0064] The feature extraction unit is used to extract features from the two-dimensional RGB color image using a convolutional neural network.
[0065] The feature mapping unit is used to perform feature mapping using fully connected layers or Transformer structures and to apply attention mechanisms.
[0066] The 3D reconstruction and optimization unit is used to generate and optimize 3D shapes through 3D convolutional networks or graph convolutional networks, and output 3D data in point cloud format.
[0067] The 3D model output unit is used to output a 3D model of the object based on the 3D data.
[0068] The candidate gripping pose generation module is used to process the 3D model using a two-stream network to generate an initial gripping pose candidate set, which includes gripping point coordinates, pose quaternions, and gripping type confidence scores.
[0069] The dynamic characteristics assessment module is used to calculate the object's inertia tensor and predict stability risks during the grasping process based on a pre-built database of object physical properties and a grasper dynamics model.
[0070] The dynamic characteristic evaluation module includes: A physical attribute retrieval unit is used to retrieve physical attribute parameters that match the three-dimensional model from the physical attribute database of the object; The dynamic model input unit is used to input the retrieved physical property parameters of the object into the dynamic model of the grabber; Inertia tensor calculation unit, used to calculate the inertia tensor of an object; The slip risk assessment unit is used to assess the slip risk of an object during the grasping process by combining the dynamic model of the gripper and the inertia tensor of the object.
[0071] The rule matching module is used to perform real-time rule matching on the initial gripping pose candidate set using a hierarchical gripping rule base and the Rete algorithm, and to perform weight decay on gripping schemes that do not conform to prior knowledge, so as to obtain a sorted initial gripping pose candidate set.
[0072] The rule matching module includes: Rule base building unit, used to build a hierarchical grasp rule base; A rule matching unit is used to compare the features of each initial grasp pose candidate set with the rules in the hierarchical grasp rule base using the Rete algorithm; The weight decay unit is used to decay the weight of grasping schemes that do not conform to the rules; The candidate set sorting output unit is used to output the sorted initial grip pose candidate set.
[0073] The execution module is used to determine the final grip pose based on the sorted initial grip pose candidate set and to execute the grip task.
[0074] This application also provides a robot device, including a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set are loaded and executed by the processor to implement the robot grasping pose generation method.
[0075] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.
[0076] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Taking running on a server as an example, Figure 5 This is a hardware structure block diagram of the server for the robot grasping pose generation method provided in this application embodiment. For example... Figure 5As shown, the server 100 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 110 (CPUs 110 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 130 for storing data, and one or more storage media 120 (e.g., one or more mass storage devices) for storing application programs 123 or data 122. The memory 130 and storage media 120 may be temporary or persistent storage. The program stored in the storage media 120 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 110 may be configured to communicate with the storage media 120 and execute the series of instruction operations stored in the storage media 120 on the server 100. Server 100 may also include one or more power supplies 160, one or more wired or wireless network interfaces 150, one or more input / output interfaces 140, and / or one or more operating systems 121, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0077] The input / output interface 140 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 100. In one example, the input / output interface 140 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 140 may be a radio frequency (RF) module used for wireless communication with the Internet.
[0078] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 100 may also include components that are more... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown.
[0079] This application embodiment also provides a storage medium storing at least one instruction or at least one program segment, wherein the at least one instruction or the at least one program segment is loaded and executed by a processor to implement the robot grasping pose generation method.
[0080] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0081] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.
[0082] Obviously, the embodiments described above are merely some of the embodiments in this specification, and not all of them. Based on the embodiments in this specification, those skilled in the art can make other variations or modifications without creative effort, and all such variations should fall within the scope of protection of the embodiments in this specification.
[0083] Other embodiments of the embodiments disclosed herein will readily occur to those skilled in the art upon consideration of the specification and practice of the methods disclosed herein. This specification is intended to cover any variations, uses, or adaptations of the embodiments thereof that follow the general principles of the embodiments thereof and include common knowledge or customary techniques in the art not disclosed in the embodiments thereof. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the embodiments thereof are indicated by the following claims.
[0084] It should be understood that the embodiments described herein are not limited to the precise structures already described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from their scope. The scope of the embodiments described herein is limited only by the appended claims.
Claims
1. A robot grasp pose generation method, characterized by, The method comprises: inputting preset angle image information of an object into a deep neural network to construct a three-dimensional model of the object; processing the three-dimensional model using a double-flow network to generate an initial grasp pose candidate set, the initial grasp pose candidate set including grasp point coordinates, attitude quaternion, and grasp type confidence score; calculating the inertia tensor of the object and predicting the stability risk in the grasping process according to a pre-constructed object physical property database and a gripper dynamics model; performing real-time rule matching on the initial grasp pose candidate set using a hierarchical grasp rule library and a Rete algorithm, and performing weight attenuation on the grasp scheme that does not conform to priori knowledge to obtain a sorted initial grasp pose candidate set; determining a final grasp pose according to the sorted initial grasp pose candidate set and performing a grasping task.
2. The robotic grasp pose generation method of claim 1, wherein, The inputting of the preset angle image information of the object into the deep neural network to construct the three-dimensional model of the object comprises: acquiring a two-dimensional RGB color image of the object at a preset angle through an image capture device; extracting features of the two-dimensional RGB color image using a convolutional neural network; performing feature mapping using a fully connected layer or a Transformer structure and applying an attention mechanism; generating and optimizing a 3D shape through a 3D convolutional network or a graph convolutional network to output 3D data in a point cloud format; outputting the three-dimensional model of the object according to the 3D data.
3. The robotic grasp pose generation method of claim 1, wherein, The double-flow network comprises an attention mechanism to improve the accuracy of grasp point recognition.
4. The robotic grasp pose generation method of claim 3, wherein, The two flows of the double-flow network are respectively responsible for processing appearance features and motion features and performing feature fusion.
5. The robotic grasp pose generation method of claim 1, wherein, The calculation of the inertia tensor of the object and the prediction of the stability risk in the grasping process according to the pre-constructed object physical property database and the gripper dynamics model comprises: retrieving physical property parameters matched with the three-dimensional model from the object physical property database; inputting the retrieved object physical property parameters into the dynamics model of the gripper; calculating the inertia tensor of the object; combining the dynamics model of the gripper and the inertia tensor of the object to evaluate the sliding risk of the object in the grasping process.
6. The robotic grasp pose generation method of claim 5, wherein, The object physical property database comprises the material density and the friction coefficient of the object.
7. The robotic grasp pose generation method of claim 1, wherein, The real-time rule matching on the initial grasp pose candidate set using the hierarchical grasp rule library and the Rete algorithm and the weight attenuation on the grasp scheme that does not conform to priori knowledge to obtain the sorted initial grasp pose candidate set comprise: constructing a hierarchical grasp rule library; comparing the features of each initial grasp pose candidate set with the rules in the hierarchical grasp rule library using a Rete algorithm; performing weight attenuation on the grasp scheme that does not conform to the rules; outputting the sorted initial grasp pose candidate set.
8. A robotic grasp pose generation system, comprising: The method comprises: a three-dimensional model construction module configured to input preset angle image information of an object into a deep neural network to construct a three-dimensional model of the object; a candidate grasp pose generation module configured to process the three-dimensional model using a double-flow network to generate an initial grasp pose candidate set, the initial grasp pose candidate set including grasp point coordinates, attitude quaternion, and grasp type confidence score; A dynamic characteristic evaluation module is configured to calculate an inertia tensor of the object and predict a stability risk in a grabbing process according to a pre-constructed object physical attribute database and a grabber dynamics model; A rule matching module is configured to perform real-time rule matching on the initial candidate set of grabbing poses by using a hierarchical grasping rule library and a Rete algorithm, to perform weight attenuation on a grabbing scheme that does not conform to prior knowledge, and to obtain a sorted initial candidate set of grabbing poses; An execution module is configured to determine a final grabbing pose according to the sorted initial candidate set of grabbing poses and to perform a grabbing task.
9. A robotic device, characterized by A processor and a memory are included, and the memory stores at least one instruction, at least one program, a code set, or an instruction set, which are loaded and executed by the processor to implement the robot grasping pose generation method according to any one of claims 1-7.
10. A storage medium, characterized by The storage medium stores at least one instruction or at least one program, which are loaded and executed by the processor to implement the robot grasping pose generation method according to any one of claims 1-7.