Machine grabbing method based on scene understanding and intelligent grabbing

Abstract

This invention discloses a machine grasping method based on scene understanding and intelligent grasping. It generates a human grasping image from an input RGB image and user prompts using a basic generative model. The generated human grasping image is then transformed into specific robot actions. The robot, based on a grasping control network (GCN), controls its manipulator to perform grasping actions according to the target posture map and stress distribution map, and uses force sensors for real-time stress control. Visual and tactile sensors determine whether the manipulator's grasping is successful, and the weights of the GCN are adaptively adjusted based on the results, achieving adaptive learning and optimization of the model. This method, while completing the core grasping task, solves secondary problems such as strategy reliability, environmental adaptability, and grasping precision, and significantly reduces the deployment cost and cycle time for complex grasping tasks.

Description

Technical Field

[0001] This invention relates to the field of home service robot technology, and in particular to a machine grasping method based on scene understanding and intelligent grasping. Background Technology

[0002] Currently, autonomous identification, grasping, and organization of personal belongings in everyday home environments, where items are randomly placed and cluttered, presents a significant challenge compared to routine cleaning tasks like sweeping and mopping. This is due to the diverse shapes of the items, their unpredictable placement, and the complex spatial structure, placing higher demands on the robot's environmental perception accuracy, motion adaptation flexibility, and force control safety. The deep integration of dexterous manipulation technology and perceptual learning technology is providing crucial support in this direction. Through a grasping framework combining contact stress prediction and a multimodal basic model, it accurately captures the physical attributes of items based on tactile-visual fusion representation. Leveraging zero-sample generalization capabilities, it overcomes the limitations of different household item categories, enabling high-success-rate grasping of cross-category, multi-shaped household items.

[0003] While existing dexterous hand vision and tactile technologies have shown some performance improvements and scene adaptability in underlying technologies such as feature fusion and strategy optimization, their overall systems remain in a state of "passive adaptation," with a tendency to "emphasize execution and neglect prediction," resulting in relatively low overall efficiency. Existing vision technologies lack tactile sensing mechanisms and associated logic, making it impossible to accurately determine the grasping state. Although related underlying tactile technologies possess certain processing capabilities in signal acquisition and force feedback control, the overall system still relies on the passive reception of real tactile signals, failing to construct an active tactile perception and state judgment system. Because a linkage logic between tactile sensing and grasping actions has not been established, and a correlation mechanism of "grasping state - tactile signal - grasping adjustment" has not been formed, the system cannot confirm whether it has correctly grasped the target object through tactile feedback, nor can it perceive key information such as grasping force and fit in real time. This leads to an inability to determine whether the grasp is stable, easily resulting in problems such as grasping slippage, excessively tight / loose grips.

[0004] Existing haptic technologies lack effective visual input and analysis mechanisms to acquire core target information, resulting in deficiencies in localization and recognition capabilities. While the underlying technologies possess some scene adaptability in image feature extraction and pose estimation, the overall system operates at a shallow processing level, failing to utilize RGB-D for coordinate and object-holding posture prediction, or stress feedback simulation. The inability to accurately identify target objects through the visual system, and the difficulty in extracting key information such as the target's spatial location, morphological details, and environmental constraints, leads to a lack of precise basis for grasping and localization, resulting in positioning errors and target recognition failures, directly impacting the effectiveness of grasping. Furthermore, existing haptic technologies lack a collaborative optimization process of "vision-touch-motion" and rely on manual training, leading to high labor costs. Although the underlying technologies possess some collaborative capabilities in multimodal fusion and cross-modal weight allocation, the overall system operates in a "passive execution + manual drive" state, lacking an automated fusion process of "dual-image analysis - simulation prediction - closed-loop feedback," further contributing to excessive labor costs and limiting large-scale application.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a machine crawling method based on scene understanding and intelligent crawling to solve the aforementioned technical problems in the prior art. The method of this invention, while completing the core crawling task, solves secondary problems such as strategy reliability, environmental adaptability, and crawling precision, and significantly reduces the deployment cost and cycle time of complex crawling tasks.

[0007] The objective of this invention is achieved through the following technical solution: A machine-based grasping method based on scene understanding and intelligent grasping, the method comprising: Step 1: Generate a human-captured image by combining the input RGB image with user prompts using a basic generative model; Step 2: Convert the generated human-grabbing images into specific robot actions; Step 3: Based on the grasping control network GCN, the robot controls the robot's manipulator to perform grasping actions according to the target posture map and stress distribution map, and uses force sensors for real-time stress control. Step 4: Use visual and tactile sensors to determine whether the robotic arm has successfully grasped the object, and adjust the weights of the grasping control network GCN adaptively based on the results to achieve adaptive learning and optimization of the model.

[0008] Compared with existing technologies, the above method, while completing the core crawling task, solves secondary problems such as strategy reliability, environmental adaptability and crawling precision, and significantly reduces the deployment cost and cycle of complex crawling tasks, and improves the overall generalization and operational accuracy of the system in complex home scenarios. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the machine crawling method based on scene understanding and intelligent crawling provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them, and do not constitute a limitation on the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0012] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0013] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0014] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0015] The technical solution provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0016] like Figure 1 The diagram shown is a flowchart of a machine grasping method based on scene understanding and intelligent grasping provided in an embodiment of the present invention. The method includes: Step 1: Generate a human-captured image by combining the input RGB image with user prompts using a basic generative model; In this step, the input RGB image and RGBD data, including point cloud and depth information, are analyzed by the basic generative model to output the spatial location, morphological features and environmental information (such as scene layout, lighting conditions, obstacle distribution, etc.) of the target object to be captured. The output, along with user prompts (supporting various formats such as language commands, grab point / region masks, and demo images), is input into the base generative model, which then generates a human-grabbed image. This includes the spatial pose of the human wrist and the morphological parameters of the hand shape, specifically: Generated human-grabbing images Represented as: ; in, ; The input is an RGB image; The input is RGBD data, including point cloud and depth information. ; Provide user prompts (supports language commands) , capture point / region mask Demonstration images (and other forms); The parameters for generating the model are based on this. Generate models based on this; The spatial pose of the human wrist, i.e., the 6D pose; These are the morphological parameters of the hand shape.

[0017] In a specific implementation, the basic generative model can be GPT-Image or Qwen-Image, etc.

[0018] Step 2: Convert the generated human-grabbing images into specific robot actions; In this step, the human-demonstrated grasping position and posture are mapped to the robot's motion space based on the human grasping image, generating the corresponding robot grasping trajectory and translating it into the robot's specific actions. ,in The 6D pose of the robotic arm's end effector; The joint angle of the robotic arm is defined as follows: (1) Hand-object reconstruction: Based on the spatial position, morphological features and environmental information of the target object to be grasped output by the basic generative model, a 3D representation of the physically interactive hand and object is reconstructed. The joint posture of the human hand is obtained by using the hand pose reconstruction model HaMeR (HandMesh Recovery). 6D wrist pose Specifically: HaMeR, a hand pose reconstruction model, is used to extract images from human grasping objects. The reconstructed hand posture is represented as follows: ; The joint pose of the human hand (including 3D coordinates of 21 joints). The spatial pose of the human wrist, i.e., the 6D pose; These are the parameters of the HaMeR model; in, ; The parameters for hand posture include 15 joints, each with 3 degrees of freedom; For hand shape parameters; By combining the Hyper3D 3D object mesh generation model with the basic generative model to analyze the morphological features of the target object to be grasped, a 3D mesh model is generated from the original RGBD data. Furthermore, based on the six-degree-of-freedom pose optimization estimation method Any6D, the object scale and 6D pose are optimized and estimated. Specifically: Hyper3D model generates a 3D mesh model. Represented as: ; in, , represents the 3D mesh model of the target object to be grabbed. For the set of grid vertices, For the set of grid edges, A collection of mesh faces; The input is RGBD data, including point clouds. and depth information ; These are the parameters of the Hyper3D model; It represents the prior features (or morphological constraints) of an object's shape, providing prior knowledge such as the object's category and geometric shape (e.g., "cylinder" or "cuboid"). By optimizing the depth axis translation component to align hand-object interaction consistency, the spatial pose of the human wrist is ultimately determined. Transform to the object coordinate system, as shown below: ; in, The parameters represent the 6 degrees of freedom of the human wrist in the object coordinate system, indicating the position and orientation of the human wrist in the target object's own coordinate system (center); The inverse matrix representing the pose of an object in the generated image is the pose (position and rotation) of the object in the camera coordinate system estimated by an image recognition algorithm (such as MegaPose). This represents the human wrist pose in the camera coordinate system. This is raw data estimated directly from the image using the HaMeR model, representing the position and orientation of the human wrist as seen by the camera; (2) Dexterous redirection: redirecting the reconstructed human hand pose to the robotic hand pose. A two-step optimization was adopted: The first step is to replicate the 6-DOF parameters of a human wrist. The parameters of structurally similar joints in the kinematic tree are used as initial values; among them, the parameters of structurally similar joints in the kinematic tree are the angle parameters of each joint of the human hand output by the MANO model, which describe the finger flexion and extension and the palm posture. The second step is to align the fingertips and minimize the error. The objective function is: ; in, The robot's pose in object coordinate system, including 6D pose. and joint angle ; Let K be the position of the k-th fingertip of the robotic arm; The position of the kth fingertip in a human hand; The square of the L2 norm is used to calculate the position error; Optimize the robot's posture using the objective function. This ensures that the movements conform to the shape and characteristics of the target object to be grasped; (3) Action conversion: Based on the environmental information of the target object to be grasped, the problem of misalignment between the generated image and the real scene is solved, and the robot arm's posture is converted. Mapped to the robot coordinate system, the 6D pose of the object in the real scene is estimated by combining the FoundationPose basic pose estimation model with the spatial position of the target object to be grasped. The grasping posture in the object coordinate system is transformed to the camera coordinate system; then, through hand-eye calibration, it is transformed to the robot coordinate system, ultimately obtaining the robot's specific executable actions, represented as follows: ; in, The executable manipulator movements in the robot coordinate system; The transformation matrix from the camera coordinate system to the robot coordinate system is obtained through hand-eye calibration; This is the transformation matrix from the object coordinate system to the camera coordinate system, based on the 6D pose of the object in the real scene. get; The pose of the robot arm in the object coordinate system.

[0019] Step 3: Based on the Grasp Control Network (GCN), the robot controls its manipulator to perform grasping actions according to the target posture map and stress distribution map, and uses force sensors for real-time stress control. In this step, a real-time color image of the target object to be captured, namely SceneRGB, and a depth image, namely Scene Depth, are first acquired through a depth camera. The acquired Scene RGB and Scene Depth data are input into a point cloud or voxel-based object perception network to generate a 3D mesh model of the target object to be grasped. Determine the three-dimensional position and three-dimensional orientation of the target object relative to the robot coordinate system. That is, a six-degree-of-freedom pose (6D Pose). The Scene RGB is input into the neural network model to estimate the MANO Pose parameter of the human hand when performing the grasping action. This parameter describes the rotation angle of each joint of the hand and determines the grasping posture of the hand (such as finger bending and palm opening and closing). The MANO Pose includes the angle and shape parameters of the hand joints. The obtained 3D mesh model of the target object to be grabbed The six-degree-of-freedom pose (6D Pose) and the human hand pose parameter (MANO Pose, Model of Anthropomorphic Hand) are input into the grasping control network (GCN). The GCN outputs key instructions to guide the robotic arm to perform grasping, including the target pose diagram and stress distribution diagram of the robotic arm. Among them, the target posture diagram of the robot arm includes the joint angle vectors or end-effector pose and joint configuration of the robot's end effector; The hand stress distribution map includes a spatial distribution map of the contact pressure or normal stress on the contact surface between the manipulator and the object when performing the corresponding posture.

[0020] In its specific implementation, the grasping control network GCN is a deep learning model with multimodal input and multi-task output, used to convert geometric perception results into grasping instructions and contact stress predictions that can be executed by the robot.

[0021] Table 1 below shows the input data for the capture control network (GCN): Table 1

[0022] GCN employs the following structure to achieve efficient feature fusion and multi-task prediction: 1. Feature Extractor: Extracts high-dimensional geometric features of mesh / point cloud data using PointNet++ or GCN. ; The MANO Pose parameters are input to a fully connected layer (FC Layer) to extract hand pose features. ; 2. Feature Fusion Module: Employs an attention mechanism or a cross-modal Transformer structure to fuse features... and Deep fusion is performed to generate a unified feature vector for data extraction. , is represented as: ; 3. Multi-Head Predictor: Captures feature vectors The input is fed into two separate prediction heads, including: The Pose Head predicts joint space commands for the robot's hand; the Stress / Force Head predicts stress distribution in the gripping contact area.

[0023] Table 2 below shows the key instructions output by GCN to guide the robot in performing grasping: Table 2

[0024] The design of GCN enables the transformation of complex object shapes and humanoid grasping intentions into specific kinematic commands for the robot, while predicting the contact force distribution required during grasping, providing prior guidance for control strategies based on impedance or admittance.

[0025] Step 4: Use visual and tactile sensors to determine whether the robotic arm has successfully grasped the object, and adjust the weights of the grasping control network GCN adaptively based on the results to achieve adaptive learning and optimization of the model.

[0026] In this step, if the robotic arm successfully grasps the object, the weights of the current grasping control network (GCN) are retained. If the robotic arm fails to grasp, the input data and failure result of this failed grasp are used as negative samples to adjust the weights of the grasping control network GCN, so as to achieve adaptive learning and optimization of the model. The binary classification loss with negative sample weights is expressed as: ; in, Indicates the first The label for each sample is 1, indicating successful crawling, and 0, indicating failed crawling. Indicates the first GCN output features of each sample; Indicates the classifier weights and biases; Represents the weight coefficient of the negative sample; the superscript T indicates the transpose sign; denoted as Sigmoid activation function; N represents the total number of training samples.

[0027] In its specific implementation, the method further includes: By designing a force-aware adaptive strategy, the problem of unstable grasping or object damage caused by open-loop execution is solved, specifically including: The target grasping force is predicted by combining the basic generative model with the analysis results of object shape and material. The force feedback of the robotic arm is measured in real time by force sensors (such as strain gauge sensors), wherein: The basic generative model obtains the pre-grabbing pose by analyzing the object's size and shape. (Fingertips offset 5cm outward along the surface normal, initialization without collision) and squeezing grip posture ( (The fingertips are offset inward by 1cm to improve contact stability). Then, force-constrained position control is implemented through a PD controller to ensure stable and safe grasping: the robot moves from the pre-grasping posture to the squeezing grasping posture. When the force feedback measured by the force sensor reaches the target grasping force threshold predicted by the basic generative model, the current position of the robot is locked to avoid excessive squeezing. The control output is defined as: ; ; in, The output of the controller is the control signal that drives the robot arm to move. For PD controller; The target joint position of the robotic arm; Let t be the joint position of the robotic arm; To compress the joint positions corresponding to the grasping posture; The force feedback measured by the force sensor at time t; The target gripping force threshold is predicted by the generative model based on this.

[0028] The significant advantage of the above method is that the high success rate of human demonstration operations has clear action semantics, and can be directly converted into the grasping strategy of the robotic arm without additional human training and data annotation, which greatly reduces the deployment cost and cycle of complex grasping tasks.

[0029] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0030] In summary, the method described in this embodiment of the invention, while completing the core grasping task, solves secondary problems such as strategy reliability, environmental adaptability, and grasping precision. Specifically, by introducing a simulation verification step, virtual testing is performed before the action is executed, which effectively improves the first-time success rate of the strategy in the real environment. By integrating an online learning mechanism, the strategy is continuously fine-tuned using a forward dynamics model and real-time data, enhancing its adaptability to unknown objects and dynamic environments.

[0031] Furthermore, this application utilizes semantic force constraint hierarchical control to achieve precise adaptive adjustment of various gripping forces, ranging from light force to grasping fragile items to heavy force to grasp heavy objects. These optimizations collectively enhance the overall generalization and operational accuracy of the system in complex home scenarios.

[0032] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A machine grasping method based on scene understanding and intelligent grasping, characterized in that, The method includes: Step 1: Generate a human-captured image by combining the input RGB image with user prompts using a basic generative model; Step 2: Convert the generated human-grabbing images into specific robot actions; Step 3: Based on the grasping control network GCN, the robot controls the robot's manipulator to perform grasping actions according to the target posture map and stress distribution map, and uses force sensors for real-time stress control. Step 4: Use visual and tactile sensors to determine whether the robotic arm has successfully grasped the object, and adjust the weights of the grasping control network GCN adaptively based on the results to achieve adaptive learning and optimization of the model.

2. The machine grasping method based on scene understanding and intelligent grasping according to claim 1, characterized in that, In step 1, the input RGB image and RGBD data, including point cloud and depth information, are analyzed by the basic generative model to output the spatial location, morphological features and environmental information of the target object to be captured. The output, along with user prompts, is fed into a base generative model, which then generates a human-grabbing image. This includes the spatial pose of the human wrist and the morphological parameters of the hand shape, specifically: Generated human-grabbing images Represented as: ； in, ; The input is an RGB image; The input is RGBD data, including point cloud and depth information. ; Provide prompts to users; The parameters for generating the model are based on this. Generate a model based on this; The spatial pose of the human wrist, i.e., the 6D pose; These are the morphological parameters of the hand shape.

3. The machine grasping method based on scene understanding and intelligent grasping according to claim 1, characterized in that, In step 2, the human-demonstrated grasping position and posture are mapped to the robot's motion space based on the human grasping image, generating the corresponding robot grasping trajectory and converting it into the robot's specific actions. ,in The 6D pose of the robotic arm's end effector; The joint angle of the robotic arm is defined as follows: (1) Hand-object reconstruction: Based on the spatial position, morphological features and environmental information of the target object to be grasped output by the basic generative model, a 3D representation of the hand and object with reasonable physical interaction is reconstructed, and the joint posture of the human hand is obtained by using the hand pose reconstruction model HaMeR. 6D wrist pose Specifically: HaMeR, a hand pose reconstruction model, is used to extract images from human grasping objects. The reconstructed hand posture is represented as follows: ； The joint posture of the human hand; The spatial pose of the human wrist, i.e., the 6D pose; These are the parameters of the HaMeR model; in, ; These are the hand posture parameters; For hand shape parameters; By combining the Hyper3D 3D object mesh generation model with the basic generative model to analyze the morphological features of the target object to be grasped, a 3D mesh model is generated from the original RGBD data. Furthermore, based on the six-degree-of-freedom pose optimization estimation method Any6D, the object scale and 6D pose are optimized and estimated. Specifically: Hyper3D model generates a 3D mesh model. Represented as: ； in, This represents the 3D mesh model of the target object to be grabbed. For the set of grid vertices, For the set of grid edges, A collection of mesh faces; The input is RGBD data, including point clouds. and depth information ; These are the parameters of the Hyper3D model; It represents the prior features of an object's shape, providing prior knowledge of the object's category and geometric form; By optimizing the depth axis translation component to align hand-object interaction consistency, the spatial pose of the human wrist is ultimately determined. Transform to the object coordinate system, as shown below: ； in, The parameters represent the 6 degrees of freedom of the human wrist in the object's coordinate system, indicating the position and orientation of the human wrist in the target object's own coordinate system. The inverse matrix representing the pose of an object in the generated image is the pose of the object in the camera coordinate system estimated by an image recognition algorithm. This represents the human wrist pose in the camera coordinate system. (2) Dexterous redirection: redirecting the reconstructed human hand pose to the robotic hand pose. A two-step optimization was adopted: The first step is to replicate the 6-DOF parameters of a human wrist. The parameters of structurally similar joints in the kinematic tree are used as initial values; among them, the parameters of structurally similar joints in the kinematic tree are the angle parameters of each joint of the human hand output by the MANO model, which describe the finger flexion and extension and the palm posture. The second step is to align the fingertips and minimize the error. The objective function is: ； in, The robot's pose in object coordinate system, including 6D pose. and joint angle ; Let K be the position of the k-th fingertip of the robotic arm; The position of the kth fingertip in a human hand; The square of the L2 norm is used to calculate the position error; Optimize the robot's posture using the objective function. This ensures that the movements conform to the shape and characteristics of the target object to be grasped; (3) Action conversion: Based on the environmental information of the target object to be grasped, the robot arm's posture is converted. Mapped to the robot coordinate system, the 6D pose of the object in the real scene is estimated by combining the basic pose estimation model with the spatial position of the target object to be grasped. The grasping posture in the object coordinate system is transformed to the camera coordinate system; then, through hand-eye calibration, it is transformed to the robot coordinate system, ultimately obtaining the robot's specific executable actions, represented as follows: ； in, The executable manipulator movements in the robot coordinate system; The transformation matrix from the camera coordinate system to the robot coordinate system is obtained through hand-eye calibration; This is the transformation matrix from the object coordinate system to the camera coordinate system, based on the 6D pose of the object in the real scene. get; The pose of the robot arm in the object coordinate system.

4. The machine grasping method based on scene understanding and intelligent grasping according to claim 1, characterized in that, In step 3, the real-time color image of the target object to be captured, namely Scene RGB, and the depth image, namely Scene Depth, are first acquired through the depth camera. The acquired Scene RGB and Scene Depth data are input into a point cloud or voxel-based object perception network to generate a 3D mesh model of the target object to be grasped. Determine the three-dimensional position and three-dimensional orientation of the target object relative to the robot coordinate system. That is, a six-degree-of-freedom pose (6D Pose). The Scene RGB is input into the neural network model to estimate the human hand posture parameter MANOPose when performing the grasping action. This parameter describes the rotation angle of each joint of the hand and determines the grasping posture of the hand. The MANOPose includes the angle and shape parameters of the hand joints. The obtained 3D mesh model of the target object to be grabbed The six-degree-of-freedom posture (6D Pose) and the human hand posture parameters (MANO Pose) are input into the grasping control network (GCN). The grasping control network (GCN) outputs key instructions to guide the robot arm to execute grasping, including the robot arm target posture diagram and the hand stress distribution diagram. The target posture diagram of the robot arm includes the joint angle vectors or end pose and joint configuration of the robot's end effector; The hand stress distribution diagram includes a spatial distribution diagram of the contact pressure or normal stress on the contact surface between the manipulator and the object when performing the corresponding posture.

5. The machine grasping method based on scene understanding and intelligent grasping according to claim 1, characterized in that, In step 4, if the robotic arm successfully grasps the object, the weights of the current grasping control network GCN are retained. If the robotic arm fails to grasp, the input data and failure result of this failed grasp are used as negative samples to adjust the weights of the grasping control network GCN, so as to achieve adaptive learning and optimization of the model. The binary classification loss with negative sample weights is expressed as: ； in, Indicates the first The label for each sample is 1, indicating successful crawling, and 0, indicating failed crawling. Indicates the first GCN output features of each sample; Indicates the classifier weights and biases; Represents the weight coefficient of the negative sample; the superscript T indicates the transpose sign; denoted as Sigmoid activation function; N represents the total number of training samples.

6. The machine grasping method based on scene understanding and intelligent grasping according to claim 1, characterized in that, The method further includes: By designing a force-aware adaptive strategy, the problem of unstable grasping or object damage caused by open-loop execution is solved, specifically including: The target grasping force is predicted by combining the basic generative model with the analysis results of object shape and material. The force feedback of the robotic arm is measured in real time by a force sensor, wherein: The basic generative model obtains the pre-grabbing posture and the squeezing posture by analyzing the size and shape of the object; Then, force-constrained position control is implemented through a PD controller to ensure stable and safe grasping: the robot moves from the pre-grasping posture to the squeezing grasping posture. When the force feedback measured by the force sensor reaches the target grasping force threshold predicted by the basic generative model, the current position of the robot is locked to avoid excessive squeezing. The control output is defined as: ； ； in, The output of the controller is the control signal that drives the robot arm to move. For PD controller; The target joint position of the robotic arm; Let t be the joint position of the robotic arm; To compress the joint positions corresponding to the grasping posture; The force feedback measured by the force sensor at time t; The target gripping force threshold is predicted by the generative model based on this.

Images