A command-driven semantic robot grasping method and system oriented to user-specified tasks

Abstract

The application discloses a command-driven semantic robot grabbing method and system for user-specified tasks, and belongs to the field of robot grabbing. A language condition grabbing architecture is constructed to realize explicit alignment of language instructions and visual features by taking visual attributes as a bridge, and a pixel attention mechanism is introduced to enhance key region feature extraction of small objects. A parameter correlation attention module is designed to model the implicit relationship among grabbing quality, angle and width, thereby improving the response capability of robots to complex semantic instructions and the success rate of grabbing tasks in unstructured environments. The system of the application comprises a mechanical arm and a sensor, a natural language analysis module, a visual positioning and instance segmentation module, a visual-language attribute alignment module and a task execution module, and solves the problems of insufficient small object grabbing precision, lack of explicit semantic attribute modeling and weak correlation of grabbing parameters in the prior art. The response capability of robots to user-specified tasks and the execution success rate in unstructured environments are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of robot grasping, specifically relating to a command-driven semantic robot grasping method and system for user-specified tasks. Background Technology

[0002] In existing technologies, visual cue-based grasping systems (such as two-step detection, single-stage heatmap methods, and multimodal fusion methods) suffer from target ambiguity in multi-object scenes, making it difficult to distinguish semantically similar objects (such as...). Figure 1 As shown on the left), existing methods lack precision in grasping small objects and are deficient in explicitly modeling long-tail attributes such as color and material. Furthermore, their ability to extract key features from low-resolution or small-sized objects in images is limited, affecting the accuracy of grasping pose estimation. While language-conditional robot manipulation technology uses natural language commands to assist task planning, it still faces technical bottlenecks such as the semantic-visual gap, the lack of implicit correlations in grasping parameters, and the real-time-accuracy contradiction. These shortcomings result in low reliability in executing complex command tasks, making it difficult to meet the demands for precise semantic-level grasping control in scenarios such as home services and industrial quality inspection. Summary of the Invention

[0003] This invention addresses the problems existing in the prior art by providing a command-driven semantic robot grasping method and system for user-specified tasks. It can directly address the perception-decision-execution closed loop of the robot operating system and focuses on solving core problems such as target ambiguity in multi-object scenes, insufficient grasping accuracy of small objects, and semantic constraint modeling for complex tasks.

[0004] To address the above technical problems, this invention provides the following technical solution: a command-driven semantic robot grasping method for user-specified tasks, comprising the following steps:

[0005] S1. For the received natural language instructions, generate standardized triples for the object images in the image, and perform instance segmentation of the image through a neural network to obtain a visual feature map. The visual feature map contains the bounding box and mask of the target object.

[0006] S2. A visual-language attribute alignment module is constructed with the visual feature map as input and the visual features corresponding to the language attributes of the objects in the map as output. A pixel attention mechanism is introduced to enhance the response of key regions of small objects. The language attributes and visual features are aligned by cosine similarity to filter semantically matched grasping targets.

[0007] S3. A grasping decision module is constructed using the visual features corresponding to the language attributes of objects as input and the optimized object grasping heatmap as output. A lightweight feature extractor and upsampling layer are used to improve real-time performance, and a parameter correlation attention mechanism is introduced to obtain the implicit relationship of parameters.

[0008] S4. A task execution module is constructed using the object grasping heatmap as input and the grasping trajectory as output. The motion trajectory is solved through inverse kinematics to achieve real-time control of the robot arm.

[0009] Furthermore, in step S1 mentioned above, the image is segmented using the YOLOv8 network.

[0010] Furthermore, in step S2 above, a pixel attention mechanism is introduced to enhance the response of key regions of small objects, as shown in the following formula:

[0011] ,

[0012] In the formula, f' represents the refined feature, and M p It is an attention mask, f img For CNN-encoded features, r is the channel attention feature bias, and f is the channel attention feature bias. refind Features enhanced by pixel attention mechanism, among which , This indicates that the broadcast operation is performed element by element. The channel attention feature weights.

[0013] Furthermore, in step S3 above, a parameter correlation attention mechanism is introduced to obtain the implicit relationship of parameters, as shown in the following equation:

[0014]

[0015] In the formula, G' is the attention map after output, FC() is the fully connected layer, ReLU() is the activation function layer, and LN() is the normalization layer.

[0016] Furthermore, in the aforementioned step S3, the object capture heatmap includes a set of image space capture candidates, containing three-dimensional information of mass Q, angle θ, and width W.

[0017] Furthermore, in step S4 mentioned above, the obstacle avoidance path is generated by solving the motion trajectory using the Movelt! algorithm.

[0018] The present invention also provides a command-driven semantic robot grasping system for user-specified tasks, including: a robotic arm and sensors, a natural language parsing module, a visual localization and instance segmentation module, a visual-language attribute alignment module, and a task execution module;

[0019] The robotic arm and sensors utilize a UR5 robotic arm paired with a Robotiq 2F-140 gripper, and are equipped with a Kinect V2 depth camera to capture RGB-D images.

[0020] The natural language parsing module generates standardized triples for the object images in the image based on the received natural language instructions; the visual localization and instance segmentation module performs instance segmentation on the image through a neural network to obtain a visual feature map, which includes the bounding box and mask of the target object.

[0021] The visual-language attribute alignment module takes a visual feature map as input and the visual features corresponding to the language attributes of objects in the map as output. It introduces a pixel attention mechanism to enhance the response of key regions of small objects and aligns language attributes and visual features through cosine similarity to filter semantically matched grasping targets.

[0022] The task execution module is constructed with the object grasping heatmap as input and the grasping trajectory as output. It solves the motion trajectory through inverse kinematics to achieve real-time control of the robot arm.

[0023] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0024] This invention achieves a significant technological breakthrough by constructing a language-condition-driven semantic crawling architecture: In terms of semantic attribute recognition, the multi-label classification network using a pixel attention mechanism improves the accuracy of visual attribute recognition by 14.7% compared to baseline models (such as SCoNE), increases the accuracy of long-tail attribute classification by 5.4%, and achieves color and material attribute recognition accuracies of 76.5% and 78.6%, respectively.

[0025] At the grasping pose estimation level, the introduced parameter correlation attention module enables the joint prediction accuracy of 98.8% (based on image segmentation) and 98.6% (based on object segmentation) on the Cornell dataset, which is 0.5%-0.9% higher than baseline models such as GR-ConvNet. In real-world scenarios, the success rate of 300 grasping tasks reaches 96.3%.

[0026] By using a language-visual attribute alignment mechanism, the problem of misgrabbing caused by semantic ambiguity in multi-object scenes is solved, and complex instructions such as "orange ceramic spoon" and "dark blue bottle" can be accurately parsed. The system is deployed on the ROS platform, supports real-time interaction between the UR5 robotic arm and the Kinect V2 camera, and has a parameter scale of only 1.625 million, which combines high real-time performance and strong compatibility.

[0027] The above results, verified through scientific experiments and real-world scenario testing, significantly improve the robot's responsiveness and execution reliability for semantic grasping tasks in unstructured environments. Attached Figure Description

[0028] Figure 1 These are comparison images of robots grasping objects.

[0029] Figure 2 This is the overall system architecture diagram.

[0030] Figure 3 This is a schematic diagram of the pixel attention mechanism.

[0031] Figure 4 This is a flowchart of the parameter correlation attention module workflow.

[0032] Figure 5 This is a hardware system deployment diagram.

[0033] Figure 6 This is a schematic diagram of the test results for a multi-attribute interference scenario.

[0034] Figure 7 This is a flowchart of the method of the present invention.

[0035] Figure 8 This is a flowchart of visual attribute recognition.

[0036] Figure 9 This is a flowchart of object attribute recognition.

[0037] Figure 10 This is a flowchart of the specific implementation process. Detailed Implementation

[0038] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0039] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0040] refer to Figure 7 and Figure 10 This invention provides a command-driven semantic robot grasping method for user-specified tasks, comprising the following steps:

[0041] S1. For the received natural language instructions, generate standardized triples for the object images in the image, and perform instance segmentation of the image through a neural network to obtain a visual feature map. The visual feature map contains the bounding box and mask of the target object.

[0042] ChatGPT is used to parse natural language instructions to generate standardized triples (such as "spoon-color-orange"), and the YOLOv8 network is used to perform instance segmentation on RGB-D images to locate the bounding boxes and masks of target objects, thus solving the problem of target confusion in multi-object scenes.

[0043] S2. A visual-language attribute alignment module is constructed using the visual feature map as input and the visual features corresponding to the language attributes of the objects in the map as output. A pixel attention mechanism is introduced to enhance the response of key regions of small objects. The language attributes and visual features are aligned by cosine similarity to filter semantically matched grasping targets.

[0044] Based on the ResNet-50 backbone network, volume visual features are extracted, and a pixel attention mechanism is introduced to enhance the response of key regions of small objects, such as... Figure 3 As shown, the formula is as follows: ( ),and By aligning linguistic attributes and visual features using cosine similarity, semantic matching targets are filtered for crawling. In the formula, f' represents the refined feature, and M... p It is an attention mask, f img For CNN-encoded features, r is the channel attention feature bias, and f is the channel attention feature bias. refined Features enhanced by pixel attention mechanism, among which , This indicates that the broadcast operation is performed element by element. The channel attention feature weights.

[0045] S3. A grasping decision module is constructed using the visual features corresponding to the language attributes of objects as input and the optimized object grasping heatmap as output. A lightweight feature extractor and upsampling layer are used to improve real-time performance, and a parameter correlation attention mechanism is introduced to obtain the implicit relationship of parameters.

[0046] A grasp map is constructed to represent the image space grasp candidate set (containing three-dimensional information of quality Q, angle θ, and width W). A lightweight feature extractor and upsampling layer are used to improve real-time performance (inference time ≤ 14ms). A parameter correlation attention module is designed (formula: [Formula omitted]). Model the implicit relationships between crawling parameters and output an optimized crawling heatmap; such as Figure 4 As shown.

[0047] S4. A task execution module is constructed using the object grasping heatmap as input and the grasping trajectory as output. The motion trajectory is solved through inverse kinematics to achieve real-time control of the robot arm.

[0048] Based on the ROS platform, the UR5 robotic arm and Kinect V2 camera are integrated. Real-time control is achieved through inverse kinematics solution and Movelt! trajectory planning algorithm, with an inference time of ≤14ms.

[0049] The grasping pose estimation module consists of three parts: a lightweight feature extractor, an upsampling layer, and an attention-enhanced grasping generation head. The attention-enhanced grasping generation head improves the accuracy of grasping pose estimation by calculating an attention map centered on the grasping parameters, highlighting grasping-related features, and modeling the implicit relationships between grasping quality, angle, and width. The process is as follows: Figure 8 , Figure 9 As shown.

[0050] like Figure 1 As shown, the traditional visual grasping system on the left suffers from target confusion in multi-object scenes, such as misidentifying the orange remote control with the yellow remote control. This invention accurately locates the target object (orange remote control) through semantic attribute alignment. Specifically, it provides a command-driven semantic robot grasping system oriented towards user-specified tasks. The hardware system deployment diagram of this invention is shown below. Figure 5 As shown, a Kinect V2 vision sensor is deployed above the gripping platform. The system adopts a layered architecture design. The hardware layer includes vision acquisition, feature processing, and pose calculation modules, which communicate with the upper-level control module through ROS middleware. The control core includes three layers: task scheduling, motion planning, and motion control. Specifically: the task management layer is responsible for parsing human-machine interaction instructions and monitoring system status; the motion planning layer generates the robot's motion trajectory (trajPlanner); and the execution control layer implements gripper servo control and gripping force closed-loop.

[0051] Figure 2 This invention demonstrates the data flow of its core modules (input preprocessing, visual-language attribute alignment, grasping decision, and task execution). The system includes: a robotic arm and sensors, a natural language parsing module, a visual localization and instance segmentation module, a visual-language attribute alignment module, and a task execution module.

[0052] Label key algorithm modules (such as ChatGPT, YOLOv8, ResNet-50, and parameter correlation attention module).

[0053] The robotic arm and sensors consist of a UR5 robotic arm paired with a Robotiq 2F-140 gripper and a Kinect V2 depth camera to capture RGB-D images.

[0054] The natural language parsing module generates standardized triples for the object images in the image based on the received natural language instructions.

[0055] The visual localization and instance segmentation module uses a neural network to perform instance segmentation on the image, obtaining visual feature maps. These visual feature maps include the bounding boxes and masks of the target objects. (Reference) Figure 3 This module uses the following steps: visual feature map input → channel adjustment (τ) → pixel attention mask (Mp) → refined feature output (frefined) annotation formula ( ),and ,in , This indicates element-wise broadcasting. Channel adjustment is achieved by applying c-dimensional learnable channel attention to the input features. Multiply.

[0056] The visual-language attribute alignment module takes a visual feature map as input and the visual features corresponding to the language attributes of objects in the map as output. It introduces a pixel attention mechanism to enhance the response of key regions of small objects and aligns language attributes and visual features through cosine similarity to filter semantically matched grasping targets.

[0057] The task execution module is constructed with the object grasping heatmap as input and the grasping trajectory as output. It solves the motion trajectory through inverse kinematics to achieve real-time control of the robot arm. Figure 6 The test results for multi-attribute interference scenarios are presented.

[0058] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A command-driven semantic robot grasping method for user-specified tasks, characterized in that, Includes the following steps: S1. For the received natural language instructions, generate standardized triples for the object images in the image, and perform instance segmentation of the image through a neural network to obtain a visual feature map. The visual feature map contains the bounding box and mask of the target object. S2. A visual-language attribute alignment module is constructed with the visual feature map as input and the visual features corresponding to the language attributes of the objects in the map as output. A pixel attention mechanism is introduced to enhance the response of key regions of small objects. The language attributes and visual features are aligned by cosine similarity to filter semantically matched grasping targets. Among them, a pixel attention mechanism is introduced to enhance the response of key regions of small objects, as shown in the following formula: ， ， In the formula, f' represents the refined feature, and M p It is an attention mask, f img For CNN-encoded features, r is the channel attention feature bias, and f is the channel attention feature bias. refined Features enhanced by pixel attention mechanism, among which , This indicates that the broadcast operation is performed element by element. For channel attention feature weights; S3. A grasping decision module is constructed using the visual features corresponding to the language attributes of objects as input and the optimized object grasping heatmap as output. A lightweight feature extractor and upsampling layer are used to improve real-time performance, and a parameter correlation attention mechanism is introduced to obtain the implicit relationship of parameters. Introducing a parameter correlation attention mechanism to obtain the implicit relationships between parameters, as shown in the following equation: ， In the formula, G' is the attention map after output, FC() is the fully connected layer, ReLU() is the activation function layer, and LN() is the normalization layer; S4. A task execution module is constructed using the object grasping heatmap as input and the grasping trajectory as output. The motion trajectory is solved through inverse kinematics to achieve real-time control of the robot arm.

2. The command-driven semantic robot grasping method for user-specified tasks according to claim 1, characterized in that, In step S1, the image is segmented using the YOLOv8 network.

3. The command-driven semantic robot grasping method for user-specified tasks according to claim 1, characterized in that, In step S3, the object capture heatmap contains a set of image space capture candidates, including three-dimensional information of mass Q, angle θ, and width W.

4. The command-driven semantic robot grasping method for user-specified tasks according to claim 1, characterized in that, In step S4, the motion trajectory is solved using the Movelt! algorithm to generate an obstacle avoidance path.

5. A command-driven semantic robot grasping system for user-specified tasks, applied to the command-driven semantic robot grasping method for user-specified tasks as described in claim 1, characterized in that, include: Robotic arm and sensors, natural language parsing module, visual localization and instance segmentation module, visual-language attribute alignment module, task execution module; The robotic arm and sensors utilize a UR5 robotic arm paired with a Robotiq 2F-140 gripper, and are equipped with a Kinect V2 depth camera to capture RGB-D images. The natural language parsing module generates standardized triples for the object images in the image based on the received natural language instructions; the visual localization and instance segmentation module performs instance segmentation on the image through a neural network to obtain a visual feature map, which includes the bounding box and mask of the target object. The visual-language attribute alignment module takes a visual feature map as input and the visual features corresponding to the language attributes of objects in the map as output. It introduces a pixel attention mechanism to enhance the response of key regions of small objects and aligns language attributes and visual features through cosine similarity to filter semantically matched grasping targets. The task execution module is constructed with the object grasping heatmap as input and the grasping trajectory as output. It solves the motion trajectory through inverse kinematics to achieve real-time control of the robot arm.

Images