Generalization of language instruction robot grabbing method in cluttered scene

Abstract

The application discloses a language instruction grabbing method with generalization ability in a cluttered scene, which comprises three core steps of information logic processing, self-correction visual positioning and target three-dimensional grabbing posture filtering. Firstly, key information extraction and position relationship reasoning of the language instruction are completed by a large model, then self-correction screening of candidate objects is realized by combining a visual positioning model and a large model visual question and answer, and finally accurate filtering and optimal selection of the grabbing posture are completed based on object 3D point cloud. The method constructs a closed-loop language-visual-grabbing integrated system, can accurately understand complex spatial language instructions, realizes robust and generalized robot grabbing in a cluttered scene, and effectively improves the response accuracy of the robot to natural language instructions and the execution efficiency of the grabbing task.

Description

Technical Field

[0001] This invention relates to the field of robot grasping and control technology, specifically to a language command grasping method with generalization capabilities that is based on the fusion of a large model and a visual grasping model and is applicable to cluttered scenarios. Background Technology

[0002] With the development of service robot technology, the understanding and execution of natural language commands by robots has become a core research direction. Especially in cluttered environments such as homes and offices, accurate grasping based on language commands is a key function of service robots. Existing technologies for language command grasping mostly employ open-loop visual localization and grasping posture generation processes, which have the following problems: First, they lack the ability to analyze complex spatial language commands, failing to accurately extract the target object, reference object, and their spatial relationship; second, visual localization models are easily interfered with by similar objects in cluttered environments, resulting in large detection errors and a lack of self-correction capabilities; third, the grasping posture generation does not incorporate precise filtering based on the object's three-dimensional features, and some high-scoring postures cannot be actually executed due to spatial position or angle issues, leading to low grasping success rates and poor generalization ability.

[0003] To address the aforementioned technical challenges, this invention integrates the advantages of large-scale natural language understanding, visual question answering capabilities, professional visual positioning, and crawling networks to construct a closed-loop integrated language-vision-cracking system. This system enables accurate parsing and robust crawling of complex language instructions in cluttered indoor environments, effectively improving the generalization ability and execution efficiency of crawling tasks. Summary of the Invention

[0004] The purpose of this invention is to provide a generalized language command robot grasping method for cluttered scenes. It can solve the problems of low language command parsing accuracy, lack of self-correction in visual positioning, and inaccurate grasping posture selection in the prior art. It is a method that enables robots to respond efficiently and accurately to natural language grasping commands in cluttered scenes.

[0005] The technical solution of this invention: A method for robot grasping generalized language commands in cluttered scenes, characterized by comprising the following steps: (1) Load the Doubao-1.5-pro large model, the LangSAM pre-trained visual localization model and the GraspNet pre-trained grasping network, and perform initialization settings so that the robot grasping system is in a state of waiting to receive external scene images and language grasping instructions. The robot grasping system consists of an information logic processing module, a self-calibrating visual positioning module, and a target 3D grasping and filtering module. These three modules are cascaded to form a closed-loop system for generalized language command robot grasping in chaotic scenarios. Each module contains dedicated functional subunits. Modules communicate via standardized data interfaces for unidirectional signal transmission and closed-loop verification. The output of the information logic processing module is connected to the input of the self-calibrating visual positioning module, and the output of the self-calibrating visual positioning module is connected to the input of the target 3D grasping and filtering module. The three modules form a closed-loop data flow according to the logic of "language understanding → visual positioning → grasping and filtering." The output of the previous module serves as the sole input for the next module, collaboratively completing the entire process of converting language commands into robot grasping actions. The specific structure and connection relationships are as follows: The information logic processing module has a built-in language instruction parsing subunit, which is the core of the system's language understanding. It is used to extract key information of language instructions and reason about spatial position relationships, and to transform fuzzy natural language into quantifiable machine-executable logic. Its input signals are external natural language capture instructions and cluttered scene RGB-D images, and its output signals are structured target object information, reference object information, spatial position relationship parameters and geometric feature parameters.

[0006] The self-correcting visual localization module is used to complete preliminary object detection, semantic secondary verification, score weighted fusion, and candidate selection. It achieves self-correcting and accurate localization of the target object through a re-detection mechanism. Specifically, it includes a visual detection subunit, a semantic verification subunit, and a score fusion subunit. The visual detection subunit receives structured object information and spatial position parameters output by the information logic processing module at its input end. Its output end connects to the input end of the semantic verification subunit, outputting the bounding box, segmentation mask, and normalized detection confidence (Logits) of the candidate target object. The semantic verification subunit receives candidate images with cropped bounding boxes, the target object name, reference object name, and spatial relationship description output by the information logic processing module at its input end. Its output end connects to the input end of the score fusion subunit, outputting a discrete verification score (Score) for semantic consistency matching. The score fusion subunit receives the normalized detection confidence (Logits) output by the visual detection subunit at its input end. The semantic verification score Score output by the semantic verification subunit is connected to a signal that outputs a weighted and fused comprehensive confidence score Final_Score. The output signal consists of the target object bounding box, segmentation mask, and comprehensive confidence score output by the score fusion subunit.

[0007] The target 3D grasping and filtering module is used to generate candidate grasping poses, filter spatial validity, filter the executability of the robotic arm, and select the optimal pose. It outputs grasping commands that can be directly sent to the robot control system. Specifically, it consists of a grasping pose generation subunit, a spatial position filtering subunit, an execution angle filtering subunit, and an optimal pose selection subunit. The target 3D grasping and filtering module receives the target object segmentation mask output by the self-calibrating visual positioning module and the scene 3D point cloud data generated from the scene RGB-D image. Its output signal is the optimal grasping pose parameters that the robot can execute. The grasping pose generation subunit receives the original scene image at its input end and outputs a set of candidate grasping poses for all objects in the global scene and the stability confidence score corresponding to each grasping pose. The spatial position filtering subunit receives the candidate grasping pose set output by the grasping pose generation subunit, the target object segmentation mask output by the self-calibrating visual positioning module, and the scene RGB-D image. The scene 3D point cloud data generated by the image outputs a set of valid grasping poses whose effective area falls within the effective range of the target object's 3D point cloud. The input of the execution angle filtering subunit receives the set of valid grasping poses output by the spatial position filtering subunit and the system's preset grasping posture angle threshold constraint parameters. The output of the execution angle filtering subunit is a set of compliant grasping poses that meet the reachability and collision-free constraints of the robotic arm. The input of the optimal pose selection subunit receives the set of compliant grasping poses output by the execution angle filtering subunit and the stability confidence scores corresponding to each grasping pose output by the grasping pose generation subunit. The output of the optimal grasping pose position parameters and attitude parameters with the highest confidence after filtering is transmitted to the robot control system.

[0008] The initialization settings in step (1) specifically refer to: (1-1) Load the Doubao-1.5-pro large model, the LangSAM pre-trained visual localization model, and the GraspNet pre-trained crawling network. Configure system-level prompts for the language instruction parsing subunit in the information logic processing module and the semantic verification subunit in the self-correcting visual localization module. The system-level prompts are used to define the task role, output format, and inference range of the Doubao-1.5-pro large model. The Doubao-1.5-pro large model is responsible for language instruction parsing and positional relationship inference. The LangSAM pre-trained visual localization model completes the initial visual localization of the object. The Doubao-1.5-pro large model and the LangSAM pre-trained visual localization model work together to achieve accurate selection of the target object. (1-2) A set of candidate grasping poses in the scene is generated by the GraspNet pre-trained grasping network. These poses are used to obtain executable grasping poses after spatial position filtering and execution angle filtering in the subsequent target 3D grasping filtering module. After 3D filtering, executable grasping postures are obtained. In the target 3D grasping filtering module, the executable grasping poses are obtained after sequential processing by the spatial position filtering subunit and the execution angle filtering subunit. After 3D filtering, executable postures are obtained. Spatial position filtering and execution angle filtering are both predetermined functions of the target 3D grasping filtering module. The subunit composition, function and signal flow of this module have been fully disclosed above.

[0009] (1-3) Preset the geometric feature parameters of candidate target objects And the mapping rules for spatial location relationships, the score fusion weight ratio, and the threshold for the grasping posture angle. Among them, , The coordinates of the center pixel of the bounding box of the candidate target object in the two-dimensional image plane; The distance is the Euclidean distance between the center point of the bounding box of the candidate target object and the reference object; the calculation formula is shown in formula (1); The relative rotation angle of the candidate target object with respect to the reference object is calculated as shown in formula (2). (1) (2) in, , The coordinates of the center point of the target object. , The coordinates are the center coordinates of the reference object; The mapping rules for spatial positional relationships in steps (1-3) specifically refer to: ① When there is no reference object, the two-dimensional image plane coordinate system is used as the judgment benchmark, and the relative position of the candidate target object in the image is determined by the pixel coordinates of the center point of the bounding box; among which, the horizontal pixel coordinates Increasing the pixel value indicates that the candidate object is positioned further to the right in the image, while decreasing it indicates that it is positioned further to the left; vertical pixel coordinates Increasing the value indicates that the candidate target object is lower in the image, while decreasing it indicates that it is higher. For cases where the language command involves "in front" or "behind", the vertical position of the image is used as an approximate criterion. That is, under the preset shooting angle, the candidate target object located at the top of the image is judged to be further back, and the candidate target object located at the bottom of the image is judged to be further forward. ② When there is a reference object, the distance between the candidate target object and the reference object is determined by the Euclidean distance d. The smaller the Euclidean distance d, the closer the candidate target object is to the reference object. ③ Determine the direction between the candidate target object and the reference object by the value of the relative rotation angle r.

[0010] (1-4) Connect the data transmission interface between the information logic processing module, the self-correcting visual positioning module, and the target 3D grasping and filtering module, and verify the smooth operation of the entire process link; wherein, the information logic processing module outputs structured language parsing results based on the Doubao-1.5-pro large model, the self-correcting visual positioning module outputs target object filtering results based on the LangSAM pre-trained visual positioning model and the Doubao-1.5-pro large model, and the target 3D grasping and filtering module outputs a set of candidate grasping poses based on the GraspNet pre-trained grasping network, and completes the grasping pose filtering according to the final target object segmentation mask; then open the scene image input interface and the natural language grasping command input interface, so that the generalized language command robot grasping system in the cluttered scene is in a state of waiting to receive external scene images and language grasping commands; The score fusion weight ratio in step (1) refers to the weighted ratio between the detection confidence Logits output by the LangSAM pre-trained visual localization model and the semantic verification score Score output by the Doubao-1.5-pro large model, which is generally 0.6:0.4.

[0011] The grasping posture angle threshold is not fixed, but is adjusted using a parameter calibration method known in the art. The determination method specifically includes: determining the reachable posture range of the end effector based on the robotic arm configuration, determining the allowable gripping direction range based on the end effector type, determining the safe approach path range based on the table height or the height of the target object's support surface, and determining the collision constraint range based on the obstacle distribution in the scene. On this basis, through grasping experiments with candidate angle thresholds, the grasping success rate, collision rate, and trajectory execution rate corresponding to each candidate angle threshold are statistically analyzed, and the angle threshold with the best overall performance is selected as the grasping posture angle threshold for actual use.

[0012] (2) Information logic analysis and extraction: The captured images of the cluttered scene and the natural language capture instructions are input into the system constructed in step (1). The Information Logic Processing Module (ILPM) converts the ambiguous language instructions into the geometric feature parameters and position information of the target object. In step (2), the Information Logic Processing Module (ILPM) converts the ambiguous language instructions into the geometric feature parameters and position information of the target object. Specifically, it refers to performing two sub-tasks: key information extraction and positional relationship reasoning. Sub-task 1 is used for key information extraction, and task 2 is used for positional relationship reasoning.

[0013] Subtask 1, which involves extracting key information, refers to the process by which the Doubao-1.5-pro large model parses the original natural language crawling instructions under the constraints of the system-level prompt, and extracts the following three core elements: ①Target Object Name: The name of the main object to be captured; ②Reference Object Name: The name of the reference object; ③Relative Position Relationship: A linguistic phrase describing the spatial relationship between a target object and a reference object; If the input language instruction only contains the target object, without a reference object or spatial relationship, fill the ReferenceObject Name and Relative Position Relationship fields with None, and finally output the extraction results in a structured template of "Target Object Name, Reference Object Name, Position Relationship Phrase".

[0014] Subtask 2 involves positional relationship reasoning, namely: ① calling the LangSAM pre-trained visual localization model loaded in step (1) to perform preliminary identification of potential target objects and reference objects in the scene image to be identified, and obtaining the pixel coordinates of the center point of the bounding box of each object in the two-dimensional image plane; ② calculating a four-element list of geometric feature parameters for each candidate target object using the Doubao-1.5-pro large model. ,at this time, , The field is fixed as None, only reserved , Coordinate parameters; ③ According to the mapping rules of the spatial position relationship predetermined in step (1), the geometric feature parameters are transformed into spatial position relationships, and finally the geometric feature parameters and position information of the candidate target object are output.

[0015] (3) Self-Calibrating Visual Localization Processing: The object name and description information output by the Information Logic Processing Module (ILPM) are input into the Self-Calibrating Visual Localization Module (SCGM). A closed-loop visual localization mechanism is adopted. First, preliminary detection is performed based on the object name and description information input by the Information Logic Processing Module (ILPM) to filter candidate target objects. The bounding boxes, segmentation masks and confidence scores (Logits) of all candidate target objects are output. Then, the Doubao-1.5-pro large model is used to verify each candidate target object one by one. Then, the scores of the two detections are fused proportionally to obtain the score of the candidate target object. Finally, objects that meet the requirements are retained according to the score to complete the selection of target objects. If no object meets the requirements, the process is repeated. Step (3) specifically includes the following: (3-1) The LangSAM model has the ability to match language and vision across modalities. It calls the LangSAM pre-trained visual localization model and performs preliminary visual detection based on language prompts on the cluttered scene image according to the target object identification information and reference object identification information extracted in step (2) to obtain at least one candidate target object in the scene. For each candidate target object, it outputs its corresponding bounding box, segmentation mask and detection confidence score Logits. This process is automatically performed by the model and the confidence score Logits value is normalized to eliminate numerical magnitude differences and ensure the effectiveness of subsequent score fusion. Among them, the LangSAM pre-trained visual localization model generates candidate regions based on language prompts and outputs bounding boxes, segmentation masks, and detection confidence scores, which belong to the existing visual localization capabilities of the pre-trained model. Based on this, the present invention normalizes the detection confidence scores Logits to uniformly map the detection confidence of different candidate target objects to a preset numerical range, reducing the impact of numerical magnitude differences on subsequent score fusion, and uses the normalized detection confidence scores, bounding boxes, and segmentation masks as inputs for semantic verification in step (3-2) and score fusion in step (3-3).

[0016] (3-2) The image of each candidate target object is cropped into its own image according to the bounding box, and input together with the name of the main object to be captured and the name of the reference object to be captured into the Doubao-1.5-pro large model constrained by the system-level prompt. The Doubao-1.5-pro large model completes the matching degree verification of the image of the candidate target object with the name of the main object to be captured and the name of the reference object extracted in step (2), that is, the secondary verification, and outputs a discrete verification score, which is used to represent the semantic matching degree between the image and the language instruction text. The discrete verification score in step (3-2) is calculated by the built-in function of the Doubao-1.5-pro large model. That is, each candidate target object is cropped according to its bounding box to obtain a candidate image, and the candidate image, together with the target object identification information, reference object identification information and spatial relationship information extracted in step (2), is input into the Doubao-1.5-pro large model constrained by the system-level prompt. The Doubao-1.5-pro large model verifies the semantic consistency between the candidate image and the language instruction and outputs the semantic verification score. The semantic verification score is output in a discrete scoring manner: in the initialization stage of step (1), a discrete scoring set is pre-set, and the Doubao-1.5-pro large model selects a score from the discrete scoring set as the semantic verification score based on the semantic consistency verification result; (3-3) To obtain the comprehensive semantic matching evaluation result of the candidate objects, the confidence score Logits output by the LangSAM pre-trained visual localization model in step (3-1) and the validation score Score output by the Doubao-1.5-pro large model in step (3-2) are weighted and fused to calculate the comprehensive score Final_Score. The fusion formula is shown in formula (3): (3) The weight ratio m:n in the weighted fusion of step (3-3) is the preset score fusion weight ratio in step (1-3), usually taken as 0.6:0.4. This weight ratio is the optimal value obtained by testing 2000 crawling tasks on the Roborefit training set. (3-4) Perform candidate target object screening based on the final score (Final_Score) calculated in step (3-3).

[0017] The rules for screening candidate target objects in step (3-4) are as follows: (3-4-1) If the final score of the candidate object is Final_Score and the confidence score is Logits, then the two verifications of "detection by the LangSAM pre-trained visual localization model and verification by the Doubao-1.5-pro large model" bring positive gain effect, that is, the image of the selected target object and the text of the language instruction are very likely to match, and the candidate target object can be accepted. (3-4-2) If the confidence score Logits of the candidate target object is ≥0.7, it means that the confidence of the preliminary detection result is high enough, and the candidate target object can be directly accepted to improve the system processing efficiency. (3-4-3) If all candidate target objects fail the above screening, the re-detection mechanism will be started automatically. That is, the segmentation mask regions of all unaccepted objects will be blocked and a new scene image will be generated. The LangSAM pre-trained visual localization model will re-perform the detection on the new image, repeating the above steps (3-1) to (3-3) and screening again until the final target object is determined.

[0018] (4) Target 3D Grasping Filtering and Execution: Input the final target object segmentation mask output by the self-calibrating visual positioning module (SCGM) in step (3) into the target 3D Grasping Filter module (Object-3D Grasp Filter). Perform a two-layer filtering of spatial position initial screening and execution angle secondary screening on the grasping posture generated by the GraspNet pre-trained grasping network, select the optimal grasping posture, and have the robot perform the target object grasping.

[0019] Step (4) specifically includes the following: (4-1) GraspNet pre-trained grasping network, as a mature pre-trained grasping network in this field, can directly process cluttered input scenes globally, automatically detect and generate candidate grasping poses for all visible objects in the scene, without needing to retrain for specific objects or environments, and has a natural foundation for cross-scene generalization. The network internally calculates and assigns a confidence score to each grasping pose based on criteria such as grasping stability, contact area, and mechanical feasibility. The higher the score, the stronger the stability of the grasping pose under ideal conditions, providing a quantitative basis for subsequent optimal selection. (4-2) Based on the segmentation mask of the candidate target object output by the self-correcting visual positioning module SCGM in step (3), extract the 3D point cloud data corresponding to the target object to achieve "only capture the target and not the interference". (4-3) The candidate grasping pose set generated by the GraspNet pre-trained grasping network in step (1-2) is traversed one by one, and the spatial correspondence between the effective area of ​​each candidate grasping pose and the 3D point cloud data of the target object extracted in step (4-2) is calculated. Only candidate grasping poses whose effective area falls on the surface of the 3D point cloud of the target object or within its preset neighborhood are retained as valid grasping poses. Candidate grasping poses whose effective area falls in non-target areas are eliminated. Through spatial position filtering, the subsequently retained grasping poses correspond one-to-one with the final target object in 3D space, thereby reducing the risk of mistakenly grasping background objects, adjacent objects or invalid areas in cluttered scenes and improving the consistency between the language command target and the actual grasping target.

[0020] The non-target regions in step (4-3) include: empty regions inside the target object without effective point cloud support, background regions outside the target object segmentation mask, point cloud surface regions corresponding to other non-target objects, and regions where effective three-dimensional point cloud data has not been formed due to occlusion, reflection, or lack of depth.

[0021] (4-4) To solve the problem that high-scoring grasping postures in cluttered scenes cannot be actually executed due to unreasonable angles, robot executability filtering is further performed on the effective grasping postures initially screened in step (4-3). In step (4-4), the effective grasping postures are further subjected to robot executability filtering. Specifically, this means: using the normal direction of the support surface where the target object is located or the opposite direction of gravity as the reference direction, calculating the angle between each grasping posture and this direction, and retaining only grasping postures with an angle ≤30°. This threshold can be flexibly adjusted according to the actual hardware and scenario, such as the configuration of the robotic arm, the type of end effector, and the height of the table. Grasping postures that are located on the side of the object, conflict with the direction of robot movement, have a rotation angle that is approximately parallel to the support surface, or are prone to slipping or collision are eliminated. This ensures that the selected postures fully conform to the motion constraints of the real robotic arm, significantly improving the actual landing capability in cluttered scenarios.

[0022] (4-5) After the two-layer filtering of spatial position filtering in step (4-3) and angle filtering in step (4-4), the grasping posture with the highest original confidence score calculated by the GraspNet pre-trained grasping network in step (4-1) is selected from the effective grasping postures as the final optimal grasping posture. The posture position parameters and posture attitude parameters corresponding to the final grasping posture are sent to the robot control system, and the robot moves to the target grasping position and performs the grasping action accordingly.

[0023] In summary, this method enables the grasping of general, robust, and practical target objects using only a single natural language command through the cooperation of the three models mentioned above, achieving a highly generalized language-to-grabbing process.

[0024] The working principle of this invention: The core of this invention is to provide a language instruction grasping method with generalization ability in cluttered scenes. It is to build an integrated system composed of three major modules: Information Logic Processing Module (ILPM), Self-Corrective Visual Localization Module (SCGM), and Object-3D Grasp Filter. With the Doubao-1.5-pro large model, LangSAM visual localization model and GraspNet grasping network as the core, it realizes generalized language instruction grasping through a closed-loop process of "language instruction parsing - visual target selection - grasping posture optimization".

[0025] This invention adopts a three-level modular architecture consisting of information logic processing, self-correcting visual positioning, and target 3D grasping and filtering. Each module functions independently and interacts smoothly. The core uses mature pre-trained models, which can flexibly adjust parameters such as parameter mapping rules, score fusion weight ratio, and grasping angle threshold according to different application scenarios. It can adapt to different types of robotic arms and various cluttered indoor scenes, with excellent compatibility and scalability. At the same time, a confidence threshold is set in the visual positioning stage, and the preliminary detection results with high confidence (Logits≥0.7) are directly accepted, eliminating invalid secondary verification steps. This improves the overall processing efficiency of the system while ensuring positioning accuracy, and better meets the real-time requirements of actual service robot applications.

[0026] Compared with existing technologies, this invention achieves comprehensive improvement in language command robot grasping tasks in cluttered scenes. Its advantages include: ① Utilizing the Doubao-1.5-pro large model to achieve structured analysis of complex spatial language commands, transforming fuzzy language spatial relationships into quantifiable geometric feature parameters. This adapts to diverse natural language expressions without requiring model retraining for specific commands, significantly improving the robot's understanding of language commands and its scene generalization ability; ② By constructing a closed-loop self-calibrating visual localization module composed of LangSAM and a large model, the large model is used for visual... The system performs a second verification on the initial detection results and can also initiate a re-detection mechanism when the verification fails. This effectively solves the detection error problem caused by similar objects and complex backgrounds in cluttered environments, achieves accurate positioning of target objects, and improves the robustness of visual positioning. In the grasping posture generation stage, a two-layer filtering method is used, which performs initial spatial position screening based on the object's 3D point cloud and secondary screening based on the execution angle when the vertical angle of gravity is ≤30°. This eliminates grasping postures with invalid spatial positions and those that conflict with the robotic arm's execution angle, ensuring that the selected optimal posture is actually executable and greatly improving the robot's grasping success rate. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall system architecture of a generalized language command robot grasping method for chaotic scenarios, as described in this invention.

[0028] Figure 2 This is a flowchart illustrating the closed-loop workflow and re-detection mechanism of the self-correcting visual positioning module in a generalized language command robot grasping method for cluttered scenes, as described in this invention.

[0029] Figure 3 shows the effect of the grasping action sequence of the language command "I want the tennis ball behind" according to an embodiment of the present invention (where 3-a is the scene input, 3b is the target localization, 3-c is the grasping posture generation, 3d is the robotic arm grasping, and 3e is the entire delivery process).

[0030] Figure 4 shows the effect of the grasping action sequence of the language instruction "Can you hand me the rightmost toothpaste box?" according to another embodiment of the present invention (where 4-a is scene input, 4b is target localization, 4-c is grasping posture generation, 4-d is robotic arm grasping, and 4-e is the entire handover process). Detailed Implementation

[0031] Example: A method for robot grasping generalized language commands in cluttered scenes, characterized by the following steps: (1) Load the Doubao-1.5-pro large model, the LangSAM pre-trained visual localization model and the GraspNet pre-trained grasping network, and perform initialization settings so that the robot grasping system is in a state of waiting to receive external scene images and language grasping instructions. The robot grasping system consists of an information logic processing module, a self-calibrating visual positioning module, and a target 3D grasping and filtering module. These three modules are cascaded sequentially to form a generalized language-command robot grasping closed-loop system for chaotic scenes. Each module contains dedicated functional subunits, and the modules communicate via standardized data interfaces to achieve unidirectional signal transmission and closed-loop verification. Figure 1 As shown in Figure 3.

[0032] The information logic processing module has a built-in language instruction parsing subunit, which is the core of the system's language understanding. It is used to extract key information of language instructions and reason about spatial position relationships, and to transform fuzzy natural language into quantifiable machine-executable logic. Its input signals are external natural language capture instructions and cluttered scene RGB-D images, and its output signals are structured target object information, reference object information, spatial position relationship parameters and geometric feature parameters.

[0033] The self-correcting visual localization module is used to complete preliminary object detection, semantic secondary verification, score weighted fusion, and candidate selection. It achieves self-correcting and accurate localization of the target object through a re-detection mechanism. Specifically, it includes a visual detection subunit, a semantic verification subunit, and a score fusion subunit. The visual detection subunit receives structured object information and spatial position parameters output by the information logic processing module at its input end, and outputs the bounding box, segmentation mask, and normalized detection confidence score (Logits) of the candidate target object at its output end. The semantic verification subunit receives the candidate image with cropped bounding box, the target object name, reference object name, and spatial position relationship description output by the information logic processing module at its input end, and outputs the discrete verification score (Score) for semantic consistency matching at its output end. The score fusion subunit receives the normalized detection confidence score (Logits) and the semantic verification score (Score) at its input end, and outputs the weighted fusion comprehensive confidence score (Final_Score). The output signal is the target object bounding box, segmentation mask, and comprehensive confidence score output by the score fusion subunit.

[0034] The target 3D grasping and filtering module is used to generate candidate grasping poses, filter spatial validity, filter the executability of the robotic arm, and select the optimal pose. It outputs grasping commands that can be directly sent to the robot control system. Specifically, it consists of a grasping pose generation subunit, a spatial position filtering subunit, an execution angle filtering subunit, and an optimal pose selection subunit. The input signals of the target 3D grasping and filtering module are the target object segmentation mask output by the self-calibrating visual positioning module and the scene 3D point cloud data generated from the scene RGB-D image. The output signal is the optimal grasping pose parameters that the robot can execute. The input end of the grasping pose generation subunit receives the original scene image, and the output end outputs the candidate grasping pose set of all objects in the global scene and the stability confidence score corresponding to each grasping pose. The input end of the spatial position filtering subunit receives the candidate grasping pose set output by the grasping pose generation subunit, the target object segmentation mask output by the self-correcting visual positioning module, and the scene 3D point cloud data. The output end outputs the effective grasping pose set whose effective area falls within the effective range of the target object's 3D point cloud. The input end of the execution angle filtering subunit receives the set of effective grasping poses output by the spatial position filtering subunit and the system preset grasping pose angle threshold constraint parameters, and the output end outputs a set of compliant grasping poses that meet the reachable poses and collision-free constraints of the robotic arm. The optimal pose selection subunit receives the set of compliant grasping poses output by the execution angle filtering subunit and the stability confidence score of each grasping pose output by the grasping pose generation subunit. The output unit outputs the optimal grasping pose position parameters and attitude parameters with the highest confidence after filtering, and transmits them to the robot control system.

[0035] The initial setup process includes: (1-1) Load the Doubao-1.5-pro large model, the LangSAM pre-trained visual localization model, and the GraspNet pre-trained crawling network. Configure system-level prompts for the language instruction parsing subunit in the information logic processing module and the semantic verification subunit in the self-correcting visual localization module. The system-level prompts are used to define the task role, output format, and inference range of the Doubao-1.5-pro large model. The Doubao-1.5-pro large model is responsible for language instruction parsing and positional relationship inference. The LangSAM pre-trained visual localization model completes the initial visual localization of the object. The Doubao-1.5-pro large model and the LangSAM pre-trained visual localization model work together to achieve accurate selection of the target object. The language instruction parsing subunit in the information logic processing module and the semantic verification subunit in the self-correcting visual localization module are both well-known functional units based on the natural language processing large model. They are implemented by relying on the language understanding and visual question answering capabilities of the Doubao-1.5-pro large model and belong to mature technologies in this field.

[0036] (1-2) A set of candidate grasping poses in the scene is generated by the GraspNet pre-trained grasping network. These poses are used to obtain executable grasping poses after spatial position filtering and execution angle filtering in the subsequent target 3D grasping filtering module. After 3D filtering, executable grasping postures are obtained. In the target 3D grasping filtering module, the executable grasping poses are obtained after sequential processing by the spatial position filtering subunit and the execution angle filtering subunit. After 3D filtering, executable postures are obtained. Spatial position filtering and execution angle filtering are both predetermined functions of the target 3D grasping filtering module. The subunit composition, function and signal flow of this module have been fully disclosed above.

[0037] (1-3) Preset the geometric feature parameters of candidate target objects And the mapping rules for spatial location relationships, the score fusion weight ratio, and the threshold for the grasping posture angle. Among them, , The coordinates of the center pixel of the bounding box of the candidate target object in the two-dimensional image plane; The distance is the Euclidean distance between the center point of the bounding box of the candidate target object and the reference object; the calculation formula is shown in formula (1); The relative rotation angle of the candidate target object with respect to the reference object is calculated as shown in formula (2). (1) (2) in, , The coordinates of the center point of the target object. , The coordinates are the center coordinates of the reference object; The mapping rules for spatial positional relationships in steps (1-3) specifically refer to: ① When there is no reference object, the two-dimensional image plane coordinate system is used as the judgment benchmark, and the relative position of the candidate target object in the image is determined by the pixel coordinates of the center point of the bounding box; among which, the horizontal pixel coordinates Increasing the pixel value indicates that the candidate object is positioned further to the right in the image, while decreasing it indicates that it is positioned further to the left; vertical pixel coordinates Increasing the value indicates that the candidate target object is lower in the image, while decreasing it indicates that it is higher. For cases where the language command involves "in front" or "behind", the vertical position of the image is used as an approximate criterion. That is, under the preset shooting angle, the candidate target object located at the top of the image is judged to be further back, and the candidate target object located at the bottom of the image is judged to be further forward. ② When there is a reference object, the distance between the candidate target object and the reference object is determined by the Euclidean distance d. The smaller the Euclidean distance d, the closer the candidate target object is to the reference object. ③ Determine the direction between the candidate target object and the reference object by the value of the relative rotation angle r.

[0038] (1-4) Connect the data transmission interface between the information logic processing module, the self-correcting visual positioning module, and the target 3D grasping and filtering module, and verify the smooth operation of the entire process link; wherein, the information logic processing module outputs structured language parsing results based on the Doubao-1.5-pro large model, the self-correcting visual positioning module outputs target object filtering results based on the LangSAM pre-trained visual positioning model and the Doubao-1.5-pro large model, and the target 3D grasping and filtering module outputs a set of candidate grasping poses based on the GraspNet pre-trained grasping network, and completes the grasping pose filtering according to the final target object segmentation mask; then open the scene image input interface and the natural language grasping command input interface, so that the generalized language command robot grasping system in the cluttered scene is in a state of waiting to receive external scene images and language grasping commands.

[0039] The score fusion weight ratio refers to the weighted ratio between the detection confidence Logits output by the LangSAM pre-trained visual localization model and the semantic verification score Score output by the Doubao-1.5-pro large model, which is generally 0.6:0.4.

[0040] The method for determining the grasping posture angle threshold specifically includes: determining the reachable posture range of the end effector based on the robotic arm configuration; determining the allowable gripping direction range based on the end effector type; determining the safe approach path range based on the table height or the height of the target object's support surface; and determining the collision constraint range based on the obstacle distribution in the scene. Based on this, through grasping experiments using candidate angle thresholds, the grasping success rate, collision rate, and trajectory execution rate corresponding to each candidate angle threshold are statistically analyzed. The angle threshold with the best overall performance is selected as the grasping posture angle threshold for actual use. In specific implementation, this threshold can be flexibly adjusted according to the actual hardware and scene, such as the robotic arm configuration, end effector type, and table height. In this embodiment, the grasping posture angle threshold is 30°.

[0041] (2) Information logic analysis and extraction: The system constructed in step (1) inputs the captured images of the cluttered scene and the natural language capture instructions. The information logic processing module ILPM converts the ambiguous language instructions (natural language describing the positional relationship of objects, such as "Give me an apple and a banana to the right" or "Give me a banana behind") into the geometric feature parameters and positional information of the target object. This means performing two sub-tasks: key information extraction and positional relationship reasoning. Sub-task 1 is used for key information extraction, and task 2 is used for positional relationship reasoning.

[0042] Subtask 1, key information extraction, refers to the process by which the Doubao-1.5-pro large model parses the original natural language crawling instructions under the constraints of the system-level prompt, and extracts the following three core elements: ①Target Object Name: The name of the main object to be captured; ②Reference Object Name: The name of the reference object; ③Relative Position Relationship: A linguistic phrase describing the spatial relationship between a target object and a reference object; If the input language instruction only contains the target object, without a reference object or spatial relationship, fill the ReferenceObject Name and Relative Position Relationship fields with None, and finally output the extraction results in a structured template of "Target Object Name, Reference Object Name, Position Relationship Phrase".

[0043] Subtask 2 performs positional relationship reasoning, namely: ① calling the LangSAM pre-trained visual localization model loaded in step (1) to perform preliminary identification of potential target objects and reference objects in the scene image to be identified, and obtaining the pixel coordinates of the center point of the bounding box of each object in the two-dimensional image plane; ② calculating the four-element list of geometric feature parameters for each candidate target object using the Doubao-1.5-pro large model. ,at this time, , The field is fixed as None, only reserved , Coordinate parameters; ③ According to the mapping rules of the spatial position relationship predetermined in step (1), the geometric feature parameters are transformed into spatial position relationships, and finally the geometric feature parameters and position information of the candidate target object are output.

[0044] (3) Self-calibrating visual localization processing: The object name and description information output by the information logic processing module ILPM are input into the self-calibrating visual localization module SCGM. A closed-loop visual localization mechanism is adopted. First, preliminary detection is performed based on the object name and description information input by the information logic processing module ILPM to filter candidate target objects. The bounding boxes, segmentation masks and confidence scores Logits of all candidate target objects are output. Then, the Doubao-1.5-pro large model is used to check each candidate target object one by one for secondary verification. Then, the scores of the two detections are fused according to the ratio to obtain the score of the candidate target object. Finally, the objects that meet the requirements are left according to the score to complete the selection of target objects. If no object meets the requirements, the process is repeated. Step (3) specifically includes the following: (3-1) Call the LangSAM pre-trained visual localization model. Based on the target object identification information and reference object identification information extracted in step (2), perform preliminary visual detection based on language prompts on the cluttered scene image to obtain at least one candidate target object in the scene. For each candidate target object, output its corresponding bounding box, segmentation mask and detection confidence score Logits. This process is automatically performed by the model. The confidence score Logits value is normalized to eliminate numerical magnitude differences and ensure the effectiveness of subsequent score fusion. (3-2) The image of each candidate target object is cropped into its own image according to the bounding box, and input together with the name of the main object to be captured and the name of the reference object to be captured into the Doubao-1.5-pro large model constrained by the system-level prompt. The Doubao-1.5-pro large model completes the matching degree verification of the image of the candidate target object with the name of the main object to be captured and the name of the reference object extracted in step (2), that is, the secondary verification, and outputs a discrete verification score, which is used to represent the semantic matching degree between the image and the language instruction text. The discrete verification score in step (3-2) is calculated by the built-in function of the Doubao-1.5-pro large model. That is, each candidate target object is cropped according to its bounding box to obtain a candidate image, and the candidate image, together with the target object identification information, reference object identification information and spatial relationship information extracted in step (2), is input into the Doubao-1.5-pro large model constrained by the system-level prompt. The Doubao-1.5-pro large model verifies the semantic consistency between the candidate image and the language instruction and outputs the semantic verification score. The semantic verification score is output in a discrete scoring manner: in the initialization stage of step (1), a discrete scoring set is pre-set, and the Doubao-1.5-pro large model selects a score from the discrete scoring set as the semantic verification score based on the semantic consistency verification result; In this embodiment, the discrete score set is preferably {0.05, 0.25, 0.5, 0.75, 0.95}, where 0.05 represents an extremely low matching probability, 0.25 represents a low matching probability, 0.5 represents a medium matching probability, 0.75 represents a relatively high matching probability, and 0.95 represents an extremely high matching probability.

[0045] (3-3) To obtain the comprehensive semantic matching evaluation result of the candidate objects, the confidence score Logits output by the LangSAM pre-trained visual localization model in step (3-1) and the validation score Score output by the Doubao-1.5-pro large model in step (3-2) are weighted and fused to calculate the comprehensive score Final_Score. The fusion formula is shown in formula (3): (3) The weight ratio m:n of the weighted fusion is the preset weight ratio of the score fusion in step (1-3), usually taken as 0.6:0.4. This weight ratio is the optimal value obtained by testing 2000 crawling tasks on the Roborefit training set.

[0046] (3-4) Perform candidate target object screening based on the final score (Final_Score) calculated in step (3-3).

[0047] (3-4-1) If the final score of the candidate object is Final_Score and the confidence score is Logits, then the two verifications of "detection by the LangSAM pre-trained visual localization model and verification by the Doubao-1.5-pro large model" bring positive gain effect, that is, the image of the selected target object and the text of the language instruction are very likely to match, and the candidate target object can be accepted. (3-4-2) If the confidence score Logits of the candidate target object is ≥0.7, it means that the confidence of the preliminary detection result is high enough, and the candidate target object can be directly accepted to improve the system processing efficiency. (3-4-3) If all candidate target objects fail the above screening, the re-detection mechanism will be started automatically. That is, the segmentation mask regions of all unaccepted objects will be blocked and a new scene image will be generated. The LangSAM pre-trained visual localization model will re-perform the detection on the new image, repeating the above steps (3-1) to (3-3) and screening again until the final target object is determined.

[0048] (4) Target 3D Grasping Filtering and Execution: Input the final target object segmentation mask output by the self-calibrating visual positioning module (SCGM) in step (3) into the target 3D Grasping Filter module (Object-3D Grasp Filter). Perform a two-layer filtering of spatial position initial screening and execution angle secondary screening on the grasping posture generated by the GraspNet pre-trained grasping network, select the optimal grasping posture, and have the robot perform the target object grasping.

[0049] (4-1) The GraspNet pre-trained grasping network automatically calculates and assigns a confidence score to each grasping posture. The higher the score, the stronger the stability of the grasping posture under ideal conditions, providing a quantitative basis for subsequent optimal selection. (4-2) Based on the segmentation mask of the candidate target object output by the self-correcting visual positioning module SCGM in step (3), extract the 3D point cloud data corresponding to the target object to achieve "only capture the target and not the interference". (4-3) The candidate grasping pose set generated by the GraspNet pre-trained grasping network in step (1-2) is traversed one by one, and the spatial correspondence between the effective area of ​​each candidate grasping pose and the 3D point cloud data of the target object extracted in step (4-2) is calculated. Only candidate grasping poses whose effective area falls on the surface of the 3D point cloud of the target object or within its preset neighborhood are retained as valid grasping poses. Candidate grasping poses whose effective area falls in non-target areas are eliminated. Through spatial position filtering, the subsequently retained grasping poses correspond one-to-one with the final target object in 3D space, thereby reducing the risk of mistakenly grasping background objects, adjacent objects or invalid areas in cluttered scenes and improving the consistency between the language command target and the actual grasping target.

[0050] Non-target regions include: empty areas inside the target object without effective point cloud support, background areas outside the target object segmentation mask, point cloud surface areas corresponding to other non-target objects, and areas where effective 3D point cloud data has not been formed due to occlusion, reflection, or lack of depth.

[0051] (4-4) Further perform robot executability filtering on the effective grasping postures initially screened out in step (4-3); In step (4-4), the effective grasping postures are further filtered for robot executability. Specifically, this means: using the normal direction of the support surface where the target object is located or the opposite direction of gravity as the reference direction, calculating the angle between each grasping posture and this direction, and retaining only grasping postures with an angle ≤30° (this threshold can be flexibly adjusted according to the actual hardware and scenario such as the configuration of the robotic arm, the type of end effector, and the height of the table). Grasping postures that are located on the side of the object, conflict with the direction of robot movement, have a rotation angle that is approximately parallel to the support surface, or are prone to slipping or collision are eliminated. This ensures that the selected postures fully conform to the motion constraints of the real robotic arm, significantly improving the actual landing capability in cluttered scenarios.

[0052] (4-5) After the two-layer filtering of spatial position filtering in step (4-3) and angle filtering in step (4-4), the grasping posture with the highest original confidence score calculated by the GraspNet pre-trained grasping network in step (4-1) is selected from the effective grasping postures as the final optimal grasping posture. The posture position parameters and posture attitude parameters corresponding to the final grasping posture are sent to the robot control system, and the robot moves to the target grasping position and performs the grasping action accordingly.

[0053] In summary, this method enables the grasping of general, robust, and practical target objects using only a single natural language command through the cooperation of the three models mentioned above, achieving a highly generalized language-to-grabbing process.

[0054] The following is a further explanation using specific examples: As shown in Figures 3 and 4, this embodiment uses a cluttered office desktop scenario as the application environment, employing a six-degree-of-freedom collaborative robotic arm as the execution entity to complete the task of grasping and delivering target objects based on natural language spatial commands. The desktop scenario contains multiple yellow spherical objects, multiple long box-shaped objects, water cups / cylinders, books / notebooks, tape-like items, and other cluttered objects. The experiment verifies two language commands respectively: a. Instruction A (corresponding to Experiment Figure 3: "I want the tennis ball in the back"); b. Instruction B (corresponding to Experiment Figure 4): "Can you hand me the toothpaste box on the far right?"

[0055] Among them, "behind" refers to the spatial relationship description of front-back (depth / distance), and "far right" refers to the spatial relationship description of the global extremum (maximum on the right). Both are easily interfered with by the detection of "multiple instances of the same type and objects with similar appearances" in cluttered scenes. To address this, this embodiment adopts a pipeline closed loop of "Information Logic Analysis and Extraction (ILPM) + Self-Corrective Visual Localization (SCGM) + Object-3D Grasp Filter" to achieve accurate grasping.

[0056] The specific execution steps of this embodiment are as follows: (1) System initialization: Load the Doubao-1.5-pro large model, LangSAM pre-trained model and GraspNet pre-trained model, and configure the system-level prompts for each module; Preset positional relationship mapping rules: "Behind / Front" mapping rule: Prioritize sorting based on depth information, and judge the one with greater depth as "behind"; When there is no explicit depth or the depth is unstable, use the vertical position of the image pixel coordinates (smaller y, higher in the image) as an auxiliary criterion; "Rightmost" mapping rule: Sort candidate instances of the same type according to the pixel center point x value, and judge the one with the largest x as "rightmost"; At the same time, set the score fusion weight ratio to 0.6:0.4 and the grasping pose angle threshold to 30°, complete the system linkage debugging, and wait to receive scene images and language commands.

[0057] (2) Information Logic Analysis and Extraction (ILPM) The system inputs desktop scene images and language commands to extract key information and infer positional relationships. Figures 3 and 4 show the continuous action frames of two different language commands. Figure 3 shows the complete execution sequence of the continuous action frames for "I want the tennis ball in the back," from left to right: scene input, target localization, grasping posture generation, and the entire process of robotic arm grasping and handing over. Figure 4 shows the complete execution sequence of the continuous action frames for "Can you hand me the toothpaste box on the far right?", from left to right: scene input, target localization, grasping posture generation, and the entire process of robotic arm grasping and handing over. 1) Command A: "I want the tennis ball in the back" (corresponding to Figure 3) Key information extraction: Doubao-1.5-pro outputs structured results under prompt constraints: tennisball, behind / back; Positional Relationship Reasoning: LangSAM detects at least two tennis ball instances (denoted as tennis ball 1 and tennis ball 2) in an image and extracts their pixel center and depth geometric feature parameters (represented by [x, y, z, ...], where z is a depth distance estimate). Based on the "behind" mapping rule, candidate tennis balls are sorted by their z-value (or y-value for assistance), and the tennis ball instance with greater depth (further away, higher in the image) is selected as the target tennis ball, thus consistent with the process shown in the upper row of the experimental image where "the robotic arm grasps the tennis ball further back".

[0058] 2) Instruction B: "Could you hand me the toothpaste box on the far right?" (See Figure 4) Key information extraction: Doubao-1.5-pro parses and outputs structured results: toothpaste box, rightmost. Positional Relationship Reasoning: LangSAM identifies multiple long, box-shaped candidates (at least three toothpaste boxes / similar packaging boxes) in the image and extracts the pixel center coordinates of each candidate. Based on the "rightmost" mapping rule, the box-shaped instance with the largest x value is selected as the target toothpaste box, and the candidate target object is output as the "rightmost toothpaste box", which corresponds to the sequence of "robotic arm grabbing and delivering the rightmost toothpaste box" shown in the lower row of the experimental image.

[0059] (3) Self-correcting visual positioning (SCGM) The object name and spatial description output by ILPM are input into the self-calibrating visual localization module SCGM to construct a closed-loop visual localization mechanism of "preliminary detection - secondary verification - score fusion - candidate screening - re-detection", which can achieve accurate screening under interference of multiple instances of the same type / similar appearance.

[0060] 1) LangSAM initial detection: For instruction A, the image is detected and segmented based on "tennis ball", and the bounding boxes and masks of multiple tennis ball candidates and their Logits are output; For instruction B, the image is detected and segmented based on "toothpaste box", and the bounding boxes and masks of multiple box-shaped candidates and their Logits are output.

[0061] 2) Doubao-1.5-pro secondary verification: The cropped image of each candidate instance is input into the large model to verify semantic consistency: Instruction A: Verify whether the candidate is "tennis ball" and check the consistency of the candidate ranking in combination with the description of "behind / far away"; Instruction B: Verify whether the candidate is "toothpaste box / toothpaste packaging box" and check the consistency of "rightmost (largest x)"; Output the score of each candidate to suppress similar-looking box-shaped objects and interference terms that do not conform to spatial constraints in multiple instances of the same type.

[0062] 3) Score-weighted fusion, fused according to a weight ratio of 0.6:0.4.

[0063] 4) Closed loop of candidate screening and re-detection Candidate selection is performed using the "fusion score - original confidence level" consistency criterion: a candidate is accepted as valid if its Final_Score can support its Logits and satisfies the spatial relation constraints of ILPM (Instruction A satisfies "further back", Instruction B satisfies "rightmost"); otherwise, it is discarded, and re-detection / re-verification of the remaining candidates is triggered until convergence to the unique target. Ultimately: Instruction A determines the target as the tennis ball instance further back; Instruction B determines the target as the toothpaste box instance on the far right.

[0064] (4) Object-3D Grasp Filtering and Execution The target object segmentation mask output by SCGM is input into the target 3D grasping filtering module to perform "double-layer filtering" on the grasping poses generated by GraspNet, selecting the optimal grasping pose and having the robotic arm execute the grasping and delivery. GraspNet generates grasping pose candidates: multiple sets of grasping pose candidates and their scores are generated for the target object from the scene image; initial spatial position screening: combining the target object mask and 3D point cloud, effective grasping poses falling on the target object surface point cloud are selected, and invalid poses falling on the background or adjacent objects are eliminated; execution angle screening: the angle between the effective grasping pose and the perpendicular direction of gravity is calculated, and the set of poses with an angle ≤30° is retained; optimal pose selection and execution: the pose with the highest score among those that meet the angle threshold is selected as the optimal grasping pose and sent to the six-degree-of-freedom collaborative robotic arm for execution. The robotic arm completes the following tasks as shown in Experiment Figures 3 and 4: grasping the tennis ball behind and completing the delivery / relocation; grasping the toothpaste box on the far right and completing the delivery.

[0065] This embodiment verifies the effectiveness and accuracy of the method of the present invention for two typical spatial language instructions in a cluttered office desktop scenario: Even with interference from multiple instances of the same type and similar-looking objects, the system can stably parse the target category and spatial constraints in the language instructions. Through "logical reasoning + self-correcting localization closed loop + 3D grasping dual-layer filtering," it obtains the optimal grasping posture for practical execution and completes the grasping task. If the language instructions or the cluttered scenario changes, only parameters such as the positional relationship mapping rules need to be adjusted; there is no need to retrain the model, demonstrating the method's generalization ability.

Claims

1. A method for robot grasping generalized language commands in cluttered scenarios, characterized in that... It includes the following steps: (1) Load the Doubao-1.5-pro large model, the LangSAM pre-trained visual localization model and the GraspNet pre-trained grasping network, and perform initialization settings so that the robot grasping system is in a state of waiting to receive external scene images and language grasping instructions. (2) Information logic analysis and extraction: The system constructed in step (1) is input with the captured images of the cluttered scene and the natural language capture instructions. The information logic processing module converts the ambiguous language instructions into the geometric feature parameters and position information of the target object. (3) Self-calibrating visual localization processing: The object name and description information output by the information logic processing module ILPM are input into the self-calibrating visual localization module. A closed-loop visual localization mechanism is adopted. First, preliminary detection is performed based on the object name and description information input by the information logic processing module ILPM to filter candidate target objects. The bounding boxes, segmentation masks and confidence scores Logits of all candidate target objects are output. Then, the Doubao-1.5-pro large model is used to check each candidate target object one by one for secondary verification. Then, the scores of the two detections are fused according to the ratio to obtain the score of the candidate target object. Finally, the objects that meet the requirements are left according to the score to complete the selection of target objects. If no object meets the requirements, the process is repeated. (4) Target 3D grasping filtering and execution: Input the final target object segmentation mask output by the self-calibration visual positioning module in step (3) into the target 3D grasping filtering module, perform a two-layer filtering of spatial position initial screening and execution angle secondary screening on the grasping posture generated by the GraspNet pre-trained grasping network, select the optimal grasping posture and have the robot perform the target object grasping.

2. The method for robot grasping generalized language commands in cluttered scenes according to claim 1, characterized in that... The robot grasping system consists of an information logic processing module, a self-correcting visual positioning module, and a target 3D grasping and filtering module. These three modules are cascaded together to form a generalized language command robot grasping closed-loop system for chaotic scenes. The information logic processing module has a built-in language instruction parsing subunit, which is the core of the system's language understanding. It is used to extract key information of language instructions and reason about spatial position relationships, and to convert fuzzy natural language into quantifiable machine-executable logic. Its input signals are external natural language capture instructions and cluttered scene RGB-D images, and its output signals are structured target object information, reference object information, spatial position relationship parameters and geometric feature parameters. The self-correcting visual localization module is used to complete preliminary object detection, semantic secondary verification, score weighted fusion, and candidate selection. It achieves self-correcting and accurate localization of the target object through a re-detection mechanism. Specifically, it includes a visual detection subunit, a semantic verification subunit, and a score fusion subunit. The input end of the visual detection subunit receives the structured object information and spatial position parameters output by the information logic processing module. Its output end is connected to the input end of the semantic verification subunit, and outputs the bounding box, segmentation mask, and normalized detection confidence Logits of the candidate target object. The semantic verification subunit receives candidate images cropped from bounding boxes, target object names, reference object names, and spatial relationship descriptions output by the information logic processing module at its input end. Its output end connects to the input end of the score fusion subunit and outputs a discrete verification score (Score) for semantic consistency matching. The score fusion subunit receives the normalized detection confidence (Logits) output by the visual detection subunit and the semantic verification score (Score) output by the semantic verification subunit at its input end. Its output signal is the target object bounding box, segmentation mask, and final confidence score (Final_Score) output by the score fusion subunit. The target 3D grasping and filtering module is used to generate candidate grasping poses, filter spatial validity, filter the executability of the robotic arm, and select the optimal pose. It outputs grasping commands that can be directly sent to the robot control system. Specifically, it consists of a grasping pose generation subunit, a spatial position filtering subunit, an execution angle filtering subunit, and an optimal pose selection subunit. The target 3D grasping and filtering module receives the target object segmentation mask output by the self-calibrating visual positioning module and the scene 3D point cloud data generated from the scene RGB-D image. Its output signal is the optimal grasping pose parameters that the robot can execute. The grasping pose generation subunit receives the original scene image at its input end and outputs a set of candidate grasping poses for all objects in the global scene and the stability confidence score corresponding to each grasping pose. The spatial position filtering subunit receives the candidate grasping pose set output by the grasping pose generation subunit, the target object segmentation mask output by the self-calibrating visual positioning module, and the scene RGB-D image. The scene 3D point cloud data generated by the image outputs a set of valid grasping poses whose effective area falls within the effective range of the target object's 3D point cloud. The input of the execution angle filtering subunit receives the set of valid grasping poses output by the spatial position filtering subunit and the system's preset grasping posture angle threshold constraint parameters. The output of the execution angle filtering subunit is a set of compliant grasping poses that meet the reachability and collision-free constraints of the robotic arm. The input of the optimal pose selection subunit receives the set of compliant grasping poses output by the execution angle filtering subunit and the stability confidence scores corresponding to each grasping pose output by the grasping pose generation subunit. The output of the optimal grasping pose position parameters and attitude parameters with the highest confidence after filtering is transmitted to the robot control system.

3. The method for robot grasping generalized language commands in cluttered scenes according to claim 2, characterized in that... The initialization settings in step (1) specifically refer to: (1-1) Load the Doubao-1.5-pro large model, the LangSAM pre-trained visual localization model, and the GraspNet pre-trained crawling network. Configure system-level prompts for the language instruction parsing subunit in the information logic processing module and the semantic verification subunit in the self-correcting visual localization module. The system-level prompts are used to define the task role, output format, and inference range of the Doubao-1.5-pro large model. The Doubao-1.5-pro large model is responsible for language instruction parsing and positional relationship inference. The LangSAM pre-trained visual localization model completes the initial visual localization of the object. The Doubao-1.5-pro large model and the LangSAM pre-trained visual localization model work together to achieve accurate selection of the target object. (1-2) The GraspNet pre-trained grasping network generates a set of candidate grasping poses in the scene, which are used to obtain executable grasping poses after spatial position filtering and execution angle filtering in the subsequent target 3D grasping filtering module, and then to obtain executable grasping postures after 3D filtering. (1-3) Preset the geometric feature parameters of candidate target objects The mapping rules for spatial location relationships, the score fusion weight ratio, and the threshold for the grasping posture angle; among them, The coordinates of the center pixel of the bounding box of the candidate target object in the two-dimensional image plane; The distance is the Euclidean distance between the center point of the bounding box of the candidate target object and the reference object; the calculation formula is shown in formula (1); The relative rotation angle of the candidate target object with respect to the reference object is calculated as shown in formula (2). (1) (2) in, , The coordinates of the center point of the target object. , The coordinates are the center coordinates of the reference object; (1-4) Connect the data transmission interface between the information logic processing module, the self-correcting visual positioning module, and the target 3D grasping and filtering module, and verify the smooth operation of the entire process link; wherein, the information logic processing module outputs structured language parsing results based on the Doubao-1.5-pro large model, the self-correcting visual positioning module outputs target object filtering results based on the LangSAM pre-trained visual positioning model and the Doubao-1.5-pro large model, and the target 3D grasping and filtering module outputs a set of candidate grasping poses based on the GraspNet pre-trained grasping network, and completes the grasping pose filtering according to the final target object segmentation mask; then open the scene image input interface and the natural language grasping command input interface, so that the generalized language command robot grasping system in the cluttered scene is in a state of waiting to receive external scene images and language grasping commands.

4. The method for robot grasping generalized language commands in cluttered scenes according to claim 3, characterized in that... The mapping rules for spatial positional relationships in steps (1-3) specifically refer to: ① When there is no reference object, the two-dimensional image plane coordinate system is used as the judgment benchmark, and the relative position of the candidate target object in the image is determined by the pixel coordinates of the center point of the bounding box; among which, the horizontal pixel coordinates Increasing the pixel value indicates that the candidate object is positioned further to the right in the image, while decreasing it indicates that it is positioned further to the left; vertical pixel coordinates Increasing the value indicates that the candidate target object is lower in the image, while decreasing it indicates that it is higher. For cases where the language command involves "in front" or "behind", the vertical position of the image is used as an approximate criterion. That is, under the preset shooting angle, the candidate target object located at the top of the image is judged to be further back, and the candidate target object located at the bottom of the image is judged to be further forward. ② When there is a reference object, use Euclidean distance. d The size of the distance between the candidate target object and the reference object is determined by the distance of the Euclidean distance. d The smaller the value, the closer the candidate target object is to the reference object; ③ By relative rotation angle r The value determines the orientation between the candidate target object and the reference object; The score fusion weight ratio in step (1) refers to the weighted ratio between the detection confidence Logits output by the LangSAM pre-trained visual localization model and the semantic verification score Score output by the Doubao-1.5-pro large model, which is generally 0.6:0.

4. The method for determining the grasping posture angle threshold specifically includes: determining the reachable posture range of the end effector based on the robotic arm configuration, determining the allowable gripping direction range based on the end effector type, determining the safe approach path range based on the table height or the height of the target object's support surface, and determining the collision constraint range based on the obstacle distribution in the scene; on this basis, through grasping experiments with candidate angle thresholds, the grasping success rate, collision rate, and trajectory execution rate corresponding to each candidate angle threshold are statistically analyzed, and the angle threshold with the best overall performance is selected as the grasping posture angle threshold for actual use.

5. The method for robot grasping generalized language commands in cluttered scenes according to claim 1, characterized in that... In step (2), the Information Logic Processing Module (ILPM) converts the ambiguous language instructions into the geometric feature parameters and position information of the target object. Specifically, it refers to performing two sub-tasks: key information extraction and positional relationship reasoning. Sub-task 1 is used for key information extraction, and task 2 is used for positional relationship reasoning.

6. The method for robot grasping generalized language commands in cluttered scenes according to claim 5, characterized in that... Subtask 1, which involves extracting key information, refers to the process by which the Doubao-1.5-pro large model parses the original natural language crawling instructions under the constraints of the system-level prompt, and extracts the following three core elements: ①Target Object Name: The name of the main object to be captured; ②Reference Object Name: The name of the reference object; ③Relative Position Relationship: A linguistic phrase describing the spatial relationship between a target object and a reference object; If the input language instruction only contains the target object, without a reference object or spatial relationship, fill the Reference ObjectName and Relative Position Relationship fields with None, and finally output the extraction results in a structured template of "target object name, reference object name, position relationship phrase"; Subtask 2 involves positional relationship reasoning, namely: ① Call the LangSAM pre-trained visual localization model loaded in step (1) to perform preliminary identification of potential target objects and reference objects in the scene image to be identified, and obtain the pixel coordinates of the center point of the bounding box of each object in the two-dimensional image plane. ② The Doubao-1.5-pro large model calculates a four-element list of geometric feature parameters for each candidate target object. ,at this time, , The field is fixed as None, only reserved , Coordinate parameters; ③ Based on the mapping rules of the spatial positional relationship predetermined in step (1), the geometric feature parameters are transformed into spatial positional relationships, and finally the geometric feature parameters and positional information of the candidate target object are output.

7. The method for robot grasping generalized language commands in cluttered scenes according to claim 1, characterized in that... Step (3) specifically includes the following: (3-1) Call the LangSAM pre-trained visual localization model. Based on the target object identification information and reference object identification information extracted in step (2), perform preliminary visual detection based on language prompts on the cluttered scene image to obtain at least one candidate target object in the scene. For each candidate target object, output its corresponding bounding box, segmentation mask and detection confidence score Logits. This process is automatically performed by the model. The confidence score Logits value is normalized to eliminate numerical magnitude differences and ensure the effectiveness of subsequent score fusion. (3-2) The image of each candidate target object is cropped into its own image according to the bounding box, and input together with the name of the main object to be captured and the name of the reference object to be captured into the Doubao-1.5-pro large model constrained by the system-level prompt. The Doubao-1.5-pro large model completes the matching degree verification of the image of the candidate target object with the name of the main object to be captured and the name of the reference object extracted in step (2), that is, the secondary verification, and outputs a discrete verification score, which is used to represent the semantic matching degree between the image and the language instruction text. (3-3) To obtain the comprehensive semantic matching evaluation result of the candidate objects, the confidence score Logits output by the LangSAM pre-trained visual localization model in step (3-1) and the validation score Score output by the Doubao-1.5-pro large model in step (3-2) are weighted and fused to calculate the comprehensive score. Final_Score The fusion formula is shown in formula (3): (3) (3-4) The comprehensive score calculated according to step (3-3) Final_Score Perform candidate target object screening.

8. The method for robot grasping generalized language commands in cluttered scenes according to claim 7, characterized in that... The discrete verification score in step (3-2) is calculated by the built-in function of the Doubao-1.5-pro large model. That is, each candidate target object is cropped according to its bounding box to obtain a candidate image, and the candidate image, together with the target object identification information, reference object identification information and spatial relationship information extracted in step (2), is input into the Doubao-1.5-pro large model constrained by the system-level prompt. The Doubao-1.5-pro large model verifies the semantic consistency between the candidate image and the language instruction and outputs the semantic verification score. The semantic verification score is output in a discrete scoring manner: in the initialization stage of step (1), a discrete scoring set is pre-set, and the Doubao-1.5-pro large model selects a score from the discrete scoring set as the semantic verification score based on the semantic consistency verification result; The weight ratio of the weighted fusion in step (3-3) m:n It is the score fusion weight ratio preset in steps (1-3), usually taken as 0.6:0.

4. This weight ratio is the optimal value obtained by testing 2000 crawling tasks on the Roborefit training set; The rules for screening candidate target objects in step (3-4) are as follows: (3-4-1) If the comprehensive score of the candidate objects Final_Score The confidence score Logits satisfies Final_Score ≥Logits indicates that the two verifications, "detection by the LangSAM pre-trained visual localization model and verification by the Doubao-1.5-pro large model", bring a positive gain effect. That is, if the image of the selected target object and the text of the language instruction are highly likely to match, then the candidate target object can be accepted. (3-4-2) If the confidence score Logits of the candidate target object is ≥0.7, it means that the confidence of the preliminary detection result is high enough, and the candidate target object can be directly accepted to improve the system processing efficiency. (3-4-3) If all candidate target objects fail the above screening, the re-detection mechanism will be started automatically. That is, the segmentation mask regions of all unaccepted objects will be blocked and a new scene image will be generated. The LangSAM pre-trained visual localization model will re-perform the detection on the new image, repeating the above steps (3-1) to (3-3) and screening again until the final target object is determined.

9. The method for robot grasping generalized language commands in cluttered scenes according to claim 2, characterized in that... Step (4) specifically includes the following: (4-1) The GraspNet pre-trained grasping network automatically calculates and assigns a confidence score to each grasping posture. The higher the score, the stronger the stability of the grasping posture under ideal conditions, providing a quantitative basis for subsequent optimal selection. (4-2) Extract the 3D point cloud data corresponding to the target object based on the segmentation mask of the candidate target object output by the self-correcting visual positioning module SCGM in step (3); (4-3) The candidate grasping pose set generated by the GraspNet pre-trained grasping network in step (1-2) is traversed one by one, and the spatial correspondence between the effective area of ​​each candidate grasping pose and the 3D point cloud data of the target object extracted in step (4-2) is calculated. Only candidate grasping poses whose effective area falls on the surface of the 3D point cloud of the target object or within its preset neighborhood are retained as valid grasping poses. Candidate grasping poses whose effective area falls on non-target areas are eliminated. Through spatial position filtering, the subsequently retained grasping poses correspond one-to-one with the final target object in 3D space, thereby reducing the risk of mistakenly grasping background objects, adjacent objects or invalid areas in cluttered scenes and improving the consistency between the language command target and the actual grasping target. (4-4) Further perform robot executability filtering on the effective grasping postures initially screened out in step (4-3); (4-5) After the two-layer filtering of spatial position filtering in step (4-3) and angle filtering in step (4-4), the grasping posture with the highest original confidence score calculated by the GraspNet pre-trained grasping network in step (4-1) is selected from the effective grasping postures as the final optimal grasping posture. The posture position parameters and posture attitude parameters corresponding to the final grasping posture are sent to the robot control system, and the robot moves to the target grasping position and performs the grasping action accordingly.

10. The method for robot grasping generalized language commands in cluttered scenes according to claim 9, characterized in that... The non-target regions in step (4-3) include: hollow regions inside the target object without effective point cloud support, background regions outside the target object segmentation mask, point cloud surface regions corresponding to other non-target objects, and regions where effective three-dimensional point cloud data has not been formed due to occlusion, reflection, or lack of depth. In step (4-4), the effective grasping postures are further subjected to robot executability filtering. Specifically, this means: taking the normal direction of the support surface where the target object is located or the opposite direction of gravity as the reference direction, calculating the angle between each grasping posture and this direction, retaining only grasping postures with an angle ≤30°, and eliminating grasping postures that are located on the side of the object, conflict with the robot's movement direction, have a rotation angle that is approximately parallel to the support surface, or are prone to slipping or collision, so that the selected postures completely conform to the motion constraints of the real robotic arm.

Images