A robot grasping target detection method and system fusing vision and language models

Abstract

The application discloses a robot target detection method and system fusing vision and language models, and belongs to the technical field of robot grasping control. The method comprises the following steps: collecting an RGB image, a depth image and a natural language grasping instruction; inputting the instruction into a large language model for multi-level semantic analysis, extracting entity attributes, spatial relationships and task intentions; extracting visual features and language features, dynamically aligning through a hierarchical cross-modal attention mechanism, introducing a spatial relationship bias matrix to enhance the expression of spatial constraints, and constructing a dynamic semantic visual graph; filtering the attributes and spatial constraints of candidate targets based on the semantic graph, and calculating the comprehensive confidence to determine the grasping target. Through deep semantic understanding, fine-grained cross-modal alignment, dynamic semantic reasoning and active interaction mechanism, the application significantly improves the understanding ability of the robot to complex language instructions and the grasping target detection accuracy in an unstructured environment.

Description

Technical Field

[0001] This invention relates to the field of robot grasping control technology, and in particular to a robot grasping target detection method and system that integrates vision and language models. Background Technology

[0002] In recent years, with the rapid development of deep learning and large-scale models, the application of vision-language multimodal fusion technology in the field of robot grasping has become a research hotspot. Target detection, as a core pre-processing step for robots to perform grasping tasks, directly affects the success or failure of the entire operation.

[0003] Currently, robot object detection is mainly developing along two technical routes: one is a purely vision-driven approach. This method takes RGB or RGB-D images as input and uses convolutional neural networks or object detection models (such as the YOLO series and Faster R-CNN) to identify object categories and predict the grasping pose. This type of method performs well in structured environments with clear targets and simple scenes. However, in unstructured environments such as homes and warehouses, purely vision-based methods struggle to distinguish between multiple similar objects and cannot understand the attribute constraints and spatial orientation in user commands. For example, when multiple cups of different colors are placed on a table, existing vision detection models can only output the category label "cup," and cannot determine whether the user intends to point to a "red ceramic cup" or a "blue glass cup."

[0004] Secondly, there are language-based grasping methods that have emerged in recent years. These methods achieve language-guided target localization and grasping decisions by fusing natural language instructions with visual features across modalities. For example, the published patent application CN119526405A proposes a robot grasping detection method based on a visual-language-action multimodal alignment strategy. However, existing language-based methods still have the following significant technical defects: (1) Insufficient language understanding. Existing methods mostly use general text encoders to encode language instructions, which are difficult to handle complex instructions containing spatial relationships (such as "left" and "farthest"), attribute combinations (such as "red ceramic"), and implicit intentions, resulting in coarse parsing granularity; (2) Insufficient depth of cross-modal fusion. The fusion of visual features and language features remains at the level of shallow feature splicing or simple attention weighting, lacking a fine-grained semantic alignment mechanism, making it difficult to achieve accurate mapping of "what is pointed to is what is seen"; (3) Lack of dynamic reasoning ability. Faced with dynamic scene changes, existing methods do not have the ability to adapt and adjust in real time, and cannot actively interact with users to confirm when instructions are ambiguous; (4) Grasping detection and posture generation are separate. The two stages of target detection and grasping posture generation are relatively independent, and there is a lack of task semantics to guide and optimize the grasping posture.

[0005] In summary, existing technologies are insufficient to meet the requirements of precise robot grasping based on complex natural language instructions in unstructured environments. There is an urgent need for a grasping target detection method that can deeply understand language semantics, finely align visual features, and achieve end-to-end joint optimization. Summary of the Invention

[0006] This invention addresses the prominent problems in existing robot grasping target detection technologies, such as insufficient understanding of complex natural language instructions, inadequate depth of vision-language cross-modal fusion, poor adaptability to dynamic scenes, and the relative independence of target detection and grasping posture generation, as well as the lack of task semantic guidance. It provides a robot grasping target detection method and system that integrates vision and language models.

[0007] This invention utilizes a large language model to perform multi-level semantic parsing of natural language commands. It combines a hierarchical cross-modal attention mechanism to achieve fine-grained dynamic alignment of visual and linguistic features, constructing a dynamic semantic visual graph for spatial relationship reasoning and semantic constraint filtering. Furthermore, it leverages task intent embedding to guide the generation and optimization of grasping postures, and introduces an incremental learning mechanism to achieve continuous model evolution. This invention aims to improve the robot's ability to understand complex language commands and the accuracy of target detection in unstructured environments, enabling the robot to accurately and robustly complete grasping tasks based on user-defined natural language commands.

[0008] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a robot-assisted target detection method that integrates visual and language models, comprising the following steps: Step S1: Acquire RGB and depth images of the target scene using an RGB-D depth camera mounted on the robot, and simultaneously acquire natural language grasping commands issued by the user through a voice acquisition module or text input interface; perform noise reduction and color correction preprocessing on the RGB image, perform hole filling and filtering preprocessing on the depth image, and perform speech recognition and text normalization processing on the natural language grasping command. Step S2: Input the preprocessed natural language capture instructions into the pre-trained large language model for multi-level semantic parsing. The multi-level semantic parsing includes: entity recognition and attribute extraction, identifying the target object name and its visual attributes in the instructions; spatial relationship parsing, identifying and quantifying the location, distance, and arrangement relationships in the instructions, and converting them into computable spatial constraints; task intent understanding, parsing the action type and its parameterized description behind the instructions; and semantic embedding generation, generating global instruction embedding, entity attribute embedding, spatial relationship embedding, and task intent embedding based on the parsing results. Step S3: Input the preprocessed RGB image into the visual encoder to extract multi-scale visual feature maps, and input the preprocessed depth image into the point cloud encoding module to extract three-dimensional geometric features; perform depth encoding on the semantic embeddings generated in Step S2 at each level through the language encoder to obtain multi-granularity language feature sequences; achieve dynamic alignment between visual features and language features through a hierarchical cross-modal attention mechanism, the dynamic alignment including three levels: pixel-word alignment, region-phrase alignment, and scene-instruction global alignment. The hierarchical cross-modal attention mechanism introduces a spatial relationship bias matrix constructed based on the spatial relationship parsing results, so that image regions that meet spatial constraints receive higher attention weights; construct a dynamic semantic visual atlas based on the alignment results, the dynamic semantic visual atlas using target object candidate regions as nodes and spatial relationships and attribute relationships as edges; Step S4: Based on the multi-scale visual feature map, a set of candidate regions for the target object is generated through a region proposal network. Each candidate region includes a bounding box, category confidence, and initial grasping quality score. The dynamic semantic visual atlas is used to perform attribute constraint filtering and spatial constraint filtering on each candidate region, selecting candidate targets that satisfy all attribute conditions and spatial constraints in the instruction. The comprehensive confidence of each candidate target is calculated, whereby the comprehensive confidence integrates visual detection confidence, attribute matching degree, spatial constraint satisfaction degree, and intention. Figure 1 The consistency scores are weighted and calculated, and the candidate target with the highest overall confidence is determined as the target to be crawled. When the highest overall confidence is lower than a preset threshold, an active interaction mechanism is triggered. Step S5: Based on the 3D point cloud data of the determined target object, a set of candidate grasping postures is predicted by the grasping posture generation network. Each grasping posture includes grasping position, grasping direction and grasping width. The task intent generated in step S2 is embedded and fused with the candidate grasping posture features. The grasping force control parameters, end effector movement speed and grasping direction constraints are dynamically adjusted according to the action type in the task intent to obtain the optimized grasping posture. Step S6: Convert the optimized grasping posture into robot motion control commands to control the robotic arm to perform grasping actions; determine whether the grasping is successful through force sensors and visual feedback and record the execution results; store successful and unsuccessful grasping cases and their corresponding language commands, scene images and grasping parameters into the experience playback pool, and periodically perform incremental fine-tuning of the vision-language model.

[0009] Furthermore, the attention score matrix for dynamic alignment described in step S3 is calculated according to formula (1): ; in, For visual feature query matrix, For language feature key matrix, For feature dimension, This is the spatial relationship bias matrix constructed based on the spatial relationship analysis results in step S2; the spatial relationship bias matrix Each element value in the algorithm is quantized and assigned a value based on the degree of geometric constraint compliance between the corresponding image region and the spatial description in the instruction. When a region meets the spatial constraint conditions in the instruction, a positive bias is assigned; when it does not meet the constraints, a negative bias or zero bias is assigned.

[0010] Furthermore, the comprehensive confidence level mentioned in step S4 Calculate according to formula (2): ; in, For visual detection confidence, For attribute matching degree, For spatial constraint satisfaction, For the purpose Figure 1 Consistency score, to For adaptive fusion weights, and The adaptive fusion weights to The settings are dynamically adjusted based on the reliability of visual detection in the current scenario and the clarity of various constraints in the language instructions. The adjustment is increased when the attribute constraint descriptions in the instructions are rich. Increase when the spatial relationship is clearly described in the instruction. .

[0011] Furthermore, the spatial relationship parsing in step S2 specifically includes: identifying the orientation relationship, distance relationship, and arrangement relationship description in the instruction, quantifying them into three-dimensional spatial coordinate offsets and sorting logic expressions with the centroid of the target object as the reference system, wherein the three-dimensional spatial coordinate offsets are referenced by the world coordinate system established with the robot's current position as the origin; when there are multiple spatial constraints in the instruction, a spatial constraint hierarchy tree is constructed, and the constraints are filtered layer by layer according to their priority order, wherein the priority of orientation relationship constraints is higher than that of distance relationship constraints, and the priority of distance relationship constraints is higher than that of arrangement relationship constraints.

[0012] Furthermore, the specific construction method of the dynamic semantic visual atlas in step S3 is as follows: each candidate region of an object detected in the scene is taken as a node, and the initial features of each node include the visual feature vector of the corresponding region, the category label, and the three-dimensional spatial position in the world coordinate system; a weighted directed graph is constructed using the relative orientation vector and attribute similarity between objects as the features of the edges; a graph attention network is used to iteratively update the node features, and the spatial constraint information and attribute constraint information in the language instructions are propagated and aggregated in the atlas through the message passing mechanism, so that each node finally obtains an enhanced feature representation that integrates the global scene context and language constraints.

[0013] Furthermore, the hierarchical cross-modal attention mechanism described in step S3 includes three levels: the first level is pixel-word alignment, which calculates the semantic relevance between each pixel position in the visual feature map and each word in the instruction, generating a visual-word attention heatmap for locating the image region corresponding to the attribute description; the second level is region-phrase alignment, which aligns the visual feature map into semantic regions with the attribute phrases and spatial relation phrases in the instruction, realizing the mapping of attribute constraints and spatial constraints in the visual space; the third level is scene-instruction global alignment, which calculates the semantic consistency score between the global visual scene features and the complete instruction embedding, used to perform global verification and confidence evaluation of the alignment results of the first two levels, and the evaluation feedback is passed to the comprehensive confidence calculation in step S4.

[0014] Furthermore, the proactive interaction mechanism described in step S4 specifically includes: when the highest overall confidence level is lower than the first preset threshold, a clarifying question is output through the speech synthesis module or graphical user interface to prompt the user to supplement the explanation of ambiguous pronouns or missing attribute information in the instruction; when the highest overall confidence level is between the first preset threshold and the second preset threshold, a candidate target list is output for the user to select and confirm; after receiving the user's feedback information, the feedback information is associated with the current scene data and encoded, and input into the large language model for secondary semantic parsing; based on the secondary parsing result, steps S3 and S4 are re-executed for target re-detection until the overall confidence level meets the requirements or the number of interaction rounds reaches the upper limit.

[0015] Furthermore, step S5, which involves constraining and refining the candidate grasping posture using task intent embedding, specifically includes: when the task intent embedding represents grasping lightly or fragile objects, reducing the target value of the grasping force control parameter and increasing the force control compliance coefficient; when the task intent embedding represents rapid grasping, increasing the end effector's movement speed and shortening the acceleration time; when the task intent embedding contains directional constraint semantics, constraining the feasible domain of the grasping direction angle within a specified range; and using a grasping quality evaluation network to score the refined grasping posture parameters, selecting the grasping posture with the highest score as the final execution posture.

[0016] Secondly, the present invention provides a robot target detection system that integrates visual and language models for performing the above-described method, the system comprising: The multimodal data acquisition module includes an RGB-D depth camera, a voice acquisition device, and a text input interface, used to acquire RGB images, depth images, and natural language grasping commands; The dynamic semantic parsing module is equipped with a pre-trained large language model, which is used to perform multi-level semantic parsing on natural language crawling instructions and output multi-granular semantic embedding vectors. The cross-modal feature extraction and alignment module is used to extract visual and linguistic features and achieve dynamic alignment of visual and linguistic features according to formula (1); The semantic visual graph construction module is used to construct a dynamic semantic visual graph with object candidate regions as nodes and spatial and attribute relationships as edges. The target reasoning and detection module is used to generate a set of candidate targets based on the region proposal network and calculate the comprehensive confidence score according to formula (2) to determine the target to be captured. The grasping pose generation and optimization module includes a grasping pose generation network and a pose refinement unit, which are used to generate candidate grasping poses and optimize the poses based on task intent embedding. The robot control and execution module is used to convert grasping postures into motion control commands and drive the robot to perform grasping actions. The feedback learning module includes a force sensor interface, a visual feedback unit, and an experience playback unit. It is used to collect contact force data through the force sensor to determine the success or failure of the grasp, verify the in-situ state of the target object after grasping through visual feedback, and store successful and failed samples in the experience playback unit for incremental learning.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: 1. Deep Semantic Understanding and Precise Intent Parsing. This invention utilizes a large language model to perform multi-level semantic parsing of natural language grasping instructions, enabling precise identification of target entities, visual attributes, spatial relationships (location, distance, arrangement), and task intent within the instructions. This overcomes the shortcomings of existing general text encoders in understanding spatial relationships, attribute combinations, and implicit intents, allowing robots to accurately parse the complete semantics of complex instructions such as "Please get me the red cup on the left side of the table."

[0018] 2. Fine-grained cross-modal dynamic alignment. This invention employs a hierarchical cross-modal attention mechanism and introduces a spatial relation bias matrix to achieve visual-linguistic feature alignment at three granularities: pixel-word, region-phrase, and scene-instruction. This significantly improves the depth and accuracy of cross-modal fusion and effectively solves the "what is pointed to is not what is seen" problem caused by shallow feature fusion in existing technologies.

[0019] 3. Dynamic Semantic Reasoning and Adaptive Interaction. The dynamic semantic visual atlas constructed in this invention explicitly models spatial and attribute relationships, and realizes the propagation and aggregation of constraint information through a graph attention network, enabling the robot to have spatial relationship reasoning capabilities; at the same time, it introduces an active interaction mechanism to actively clarify when instructions are ambiguous, improving the system's adaptability to complex and uncertain scenarios.

[0020] 4. Task-driven end-to-end optimization and continuous evolution. This invention embeds task intent into the grasping posture generation process, enabling semantic-level guidance of grasping force control parameters, motion speed, and direction constraints through language commands, overcoming the shortcomings of the separation between detection and grasping; coupled with an incremental learning mechanism, the system can continuously evolve from successful and unsuccessful grasping experiences, constantly improving the grasping success rate and intelligence level. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall process of the robot grasping target detection method that integrates vision and language models according to the present invention. Figure 2 This is a schematic diagram illustrating the workflow of the dynamic semantic parsing module in this invention; Figure 3 This is a schematic diagram of the network architecture of the hierarchical cross-modal attention mechanism in this invention; Figure 4 This is a schematic diagram illustrating the construction and reasoning of the dynamic semantic visual graph in this invention; Figure 5 This is a block diagram of the module structure of the robot grasping target detection system that integrates vision and language models according to the present invention; Figure 6 The figure shows the experimental results comparing the performance of the method of the present invention with other existing methods in the target detection task. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0023] Example 1 This implementation example uses a home service robot executing the command "Please get me the silver metal thermos on the coffee table to the left of the sofa" in a living room setting. Figure 1 This invention describes the complete process.

[0024] like Figure 1 As shown, after the robot starts, it first performs step S1, data acquisition and preprocessing: It acquires color and depth images of the current scene using an Intel RealSense D435iRGB-D camera mounted at the end of the robotic arm, and simultaneously receives voice commands via a microphone array, which are then converted into text by a local speech recognition engine (such as Whisper). Gaussian denoising and histogram equalization are performed on the RGB images, while median filtering and hole filling are applied to the depth image.

[0025] Step S2: Dynamic Semantic Parsing. The standardized instruction text is input into the locally deployed Qwen-7B large language model for multi-level parsing. Model output: entity "thermos cup", attributes "silver", "metal", "with lid", spatial relationship "on the coffee table to the left of the sofa" (parsed as the offset relative to the sofa coordinates), and task intent "take". Simultaneously, a 768-dimensional semantic embedding vector is generated.

[0026] Step S3: Cross-modal feature extraction and alignment: SwinTransformer is used to extract multi-scale feature maps of the RGB image (resolutions of 1 / 4, 1 / 8, and 1 / 16 respectively), and PointNet++ is used to extract point cloud geometric features. Language features are obtained as word-level sequences through a BERT encoder. Figure 3 The hierarchical attention mechanism shown performs pixel-word alignment (heatmap location of the silver area), region-phrase alignment (matching the region on the coffee table), and scene-instruction global alignment (consistency score of 0.92) sequentially. Based on the alignment results, a... Figure 4 The dynamic semantic graph shown includes nodes such as thermos cups, sofas, coffee tables, and remote controls, with edges labeled with relationships such as "left neighbor" and "top".

[0027] Step S4 Target Reasoning and Detection: Faster R-CNN's RPN generates 12 candidate boxes. After filtering by attribute constraints (silver, metal) and spatial constraints (located on the coffee table to the left of the sofa), only the thermos cup candidate box remains. The overall confidence score is calculated to be 0.96 (higher than the threshold of 0.7), and it is determined to be the target to be grasped.

[0028] Step S5: Grasping Pose Generation and Optimization: Extract grabbing candidates from the target point cloud, optimize the grabbing width and force based on the task intent "grab", and generate the end pose.

[0029] Step S6: Grasping Execution and Feedback Learning: The robotic arm moves along the trajectory. Once the force sensor detects that the contact force has reached a preset value, the gripper closes, lifts the robot, and moves it to its chest, accompanied by a voice announcement saying "Received." The entire process is as follows... Figure 1 As shown, the modules work together to complete the task.

[0030] Example 2 This implementation details the specific calculation method of the attention score matrix in cross-modal dynamic alignment, corresponding to... Figure 3 The internal structure of the alignment module is shown.

[0031] In the cross-modal alignment process, the visual query matrix The language key matrix is ​​obtained by linear transformation of the visual feature map, and after flattening the spatial dimensions, the dimensions are N×256 (where N=H×W, and H and W are the height and width of the feature map, respectively). It is obtained by transforming the language feature sequence, with a dimension of T×256. Take 256.

[0032] The attention score matrix is ​​calculated using the following formula:

[0033] Spatial relation bias matrix The construction method is as follows: Based on the spatial description of "on the coffee table to the left of the sofa" parsed in step S2, each pixel or region in the scene is projected to a 3D world coordinate system (with the robot's initial position as the origin), and its relative position to the "sofa" and "coffee table" is calculated. If a region is located within the coffee table plane and to the left of the sofa (with the negative X-axis of the world coordinate system as the left), a positive offset value of +0.5 is assigned; if it is on the coffee table but not to the left, a zero offset is assigned; if it does not meet any of the conditions, a negative offset of -1.0 is assigned. In this specific example, the area where the thermos cup is located satisfies all spatial conditions. The corresponding position in the center is +0.5; the remote control area is on the coffee table but not on the left side, so it's 0; the area of ​​books on the floor is -1.0. and After summing, softmax is performed to obtain attention matrix A, which gives higher attention weights to regions that satisfy spatial constraints. Figure 3 The dashed box in the middle illustrates the process of generating the bias matrix.

[0034] Example 3 This implementation details the calculation method of the overall confidence score and the dynamic adjustment method of the fusion weights, corresponding to... Figure 1 The goal setting stage.

[0035] In one experiment, the instruction was "Take the red square box." The scene contained a red square box, a red circular box, and a blue square box. RPN provided three candidates, and the visual detection confidence score was... The scores were 0.85, 0.80, and 0.82 respectively. Attribute matching degree. Based on color and shape: the red square box is 1.0 (perfect match), the red round box is 0.5 (color match but shape mismatch), and the blue square box is 0.5. Spatial constraints. (This example has no explicit spatial description, so it is uniformly set to 0.5). Figure 1 Desire Based on the "grabbing" action and the graspability of the object: the square box is easier to grasp, scoring 0.9, while the round box scores 0.7.

[0036] The overall confidence level is calculated using the following formula:

[0037] The weights are dynamically adjusted based on the keyword density and number of modifiers for various constraints in the instructions. In the current scenario, the color and shape descriptions are clear (containing two explicit attribute words: "red" and "square"), therefore... Set it to 0.5. Set it to 0.3. Set it to 0.1 (because there are no space constraints). Set it to 0.1. The calculation yields: the red square box... The red circular box has a score of 0.24 + 0.25 + 0.05 + 0.07 = 0.61, and the blue square box has a score of 0.246 + 0.25 + 0.05 + 0.09 = 0.636. The highest score is for the red square box, which is determined as the target. If the command is changed to "the leftmost box", the spatial constraints are clear (including the directional word "left" and the ordinal number "leftmost"), and the system will automatically select the leftmost box. Increase to 0.4, Reduced to 0.2 to highlight spatial weight.

[0038] Example 4 This implementation details the specific implementation of spatial relationship parsing, including the quantification of orientation, distance, and arrangement relationships, and the construction of a constraint hierarchy tree. Figure 2 The semantic parsing process is shown below.

[0039] like Figure 2 As shown, the large language model parses the instruction "Take the third box from the left on the second shelf". First, it identifies "second shelf" as a hierarchical relationship, "left" as a positional relationship, and "third" as a sorting relationship. The system establishes a world coordinate system with the robot's initial position as the origin, and layers the boxes on the shelf according to their Y-axis height in the world coordinate system. It detects 6 boxes on the second shelf; within this layer, they are sorted from left to right according to their X-coordinate in the world coordinate system, with numbers 1-6; "third from the left" means taking the third box after sorting. The positional relationship is quantified as a three-dimensional spatial coordinate offset, using the target object's centroid as the reference frame. In the world coordinate system, the left side is defined as the negative X-axis direction, and the sorting logic expression is: Simultaneously, a spatial constraint hierarchy tree is constructed: the root node is the "box" category, the first level is the hierarchical constraint (second level), the second level is the orientation constraint (left side), and the third level is the ordinal constraint (third). During filtering, all boxes are filtered out first, then the boxes in the second level are kept, then those with X-coordinates less than the median value are kept (left side), and finally, the third box is selected in ascending order of X. This hierarchy tree structure allows constraints to be applied level by level, improving filtering efficiency. Figure 2 The right side of the image shows a graphical representation of the constraint tree. When multiple spatial constraints exist in an instruction, they are filtered layer by layer in descending order of priority, with orientation relationships having higher priority than distance relationships, and distance relationships having higher priority than permutation relationships.

[0040] Example 5 This implementation details the construction of the dynamic semantic visual graph and the updating process of the graph attention network, corresponding to... Figure 4 The diagram structure is shown.

[0041] like Figure 4 As shown, in a kitchen scene, the following object nodes are detected: cup A, cup B, kettle, plate, and dishcloth. Each node's initial features include visual features (2048-dimensional), a one-hot label, and its 3D spatial position (x, y, z) in the world coordinate system. Edge features are relative orientation vectors (e.g., if cup A is to the left and in front of cup B, the vector is (-0.2, 0.1, 0)) and attribute similarities (color similarity, shape similarity, etc.). A GATv2 network is used, with four attention heads per layer and two rounds of message passing. During message passing, the linguistic constraint "red cup" is used as a global condition, propagating the "red" attribute information from color-similar regions to neighboring nodes through node feature updates. After two rounds of updates, the features of cup A (red) incorporate the enhanced signal from the kettle (red), while cup B (blue) does not receive any enhancement. In the final graph, the confidence of red cup A is significantly improved, while the features of other nodes are suppressed. Figure 4 The middle arrow indicates the direction and weight of message passing. The graph adopts a per-frame update strategy, that is, the graph is reconstructed after each frame of image acquisition, and nodes and edges are dynamically added or removed when the target object moves or a new object enters the scene.

[0042] Example 6 This implementation details the specific roles and implementation process of the three levels in the hierarchical cross-modal attention mechanism, corresponding to... Figure 3 The multi-level alignment architecture shown.

[0043] The instruction is "Please pick up the white mug with a handle on the table." First-level pixel-word alignment: The visual feature map (224×224) is flattened into a pixel sequence, and cross-attention is calculated with the word sequence ("with," "handle," "white," "mug") to generate an attention heatmap. In the heatmap, the handle area and the white area of ​​the mug receive high responses, while the desktop background receives very low responses. Second-level region-phrase alignment: The image is divided into 20 semantic regions using superpixel segmentation. The attribute phrases "with handle" and "white" are matched with the region features respectively. The regions with the highest matching scores are labeled as "handle candidate region" and "white candidate region," and their intersection identifies the mug region. Third-level scene-instruction global alignment: The cosine similarity between the global features of the entire scene (average pooling) and the complete instruction embedding ([CLS] vector) is calculated, resulting in a global consistency score of 0.89. This score is used to check for mismatches in the first two layers; in this embodiment, the preset threshold is 0.6 (in actual implementation, it can be adjusted within the range of 0.5 to 0.7 depending on the complexity of the scenario). If the consistency score is lower than this threshold, realignment is triggered. In this example, the score is normal, and the region coordinates obtained from the second layer are passed to subsequent steps. The global consistency score is also passed to the comprehensive confidence calculation in step S4 as a measure of consistency. Figure 1 Consistency score Components of. Figure 3 The input and output of the three levels are marked with different grayscale areas.

[0044] Example 7 This implementation details the specific process of system-user interaction and confirmation when the command is ambiguous or lacks sufficient confidence. Figure 1 The feedback loop in.

[0045] In one scenario, the instruction is "Take that bottle," and there are three bottles in the scene (a mineral water bottle, a glass bottle, and a plastic bottle), with no other attribute constraints. The system calculates the highest overall confidence level. The value is only 0.45 (below the first preset threshold of 0.6). This triggers active interaction: the robot asks via speech synthesis, "Which bottle are you referring to? Please describe its color or location." The user adds, "The green one." The system concatenates the added text with the original instruction to "Take that green bottle," and re-executes steps S2 to S4 of semantic parsing, cross-modal alignment, and object detection. At this point, the green mineral water bottle... Improved to 0.92, successfully captured.

[0046] like If the value is 0.55 (between the first threshold of 0.6 and the second threshold of 0.4), the system outputs a candidate list (displaying cropped images of three bottles on the interface) and asks the user to click to confirm. After the user makes a selection, the system stores the selection result as a positive sample in the experience pool for subsequent optimization. When the user inputs feedback information, the system associates and encodes the feedback information with the current scene data and inputs it into a large language model for secondary semantic parsing. Based on the secondary parsing result, steps S3 and S4 are re-executed for target re-detection until the overall confidence level meets the requirements or the number of interaction rounds reaches the upper limit (in this embodiment, the upper limit is set to 3 times). This interaction process ensures a high success rate for capturing under fuzzy commands. Figure 1 The dashed arrows in the diagram represent interactive loops.

[0047] Example 8 This implementation method corresponds to Figure 1 The posture optimization step (step S5) details the specific rules for constraining and refining the grasping posture using task intent embedding.

[0048] The instruction is "Pick up the egg gently." The task intent embedding is parsed to "pick gently," corresponding to the action type "pick_and_place" with an additional force control compliance requirement. During the gripping posture optimization phase, the system lowers the target value of the gripping force control parameter from the default 20N to 5N, while increasing the force control compliance coefficient from 0.1 to 0.5, allowing the gripper to adaptively deform upon contact. The end effector's movement speed is reduced from 0.5m / s to 0.2m / s, and the acceleration is also reduced accordingly to avoid excessive inertia causing the egg to break. Gripping direction constraint: no direction restriction is parsed; the default is a top-down vertical gripping.

[0049] If the instruction is "grab the plate from the side," the system constrains the grasping direction angle within ±30° of the horizontal direction. If the instruction is "quickly pick up the wrench," the end effector's movement speed is increased to 1.0 m / s and the acceleration time is shortened. After optimization, the 10 generated candidate postures are scored using a grasping quality evaluation network, and the grasping posture with the highest score is selected as the final execution posture. In this embodiment, the gentle grasping posture scores 0.96, higher than other candidates, and is selected as the final execution posture.

[0050] Example 9 This implementation describes a specific deployment example of a complete system in a logistics warehouse, corresponding to... Figure 5 The module structure shown.

[0051] like Figure 5As shown, the system comprises eight modules. The multimodal data acquisition module (including a D435i camera, microphone array, and touchscreen) is mounted on the mobile robot chassis. The dynamic semantic parsing module uses an edge server equipped with an RTX4090 GPU to deploy the LLaMA-3-8B model, with an inference latency of approximately 200ms. The cross-modal feature extraction and alignment module is implemented using PyTorch, and the attention is calculated according to formula (1), with the entire alignment process taking approximately 150ms. The semantic visual graph construction module constructs the graph structure based on NetworkX and uses DGL for GAT updates, with each frame update taking approximately 80ms. The target inference and detection module uses Faster R-CNN accelerated by TensorRT, with a detection frame rate of 30fps, and the comprehensive confidence calculation is performed in real time according to formula (2). The grasping posture generation and optimization module includes a posture generation network based on Contact-GraspNet and a refinement unit based on task embedding. The robot control and execution module communicates with the UR5e robotic arm through ROS2 to send trajectory commands. In the feedback learning module, the force sensor interface collects grasping contact force data at a sampling rate of 1kHz. A successful grasp is determined when the contact force remains stable and exceeds a preset threshold within 2 seconds after the gripper closes. The visual feedback unit verifies the grasping status by comparing the position changes of the target object's bounding box before and after grasping. LoRA incremental fine-tuning is triggered every 100 successful and failed samples. The modules interact via shared memory and a ZeroMQ message queue. With the hardware configuration used in this embodiment, the overall system latency is controlled within 1.5 seconds, meeting real-time requirements. Figure 5 Arrows between modules indicate the direction of data flow.

[0052] Example 10 This embodiment provides a specific storage medium implementation method.

[0053] The computer program code of the method described in this invention (including the complete logic of steps S1 to S6, and the calculation logic of formulas (1) and (2)) is written in a mixture of Python and C++, compiled into an executable file, and stored in a 64GB microSD card (format ext4). The storage medium also includes a configuration file for setting parameters such as the preset threshold θ, the initial values ​​of the fusion weights λ1 to λ4, and the incremental learning period (per 100 samples). When the robot is deployed, the microSD card is inserted into the card reader of the robot's main control board, and the main control board loads and executes the program. The storage medium can also be a solid-state drive, a USB flash drive, or cloud storage. As long as the program stored on it is executed by the processor, the method of this invention can be implemented. In this embodiment, the storage medium also includes model weight files (large language model, visual encoder, grasping and generating network, etc.), with a total size of about 12GB, which are stored after compression. At the first startup, the main control board loads the model into the video memory. During subsequent execution, new empirical data is read from the storage medium for incremental fine-tuning. The fine-tuned new weights are also stored back into the storage medium to achieve closed-loop update.

[0054] Example 11 This embodiment verifies the effectiveness of the method of the present invention in the target detection task through comparative experiments. The experimental results are as follows: Figure 6 As shown.

[0055] Experimental setup: The experiment was conducted on a self-constructed robot grasping dataset, which contains 24,000 RGB-D images of 12 common household and office objects (cups, bowls, bottles, boxes, jars, books, tools, fruits, tableware, stationery, toys, and electronic products). Each image is accompanied by corresponding natural language grasping instructions (each object contains an average of 3 different descriptive instructions, covering attribute descriptions, spatial descriptions, and composite descriptions), as well as manually annotated bounding boxes of the grasping targets. The dataset was divided into training, validation, and test sets in a 7:1.5:1.5 ratio.

[0056] Choose from the following four methods for comparison: Method A (the method of this invention): The method of this invention is adopted, using Qwen-7B as a large language model and Swin-Tiny as a visual encoder.

[0057] Method B (pure visual detection method): YOLOv8 is used for object detection, combined with GG-CNN for grasping pose prediction. No verbal instructions are input, and the object with the highest confidence is selected as the grasping target based solely on the visual detection results.

[0058] Method C (standard multimodal fusion method): CLIP (ViT-B / 16) is used as a vision-language joint encoder. Image features and text features are fused by simple dot product attention and then input into the crawling and detection network. No large language model is used for semantic parsing, and no spatial relation bias matrix or dynamic semantic graph is used.

[0059] Method D (existing patent method): Reproduce the method described in CN119526405A, and perform grasping detection based on a visual language action multimodal alignment strategy.

[0060] The evaluation metric is the target detection accuracy (Top-1 Accuracy), which is the proportion of samples in which the target output by the model matches the manually labeled target to the total number of test samples.

[0061] Experimental results: like Figure 6 As shown, the horizontal axis represents the amount of training data (unit: thousands), increasing from 0.5 thousand to 24,000 (i.e., the complete dataset); the vertical axis represents the target detection accuracy (percentage).

[0062] The experimental results show that: (1) On the complete dataset (24,000 images), the accuracy of method A reached 94.7%, method D 83.2%, method C 76.8%, and method B 62.3%. The method of this invention improves the accuracy by 11.5 percentage points compared with the existing patent method (CN119526405A), by 17.9 percentage points compared with the standard multimodal fusion method, and by 32.4 percentage points compared with the pure vision method.

[0063] (2) Under small sample conditions (training data of 0.5 thousand images), the accuracy of method A is 68.5%, while that of method B is only 35.2%, method C is 42.1%, and method D is 51.3%. The method of this invention still maintains a high accuracy under small sample conditions, thanks to the semantic understanding advantage brought by the pre-training knowledge of the large language model and the structured prior introduced by the spatial relation bias matrix.

[0064] (3) In terms of convergence speed, Method A achieves an accuracy of over 90% when the training data reaches 8,000 images, while Method D requires approximately 18,000 images to reach the same level, and Methods C and B still do not reach 90% with 24,000 images. The convergence speed of the method of this invention is significantly better than that of existing methods, indicating that this invention has a clear advantage in data utilization efficiency.

[0065] (4) On a test subset containing descriptions of complex spatial relationships (such as “the red box on the left of the second shelf”), the accuracy of method A was 91.2%, method D was 68.7%, method C was 55.4%, and method B was 41.8%. This result shows that the structured spatial reasoning capability introduced by the present invention through the spatial relationship bias matrix and dynamic semantic visual graph significantly improves the understanding and positioning accuracy of spatial relationship descriptions.

[0066] (5) On a test subset containing ambiguous references (such as using only "that bottle" to describe a scenario where there are multiple bottles), Method A improved its accuracy from 58.3% in the first inference to 89.6% after interaction through an active interaction mechanism, while other comparative methods, which lacked interactive capabilities, all had accuracy rates below 60%. This result verifies the effectiveness of the active interaction mechanism of this invention in scenarios with ambiguous instructions.

[0067] The experimental results above demonstrate that this invention significantly improves the robot's ability to understand complex natural language instructions and the accuracy of target detection in unstructured environments through the synergistic effects of multi-level semantic parsing of a large language model, hierarchical cross-modal attention dynamic alignment, spatial relation bias matrix, dynamic semantic visual graph reasoning, and active interaction mechanism.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot-based target detection method integrating visual and language models, characterized in that, Includes the following steps: Step S1: Acquire RGB and depth images of the target scene using an RGB-D depth camera mounted on the robot, and simultaneously acquire natural language grasping commands issued by the user through a voice acquisition module or text input interface; perform noise reduction and color correction preprocessing on the RGB image, perform hole filling and filtering preprocessing on the depth image, and perform speech recognition and text normalization processing on the natural language grasping command. Step S2: Input the preprocessed natural language capture instructions into the pre-trained large language model for multi-level semantic parsing. The multi-level semantic parsing includes: entity recognition and attribute extraction, identifying the target object name and its visual attributes in the instructions; spatial relationship parsing, identifying and quantifying the location, distance, and arrangement relationships in the instructions, and converting them into computable spatial constraints; task intent understanding, parsing the action type and its parameterized description behind the instructions; and semantic embedding generation, generating global instruction embedding, entity attribute embedding, spatial relationship embedding, and task intent embedding based on the parsing results. Step S3: Input the preprocessed RGB image into the visual encoder to extract multi-scale visual feature maps, and input the preprocessed depth image into the point cloud encoding module to extract three-dimensional geometric features; perform depth encoding on the semantic embeddings generated in Step S2 at each level through the language encoder to obtain multi-granularity language feature sequences; achieve dynamic alignment between visual features and language features through a hierarchical cross-modal attention mechanism, the dynamic alignment including three levels: pixel-word alignment, region-phrase alignment, and scene-instruction global alignment. The hierarchical cross-modal attention mechanism introduces a spatial relationship bias matrix constructed based on the spatial relationship parsing results, so that image regions that meet spatial constraints receive higher attention weights; construct a dynamic semantic visual atlas based on the alignment results, the dynamic semantic visual atlas using target object candidate regions as nodes and spatial relationships and attribute relationships as edges; Step S4: Based on the multi-scale visual feature map, a set of candidate regions for target objects is generated through a region proposal network. Each candidate region includes a bounding box, category confidence, and initial grasping quality score. The dynamic semantic visual atlas is used to perform attribute constraint filtering and spatial constraint filtering on each candidate region to select candidate targets that meet all attribute conditions and spatial constraint conditions in the instruction. The comprehensive confidence of each candidate target is calculated. The comprehensive confidence is calculated by weighting the visual detection confidence, attribute matching degree, spatial constraint satisfaction degree, and intent consistency score. The candidate target with the highest comprehensive confidence is determined as the grasping target. When the highest comprehensive confidence is lower than a preset threshold, an active interaction mechanism is triggered. Step S5: Based on the 3D point cloud data of the determined target object, a set of candidate grasping postures is predicted by the grasping posture generation network. Each grasping posture includes grasping position, grasping direction and grasping width. The task intent generated in step S2 is embedded and fused with the candidate grasping posture features. The grasping force control parameters, end effector movement speed and grasping direction constraints are dynamically adjusted according to the action type in the task intent to obtain the optimized grasping posture. Step S6: Convert the optimized grasping posture into robot motion control commands to control the robotic arm to perform grasping actions; determine whether the grasping is successful through force sensors and visual feedback and record the execution results; store successful and unsuccessful grasping cases and their corresponding language commands, scene images and grasping parameters into the experience playback pool, and periodically perform incremental fine-tuning of the vision-language model.

2. The method according to claim 1, characterized in that, The attention score matrix for dynamic alignment mentioned in step S3 is calculated according to formula (1): ; in, For visual feature query matrix, For language feature key matrix, For feature dimension, This is the spatial relationship bias matrix constructed based on the spatial relationship analysis results in step S2; the spatial relationship bias matrix Each element value in the algorithm is quantized and assigned a value based on the degree of geometric constraint compliance between the corresponding image region and the spatial description in the instruction. When a region meets the spatial constraint conditions in the instruction, a positive bias is assigned; when it does not meet the constraints, a negative bias or zero bias is assigned.

3. The method according to claim 1, characterized in that, The comprehensive confidence level mentioned in step S4 Calculate according to formula (2): ; in, For visual detection confidence, For attribute matching degree, For spatial constraint satisfaction, To score for consistency of intent, to For adaptive fusion weights, and The adaptive fusion weights to The settings are dynamically adjusted based on the reliability of visual detection in the current scenario and the clarity of various constraints in the language instructions. The adjustment is increased when the attribute constraint descriptions in the instructions are rich. Increase when the spatial relationship is clearly described in the instruction. .

4. The method according to claim 1, characterized in that, The spatial relationship parsing in step S2 specifically includes: identifying the orientation relationship, distance relationship, and arrangement relationship description in the instruction, quantifying them into three-dimensional spatial coordinate offsets and sorting logic expressions with the centroid of the target object as the reference system, wherein the three-dimensional spatial coordinate offsets are based on the world coordinate system established with the robot's current position as the origin; when there are multiple spatial constraints in the instruction, a spatial constraint hierarchy tree is constructed, and the constraints are filtered layer by layer according to their priority order, wherein the priority of orientation relationship constraints is higher than that of distance relationship constraints, and the priority of distance relationship constraints is higher than that of arrangement relationship constraints.

5. The method according to claim 1, characterized in that, The specific construction method of the dynamic semantic visual atlas in step S3 is as follows: each candidate region of an object detected in the scene is taken as a node, and the initial features of each node include the visual feature vector of the corresponding region, the category label, and the three-dimensional spatial position in the world coordinate system; the relative orientation vector and attribute similarity between objects are used as the features of the edges to construct a weighted directed graph. A graph attention network is used to iteratively update node features. Spatial and attribute constraints in language instructions are propagated and aggregated in the graph through a message passing mechanism, so that each node finally obtains an enhanced feature representation that integrates global scene context and language constraints.

6. The method according to claim 1, characterized in that, The hierarchical cross-modal attention mechanism described in step S3 includes three levels: the first level is pixel-word alignment, which calculates the semantic correlation between each pixel position in the visual feature map and each word in the instruction, and generates a visual-word attention heatmap, which is used to locate the image region corresponding to the attribute description; The second level is region-phrase alignment, which aligns the visual feature map into semantic regions with the attribute phrases and spatial relation phrases in the instructions, thereby realizing the mapping of attribute constraints and spatial constraints in visual space. The third level is scene-instruction global alignment, which calculates the semantic consistency score between global visual scene features and complete instruction embedding. This score is used to perform global verification and confidence evaluation of the alignment results of the first two levels, and the evaluation feedback is passed to the comprehensive confidence calculation in step S4.

7. The method according to claim 1, characterized in that, The proactive interaction mechanism described in step S4 specifically includes: when the highest overall confidence level is lower than the first preset threshold, a clarifying question is output through the speech synthesis module or graphical user interface to prompt the user to supplement the explanation of ambiguous pronouns or missing attribute information in the instruction; when the highest overall confidence level is between the first preset threshold and the second preset threshold, a candidate target list is output for the user to select and confirm; after receiving the user's feedback information, the feedback information is associated with the current scene data and encoded, and input into the large language model for secondary semantic parsing; based on the secondary parsing result, steps S3 and S4 are re-executed for target re-detection until the overall confidence level meets the requirements or the number of interaction rounds reaches the upper limit.

8. The method according to claim 1, characterized in that, Step S5, which involves constraining and refining candidate grasping postures using task intent embedding, specifically includes: when the task intent embedding represents grasping lightly or fragile objects, reducing the target value of the grasping force control parameter and increasing the force control compliance coefficient; when the task intent embedding represents rapid grasping, increasing the end effector's movement speed and shortening the acceleration time; when the task intent embedding contains directional constraint semantics, constraining the feasible domain of the grasping direction angle within a specified range; and using a grasping quality evaluation network to score the refined grasping posture parameters, selecting the grasping posture with the highest score as the final execution posture.

9. A robot-based target detection system integrating visual and language models, characterized in that, The system for performing the method as described in any one of claims 1 to 8 includes: The multimodal data acquisition module includes an RGB-D depth camera, a voice acquisition device, and a text input interface, used to acquire RGB images, depth images, and natural language grasping commands; The dynamic semantic parsing module is equipped with a pre-trained large language model, which is used to perform multi-level semantic parsing on natural language crawling instructions and output multi-granular semantic embedding vectors. The cross-modal feature extraction and alignment module is used to extract visual and linguistic features and achieve dynamic alignment of visual and linguistic features according to formula (1); The semantic visual graph construction module is used to construct a dynamic semantic visual graph with object candidate regions as nodes and spatial and attribute relationships as edges. The target reasoning and detection module is used to generate a set of candidate targets based on the region proposal network and calculate the comprehensive confidence score according to formula (2) to determine the target to be captured. The grasping pose generation and optimization module includes a grasping pose generation network and a pose refinement unit, which are used to generate candidate grasping poses and optimize the poses based on task intent embedding. The robot control and execution module is used to convert grasping postures into motion control commands and drive the robot to perform grasping actions. The feedback learning module includes a force sensor interface, a visual feedback unit, and an experience playback unit. It is used to collect contact force data through the force sensor to determine the success or failure of the grasp, verify the in-situ state of the target object after grasping through visual feedback, and store successful and failed samples in the experience playback unit for incremental learning.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images