Target positioning and capturing method and system based on multi-modal semantics

Abstract

The invention discloses a target positioning and capturing method and system based on multi-modal semanteme, and relates to the technical field of image recognition and mechanical control, and the method comprises the steps: obtaining a two-dimensional image and a natural language interaction instruction; visual-language feature alignment processing is carried out, if semantic ambiguity exists in a natural language interaction instruction in the alignment processing process, a reverse question statement is generated based on a generative interaction mechanism, and a unique target is determined; collecting a multi-view two-dimensional image of a target, and fusing to obtain a three-dimensional point cloud model; inputting a spatial pose generation network to obtain candidate six-degree-of-freedom spatial poses; performing semantic common sense filtering and geometric interference filtering in sequence; calculating a posture score of the filtered candidate six-degree-of-freedom space posture, and selecting an optimal target three-dimensional posture; generating a grabbing operation parameter sequence; and executing the grabbing operation parameter sequence. According to the method provided by the invention, the problem of unsuccessful grabbing caused by semantic ambiguity and shielding is solved.

Description

Technical Field

[0001] This application relates to the fields of image recognition and mechanical control technology, and in particular to a target localization and grasping method based on image and natural semantic analysis. Background Technology

[0002] In recent years, with the rapid development of computer vision and embodied intelligence technologies, 3D target pose estimation techniques based on multimodal understanding have become a research hotspot. However, in complex unstructured scenes with targets of various shapes and attributes, existing visual perception and 3D modeling algorithms still face many challenges.

[0003] First, at the semantic understanding level, existing vision systems lack sufficient depth in understanding natural language commands, making it difficult to handle implicit or ambiguous instructions commonly used by users. Furthermore, when multiple similar objects exist in a visual scene, traditional target localization algorithms struggle to eliminate semantic ambiguity with a single command, leading to a significant decrease in target localization accuracy. Second, in terms of 3D perception modeling, most existing methods rely on single-view RGB-D (red, green, blue, and depth) images. Due to severe occlusion and geometric information asymmetry in single-view images, the generated 3D point cloud models are incomplete and lack precision. This results in subsequent 6-DoF pose estimation failing to meet the accuracy and robustness requirements of complex application scenarios, ultimately leading to unsuccessful target grasping. Summary of the Invention

[0004] This application provides a target localization and grasping method and system based on multimodal semantics to solve the problems in the prior art that it is difficult to eliminate semantic ambiguity with a single command and that the 3D point cloud model is incomplete due to single-view images, which ultimately leads to unsuccessful grasping.

[0005] On the one hand, embodiments of this application provide a target localization and capture method based on multimodal semantics, including: Acquire two-dimensional images and natural language interaction commands input by the user; Visual-linguistic feature alignment is performed on 2D images and natural language interaction instructions. During the alignment process, the confidence and number of candidate targets are detected. If there is semantic ambiguity in the natural language interaction instructions, a rhetorical question is generated based on the generative interaction mechanism. The user's natural language feedback is received and parsed, and the semantic labels of the candidate targets are updated until there is no ambiguity in the natural language interaction instructions, and a unique target is determined. Acquire multi-view two-dimensional images of the target; By fusing multi-view 2D images, a complete 3D point cloud model of the target is obtained; By inputting a 3D point cloud model into a deep learning-based spatial pose generation network, multiple candidate six-DOF spatial poses are obtained. Multiple candidate six-DOF spatial poses are sequentially subjected to semantic common sense filtering and geometric interference filtering; Calculate the attitude score for each filtered candidate six-DOF spatial pose, and select the candidate six-DOF spatial pose with the highest attitude score as the optimal target three-dimensional pose; Generate a sequence of grasping operation parameters corresponding to the optimal target 3D pose; The sequence of parameters for the grabbing operation is executed to grab the target.

[0006] On the other hand, embodiments of this application also provide a target localization and capture system based on multimodal semantics, including: The data input layer is used to acquire two-dimensional images and natural language interaction commands input by the user; The multimodal computing core includes: The semantic understanding engine is used to perform visual-linguistic feature alignment processing on two-dimensional images and natural language interaction instructions. During the alignment process, the confidence and number of candidate targets are detected. If there is semantic ambiguity in the natural language interaction instructions, a rhetorical question is generated based on the generative interaction mechanism. The user's natural language feedback is received and parsed, and the semantic labels of candidate targets are updated until there is no ambiguity in the natural language interaction instructions, thus determining a unique target. The spatial computing engine is used to acquire multi-view 2D images of the target collected by the data input layer, fuse the multi-view 2D images to obtain a complete 3D point cloud model of the target; input the 3D point cloud model into a deep learning-based spatial pose generation network to obtain multiple candidate 6DOF spatial poses; pass the multiple candidate 6DOF spatial poses through semantic common sense filtering and geometric interference filtering in sequence; calculate the pose score of each filtered candidate 6DOF spatial pose, and select the candidate 6DOF spatial pose with the highest pose score as the optimal target 3D pose; The results output layer is used to generate a sequence of grasping operation parameters corresponding to the optimal target 3D pose. The action execution module is used to execute the sequence of grabbing operation parameters to achieve the grabbing of the target.

[0007] The target localization and capture method and system based on multimodal semantics in this application have the following advantages: (1) It solves the problem of semantic ambiguity and improves the accuracy of positioning. The introduction of the vision-language big model can not only handle explicit instructions, but also understand implicit operation intentions; through a multi-round natural language interaction mechanism, it actively eliminates the semantic ambiguity caused by object attributes (color, type, position) in the scene, ensuring the uniqueness and accuracy of target positioning.

[0008] (2) The occlusion problem was overcome and the integrity of the 3D modeling was improved. The multi-view RGB-D image and point cloud fusion strategy was adopted, which effectively solved the problem of incomplete geometric information caused by the lack of point cloud in single view. A more complete and high-precision 3D point cloud model of the target object was constructed, which provided a reliable data foundation for subsequent six-degree-of-freedom pose estimation.

[0009] (3) The pose evaluation mechanism was optimized, and the robustness of the estimation was improved. A hierarchical evaluation mechanism of semantic common sense filtering and three-dimensional geometric constraint analysis was adopted. Semantic filtering uses the common sense knowledge of the large model for initial screening; geometric constraint analysis uses a three-dimensional bounding box model to perform spatial interference detection, filtering out poses with potential interference risks, which greatly improves the accuracy of the final output pose parameters and the robustness in complex scenes, thereby improving the success rate of grasping. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a target localization and capture method based on multimodal semantics provided in this application embodiment.

[0012] Figure 2 This is a composition diagram of a target localization and capture system based on multimodal semantics provided in an embodiment of this application.

[0013] Figure 3 This is a general framework diagram of a target localization and capture method based on multimodal semantics provided in the embodiments of this application.

[0014] Figure 4 This is a schematic diagram illustrating an interactive strategy generation method for target localization and capture based on multimodal semantics, provided in an embodiment of this application.

[0015] Figure 5 This is a schematic diagram of a grasping posture evaluation method based on multimodal semantics for target localization and grasping, provided in an embodiment of this application.

[0016] Figure 6 This is a simplified experimental scenario diagram illustrating a target localization and capture method based on multimodal semantics, provided in an embodiment of this application.

[0017] Figure 7 This is a schematic diagram of a complex experimental scenario for a target localization and grasping method based on multimodal semantics provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Figure 1 A flowchart illustrating a target localization and crawling method based on multimodal semantics, provided in an embodiment of this application. This application provides a target localization and crawling method based on multimodal semantics, including: S100 acquires two-dimensional images and natural language interaction commands input by the user.

[0020] For example, a two-dimensional image containing the target in the current scene can be acquired through a visual acquisition device; specifically, this two-dimensional image is an RGB-D image. Natural language interaction commands can be input in the form of voice, text, etc.

[0021] S110: Perform visual-linguistic feature alignment processing on the two-dimensional image and natural language interaction instructions. During the alignment process, detect the confidence and number of candidate targets. If there is semantic ambiguity in the natural language interaction instructions, generate a rhetorical question based on the generative interaction mechanism, receive and parse the user's natural language feedback, update the semantic labels of candidate targets, until there is no ambiguity in the natural language interaction instructions, and determine the unique target.

[0022] For example, embodiments of this application utilize a pre-trained visual language large model for alignment processing, semantic ambiguity detection, and generative interaction. The visual language large model employs GPT-4o (Generative Pre-trained Transformer 4o). GPT-4o is designed with a single Transformer architecture, unifying data from different modalities such as text, images, and audio into a single neural network for processing. The core of this architecture is the Transformer, which processes the input sequence data through a self-attention mechanism.

[0023] Specifically, alignment processing involves performing high-dimensional deep alignment between the features of the input two-dimensional image and the semantic features of the natural language interaction instructions.

[0024] Semantic ambiguity in natural language interaction commands occurs when multiple candidate targets meet the requirements of the natural language interaction command. In this case, GPT-4o will be used to generate targeted rhetorical questions, such as "Do you mean the red apple on the left or the green apple on the right?" After receiving the rhetorical question, the user will input their choice by voice or text. The system will then receive and parse the user's natural language feedback, dynamically update the semantic tags of the targets, until a unique target ID and its corresponding two-dimensional pixel-level mask are locked.

[0025] As can be seen, in this embodiment, GPT-4o takes a visual scene and a natural language command g as input. Based on the multimodal features of text and images, combined with command semantics and task context, it ensures that a unique target object can be accurately located through one or more rounds of language interaction with the user. After the target is locked, the system uses the semantic segmentation network LANG-SAM to perform pixel-level segmentation of the target, obtaining the target's bounding box and two-dimensional pixel-level mask.

[0026] LANG-SAM uses an image encoder (typically a visual transformer, ViT) to extract image embeddings as the basis for mask prediction. The model also includes a cue encoder that encodes various types of input cues, such as point coordinates, bounding boxes, and low-resolution mask inputs. These encoded cues, along with the image embeddings, are then fed into a mask decoder to generate the final object mask.

[0027] LANG-SAM is designed to handle a variety of cues, including: Mask: can provide a coarse, low-resolution binary mask as initial input to guide the model.

[0028] Point: Users can enter [x, y] coordinates and their type (foreground or background) to help define the object's boundaries.

[0029] Box: You can specify the bounding box using coordinates [x1, y1, x2, y2] to indicate the position and size of the model object.

[0030] Text: Text hints can also be used to provide additional context or to specify objects of interest.

[0031] S120 acquires multi-view two-dimensional images of the target.

[0032] For example, after a two-dimensional pixel-level mask of the target, multi-view two-dimensional images corresponding to the position of the two-dimensional pixel-level mask will be acquired, and RGB-D images of at least four views of the target (front, back, left, and right) will be acquired.

[0033] S130 fuses multi-view two-dimensional images to obtain a complete three-dimensional point cloud model of the target.

[0034] For example, this application preferably employs the Iterative Closest Point (ICP) algorithm to integrate local point cloud data from multi-view 2D images into the world coordinate system. This step effectively compensates for the loss of geometric information caused by object self-occlusion in a single viewpoint, generating a high-fidelity complete 3D point cloud model of the target.

[0035] S140 inputs the 3D point cloud model into a deep learning-based spatial pose generation network to obtain multiple candidate six-DOF spatial poses.

[0036] For example, in the embodiments of this application, the spatial pose generation network based on deep learning preferably uses GraspNet or a similar 6-DoF generation network. The candidate six-DOF spatial pose output by the network includes position coordinates (x, y, z) and pose direction (roll, pitch, yaw).

[0037] GraspNet is a deep learning model specifically designed for predicting 6-DOF (6 Degrees of Freedom) grasping poses. It plays a crucial role in robotic grasping by simulating human grasping behavior to select the optimal grasping pose. The GraspNet model uses a deep neural network architecture and integrates multiple data sources, such as visual images and physical properties of objects. By learning from this data, it can output grasping strategies for different objects.

[0038] The network architecture of this model typically includes a convolutional neural network (CNN) as the main feature extraction component, combined with a recurrent neural network (RNN) or attention mechanism to process sequential data, ensuring the coherence and adaptability of the grasping action. For complex 3D spatial grasping, GraspNet may also integrate additional network components, such as a transformer architecture, to handle long-range dependencies and complex 3D spatial information.

[0039] S150 sequentially applies semantic common sense filtering and geometric interference filtering to multiple candidate six-DOF spatial poses.

[0040] For example, semantic commonsense filtering includes: using a commonsense knowledge base to eliminate candidate six-DOF spatial poses that do not conform to the object's operational logic; for example, for fruit objects, eliminating unstable poses that only contact the edge of the fruit stem. The commonsense knowledge base is set in the visual language large model.

[0041] Specifically, methods for eliminating candidate six-DOF spatial poses using a common-sense knowledge base include: The initially generated candidate six-DOF spatial poses and target category information are input into the visual language large model; The visual language big model generates corresponding filtering criteria based on a common sense knowledge base. Candidate six-DOF spatial poses that do not conform to the object operation logic are eliminated according to the filtering criteria.

[0042] Furthermore, geometric interference filtering includes: constructing a 3D bounding box geometric model of the gripper, calculating the spatial overlap between the 3D bounding box geometric model and the scene background point cloud acquired by the built-in depth camera in each candidate six-DOF spatial pose, and removing candidate six-DOF spatial poses with spatial overlap exceeding a threshold.

[0043] Specifically, methods for calculating the spatial overlap between the 3D bounding box geometry model and the scene background point cloud include: The scene background point cloud is transformed from the world coordinate system to the local coordinate system of the gripper through coordinate transformation; In the local coordinate system of the gripper, the axis-aligned bounding box determination algorithm is used to detect the number of points in the 3D bounding box geometry that belong to the scene background point cloud, i.e., the spatial overlap.

[0044] S160, calculate the attitude score of each filtered candidate six-DOF spatial pose, and select the candidate six-DOF spatial pose with the highest attitude score as the optimal target three-dimensional pose.

[0045] S170 generates a sequence of grasping operation parameters corresponding to the optimal target 3D pose.

[0046] S180 executes the grabbing operation parameter sequence to grab the target.

[0047] This application also provides a target localization and capture system based on multimodal semantics, such as... Figure 2 As shown, the system includes: The data input layer is used to acquire two-dimensional images and natural language interaction commands input by the user; The multimodal computing core includes: The semantic understanding engine is used to perform visual-linguistic feature alignment processing on two-dimensional images and natural language interaction instructions. During the alignment process, the confidence and number of candidate targets are detected. If there is semantic ambiguity in the natural language interaction instructions, a rhetorical question is generated based on the generative interaction mechanism. The user's natural language feedback is received and parsed, and the semantic labels of candidate targets are updated until there is no ambiguity in the natural language interaction instructions, thus determining a unique target. The spatial computing engine is used to acquire multi-view 2D images of the target collected by the data input layer, fuse the multi-view 2D images to obtain a complete 3D point cloud model of the target; input the 3D point cloud model into a deep learning-based spatial pose generation network to obtain multiple candidate 6DOF spatial poses; pass the multiple candidate 6DOF spatial poses through semantic common sense filtering and geometric interference filtering in sequence; calculate the pose score of each filtered candidate 6DOF spatial pose, and select the candidate 6DOF spatial pose with the highest pose score as the optimal target 3D pose; The results output layer is used to generate a sequence of grasping operation parameters corresponding to the optimal target 3D pose. The action execution module is used to execute the sequence of grabbing operation parameters to achieve the grabbing of the target.

[0048] For example, the data input layer, acting as the system's perception interface, is responsible for receiving the raw data stream. Specifically, it includes: Command Interface: Receives natural language interaction commands from users via voice or text input. Vision Interface: Connects to the RGB-D sensor array to receive real-time 2D images and depth point cloud flows of the scene.

[0049] The multimodal computing core employs a dual-engine architecture that parallelizes semantic understanding and spatial computation. It incorporates a pre-trained large-scale vision-language model for feature alignment and intent reasoning, and includes an interactive disambiguation controller to monitor semantic ambiguity. When ambiguity is detected, the controller takes over the dialogue flow, generates a query, and updates the semantic labels based on user feedback, thus solving the "long-tail" recognition challenge in complex, unstructured scenarios.

[0050] The spatial computing engine comprises a 3D point cloud reconstruction module and a pose generation and geometry analyzer. The 3D point cloud reconstruction module receives information about the locked target region and schedules a multi-view registration algorithm to generate a high-precision 3D geometric model, overcoming the limitations of single-view occlusion. The pose generation and geometry analyzer is a cascaded computational unit. It first uses a deep network to generate a high-dimensional pose space, and then uses virtual geometric bounding box technology to perform rigorous spatial interference detection. This analyzer ensures that the pose output by the system is "collision-free" and "executable" in physical space.

[0051] The output layer transforms the data processed by the computation core into standardized control signals or visualization data, and outputs geometrically optimized coordinate matrices and quaternions for use by downstream execution modules or rendering engines; it can also render the bounding box of the target, recommended operation center point, or interference warning area on the user interface.

[0052] For example, such as Figure 3 , 4As shown, the multimodal computing core proposes a multimodal human-computer interaction system architecture that integrates visual perception and natural language understanding capabilities, enabling intuitive and efficient object manipulation tasks. The core consists of three parts: 1. Visual-language processing; 2. Interactive target selection; 3. Dynamic, context-aware strategy generation and execution. A layered design principle is adopted, with standardized interfaces facilitating data exchange between components to ensure system scalability and robustness. The system architecture can be represented as a multimodal mapping function. :

[0053] in This represents visual input, i.e., a two-dimensional image. This represents the user's initial natural language command. This represents a sequence of grabbing operation parameters that may contain multiple steps and ultimately lead to a target.

[0054] First, in vision-language processing, the large vision-language model supports vision. and instructions Initial analysis is performed to identify potential candidate targets. If multiple candidate targets lead to ambiguity, an interactive disambiguation controller will be activated, interacting with the user through natural language. To clearly define the user's ultimate intent and the target to be crawled, this is called... .

[0055] In the only goal Once confirmed, strategy generation does not directly plan the pair. Instead, GPT-4o will be used for capture. It involves detailed contextual understanding and strategic planning for the central local scenario.

[0056] The pose generation and geometry analyzer proposes a hierarchical grasping pose evaluation method based on a large model, aiming to select the optimal and safest six-DOF spatial pose from numerous candidate poses generated by GraspNet. For example... Figure 5 As shown, the entire process is divided into two stages: first, GPT-4o uses prior knowledge to perform preliminary heuristic filtering, and then performs refined collision detection and multi-dimensional scoring on the poses that pass the initial screening.

[0057] The first stage: Before performing precise geometric calculations, heuristic pose filtering utilizes GPT-4o's common-sense reasoning capabilities to perform a rapid semantic filtering of candidate six-DOF spatial poses. For example... Figure 5As shown, the system submits the initially generated set of candidate six-DOF spatial poses along with the target object's category information (e.g., "mango") to GPT-4o. The large model generates a filtering criterion based on its built-in prior knowledge. For example, it might determine: "For fruits like mangoes, the grasping position should be as close to the center as possible, and grasping points at the edges should be filtered out." Based on this high-level instruction, the system immediately eliminates grasping poses located at the object's edges that clearly defy common sense, thus greatly reducing the search space for subsequent processing.

[0058] The second stage: Multi-dimensional pose scoring based on collision detection. The candidate six-DOF spatial poses filtered in the first stage will enter a more rigorous quantitative evaluation phase. The core of this stage is efficient bounding box-based collision detection and a comprehensive scoring system. An efficient bounding box-based collision detection algorithm is employed, which has the advantages of low computational cost and simple implementation. The core idea of ​​collision detection is to construct a virtual bounding box model of the gripper and detect whether there are enough points in the environment point cloud to fall into the bounding box, thereby determining potential collision risks.

[0059] In its implementation, the system first constructs a directed bounding box (OBB) model of the gripper. Given the gripper's width (default value 0.08m), thickness (0.04m), and height (0.12m), the system establishes a corresponding bounding box geometric representation for each candidate six-DOF spatial pose, referred to as the candidate grasping pose. Subsequently, the system transforms the scene background point cloud from the world coordinate system to the gripper's local coordinate system through coordinate transformation.

[0060] in This is the homogeneous transformation matrix of the candidate grasping pose, containing position and rotation information. For local coordinates, Used as world coordinates.

[0061] In the local coordinate system of the gripper, the system employs an axis-aligned bounding box detection algorithm to detect whether the point cloud interferes with the gripper. When the number of points within the bounding box exceeds a preset threshold (10 points), a collision risk is identified. Compared to collision detection using complex geometric models, this method improves computational efficiency by approximately 70% while maintaining acceptable detection accuracy.

[0062] To select the optimal solution from the generated candidate grasping poses, the system designed a multi-dimensional scoring mechanism that comprehensively considers four key factors: workspace constraints, pose angle, collision risk, and target proximity. The pose score is defined as follows:

[0063] Among them, among them, Indicates the grasping posture. express Posture score, , , , The weighting coefficients for each item were set to 0.2, 0.15, 0.15, and 0.5 in the experiment. For workspace constraints, For the attitude angle scoring item, For the target distance scoring item, This is the collision risk scoring system. The scoring items are as follows: 1. Workspace constraints Ensure the gripping position is within the robot's workspace. The system checks whether the gripping position meets the lower and upper boundaries of the workspace.

[0064] 2. Posture Angle Scoring Item The system prioritizes a vertically downward gripping posture to reduce lateral forces. It calculates the angle between the gripping direction and the vertical direction.

[0065] 3. Target Distance Scoring Item Assess the proximity between the grasping position and the center of the target object. The system calculates the Euclidean distance between them.

[0066] 4. Collision Risk Rating Items The bounding box collision detection algorithm described above is used to assess the collision risk during the grasping process. Pose poses with no collision risk receive additional weighted scores.

[0067] The system calculates a normalized score for each candidate grasping posture and sorts them in descending order of score. The candidate grasping posture with the highest score is ultimately selected as the execution plan. In addition, the system retains the second-best posture with a score above 80% of the highest score as an alternative plan, enhancing the system's robustness.

[0068] To verify the effectiveness of this invention, experiments were conducted in different scenarios: like Figure 6As shown in Table 1, the experimental scenario and corresponding instructions were set up. In this scenario, the benchmark performance of each method in a undisturbed, target-defined environment was evaluated. Table 1 shows that the IMRG (Interactive Multimodal Robot Grasping Framework for Complex Environments) framework proposed in this application has already demonstrated its advantages in simple scenarios, achieving an average success rate of 86%, significantly better than VLG (Visual-Language-Action Joint Modeling for Target Grasping in Cluttered Environments) (65.2%) and ThinkGrasp (68.3%). More importantly, IMRG completes the task with an average of only 2.22 steps, far lower than VLG's 4.6 steps and ThinkGrasp's 3.01 steps. This result indicates that IMRG not only has a higher grasping success rate but also leads in decision-making efficiency.

[0069] Table 1. Comparison of the method provided in this application with other methods.

[0070] A deeper analysis of the results revealed that all methods performed well with explicit instructions like "grab a mango." IMRG's advantage lies primarily in handling implicit instructions like "I need a fruit." VLG and ThinkGrasp tend to select a pre-defined or random target when faced with such instructions, increasing the likelihood of incorrect selection. In contrast, IMRG's GPT-4o module accurately understands the category intent of the instruction. Even with multiple "fruit" options in the scene, it can identify the correct object through user interaction (or by directly selecting the most reasonable target when there is no ambiguity), thus achieving the desired result in one step. This directly explains why it completes the task with fewer average steps.

[0071] like Figure 7 As shown in Table 2, a more complex scenario was set up. Under extremely complex conditions, IMRG's performance remained robust, achieving an average success rate of 82.67%, while the success rates of the comparison methods VLG and ThinkGrasp dropped to 71.1% and 74.7%, respectively. This demonstrates the comprehensive advantages of the IMRG framework. Analysis of failure cases reveals that IMRG's success is mainly attributed to the synergistic effect of two key modules: Multi-view point cloud reconstruction: When an object is severely occluded, single-view observations lose a significant amount of geometric information, leading to poor or completely incorrect grasping poses generated by GraspNet. IMRG, by fusing point clouds from four perspectives, constructs a more complete 3D model of the target object, providing a reliable data foundation for generating high-quality grasping poses.

[0072] Collision Detection and Grasp Attitude Scoring System: In cluttered environments, many theoretically feasible grasping attitudes will collide with surrounding obstacles during actual execution. VLG and ThinkGrasp do not adequately consider this, leading to grasping failures. IMRG's collision detection module can filter out these high-risk actions, while the scoring system can select the optimal solution from the remaining safe attitudes, greatly improving the grasping success rate in confined spaces.

[0073] In particular, IMRG can plan multi-step operations, such as removing obstructions before grabbing the target. This further explains why its average number of actions of 3.5, although higher than that of simple scenarios, is still significantly better than other methods.

[0074] Table 2 Comparison of the method provided in this application with other methods

[0075] As shown in Tables 3 and 4, ablation experiments were conducted on a complex scene dataset. To quantitatively analyze the independent contribution and synergistic effect of each core component in the IMRG framework (Interactive Policy Generation I-GPT 4o, Multi-view Point Cloud Fusion M-point cloud, and Collision Detection cd), detailed ablation experiments were performed in a scene containing repeating objects. The experiments used the complete IMRG framework integrating all modules as a benchmark, measuring the impact on the overall system performance by removing key modules one by one. The experimental results clearly demonstrate the indispensability of each module and the close synergistic relationship between them.

[0076] Table 3 Baseline Indicators Provided in This Application

[0077] Table 4 Ablation test results of the method provided in this application

[0078] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0079] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A target localization and capture method based on multimodal semantics, characterized in that, include: Acquire two-dimensional images and natural language interaction commands input by the user; The two-dimensional image and the natural language interaction command are subjected to visual-language feature alignment processing. During the alignment process, the confidence and number of candidate targets are detected. If the natural language interaction command has semantic ambiguity, a rhetorical question is generated based on the generative interaction mechanism. The user's natural language feedback is received and parsed, and the semantic labels of the candidate targets are updated until the natural language interaction command is unambiguous and a unique target is determined. Acquire multi-view two-dimensional images of the target; The multi-view two-dimensional images are fused to obtain a complete three-dimensional point cloud model of the target; The three-dimensional point cloud model is input into a deep learning-based spatial pose generation network to obtain multiple candidate six-degree-of-freedom spatial poses. The candidate six-degree-of-freedom spatial poses are sequentially subjected to semantic commonsense filtering and geometric interference filtering; Calculate the attitude score for each of the filtered candidate six-DOF spatial poses, and select the candidate six-DOF spatial pose with the highest attitude score as the optimal target three-dimensional pose; Generate a sequence of grasping operation parameters corresponding to the optimal target 3D pose; The target is captured by executing the sequence of capture operation parameters.

2. The target localization and grasping method based on multimodal semantics according to claim 1, characterized in that, While determining the target, a two-dimensional pixel-level mask of the target is also determined, and then the multi-view two-dimensional image corresponding to the position of the two-dimensional pixel-level mask is acquired.

3. The target localization and grasping method based on multimodal semantics according to claim 1, characterized in that, The semantic commonsense filtering includes: using a commonsense knowledge base to eliminate candidate six-degree-of-freedom spatial poses that do not conform to the object operation logic.

4. The target localization and grasping method based on multimodal semantics according to claim 3, characterized in that, Alignment processing, semantic ambiguity detection, and generative interaction are performed using a pre-trained visual language large model, with the common sense knowledge base set in the visual language large model.

5. The target localization and grasping method based on multimodal semantics according to claim 4, characterized in that, The large visual language model uses GPT-4o.

6. The target localization and grasping method based on multimodal semantics according to claim 4, characterized in that, The method for eliminating candidate six-DOF spatial poses using the aforementioned common sense knowledge base includes: The initially generated candidate six-DOF spatial pose and target category information are input into the visual language large model; The visual language big model generates corresponding filtering criteria based on the common sense knowledge base. Candidate six-degree-of-freedom spatial poses that do not conform to the object operation logic are eliminated according to the filtering criteria.

7. The target localization and grasping method based on multimodal semantics according to claim 1, characterized in that, The geometric interference filtering includes: constructing a three-dimensional bounding box geometric model of the gripper, calculating the spatial overlap between the three-dimensional bounding box geometric model and the scene background point cloud under each of the candidate six degrees of freedom spatial poses, and removing the candidate six degrees of freedom spatial poses whose spatial overlap exceeds a threshold.

8. The target localization and grasping method based on multimodal semantics according to claim 7, characterized in that, The method for calculating the spatial overlap between the 3D bounding box geometry model and the scene background point cloud includes: The scene background point cloud is transformed from the world coordinate system to the local coordinate system of the gripper through coordinate transformation; In the local coordinate system of the gripper, the number of points belonging to the scene background point cloud within the three-dimensional bounding box geometry model is detected using an axis-aligned bounding box determination algorithm, i.e., the spatial overlap.

9. The target localization and grasping method based on multimodal semantics according to claim 1, characterized in that, The attitude score is calculated according to the following formula: in, Indicates the grasping posture. express Posture score, , , , These are the weighting coefficients for each item. For workspace constraints, For the attitude angle scoring item, For the target distance scoring item, This is a collision risk scoring item.

10. A system applying the target localization and grasping method based on multimodal semantics as described in any one of claims 1-9, characterized in that, include: The data input layer is used to acquire two-dimensional images and natural language interaction commands input by the user; The multimodal computing core includes: A semantic understanding engine is used to perform visual-linguistic feature alignment processing on the two-dimensional image and the natural language interaction command. During the alignment process, the confidence and number of candidate targets are detected. If there is semantic ambiguity in the natural language interaction command, a rhetorical question is generated based on the generative interaction mechanism. The user's natural language feedback is received and parsed, and the semantic labels of the candidate targets are updated until there is no ambiguity in the natural language interaction command, and a unique target is determined. A spatial computing engine is used to acquire multi-view two-dimensional images of the target collected by the data input layer, fuse the multi-view two-dimensional images to obtain a complete three-dimensional point cloud model of the target, input the three-dimensional point cloud model into a deep learning-based spatial pose generation network to obtain multiple candidate six-degree-of-freedom spatial poses, pass the multiple candidate six-degree-of-freedom spatial poses through semantic common sense filtering and geometric interference filtering in sequence, calculate the pose score of each filtered candidate six-degree-of-freedom spatial pose, and select the candidate six-degree-of-freedom spatial pose with the highest pose score as the optimal target three-dimensional pose. The result output layer is used to generate a sequence of grasping operation parameters corresponding to the optimal target 3D pose. The action execution module is used to execute the grasping operation parameter sequence to grasp the target.

Images