A target positioning and grabbing method and system based on multi-modal semantics

Abstract

The application discloses a target positioning and grabbing method and system based on multi-modal semantics, and relates to the technical fields of image recognition and mechanical control. The method comprises the following steps: acquiring a two-dimensional image and a natural language interactive instruction; performing visual-language feature alignment processing; if there is semantic ambiguity in the natural language interactive instruction during the alignment processing, generating a rhetorical question based on a generative interactive mechanism to determine a unique target; collecting multi-view two-dimensional images of the target and fusing to obtain a three-dimensional point cloud model; inputting the three-dimensional point cloud model into a spatial pose generation network to obtain a candidate six-degree-of-freedom spatial pose; sequentially performing semantic common sense filtering and geometric interference filtering; calculating the pose score of the filtered candidate six-degree-of-freedom spatial pose, and selecting an optimal target three-dimensional pose; generating a grabbing operation parameter sequence; and executing the grabbing operation parameter sequence. The method solves the problem of unsuccessful grabbing caused by semantic ambiguity and occlusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition and mechanical control, and particularly relates to a target positioning and grasping method based on image and natural semantic analysis. BACKGROUND

[0002] In recent years, with the rapid development of computer vision and embodied intelligence technology, three-dimensional target pose estimation technology based on multi-modal understanding has become a research hotspot. However, in the actual complex unstructured scene where there are multiple targets with different shapes and attributes, the existing visual perception and three-dimensional modeling algorithms still face many challenges.

[0003] Firstly, in the aspect of semantic understanding, the existing visual system has insufficient semantic understanding of natural language instructions, and it is difficult to handle the implicit expression or ambiguous instructions commonly used by users. In addition, when there are multiple similar objects in the visual scene, the traditional target positioning algorithm is difficult to eliminate semantic ambiguity through a single instruction, resulting in a serious decline in the accuracy of target positioning. Secondly, in the aspect of three-dimensional perception modeling, the existing methods mostly rely on single-view RGB-D (Red Green Blue-Depth) images. Due to the serious occlusion and asymmetric geometry information of single-view, the three-dimensional point cloud model generated by it is incomplete and incomplete, which leads to the precision and robustness of subsequent six-degree-of-freedom (6-DoF) pose estimation cannot meet the needs of complex application scenarios, and further makes the target grasping unsuccessful. SUMMARY

[0004] Embodiments of the present application provide a target positioning and grasping method and system based on multi-modal semantics to solve the problem that it is difficult to eliminate semantic ambiguity through a single instruction and the three-dimensional point cloud model is incomplete due to single-view images, which ultimately leads to unsuccessful grasping.

[0005] In one aspect, the embodiments of the present application provide a target positioning and grasping method based on multi-modal semantics, comprising:

[0006] acquiring a two-dimensional image and a natural language interactive instruction input by a user;

[0007] performing visual-language feature alignment processing on the two-dimensional image and the natural language interactive instruction, detecting the confidence and number of candidate targets in the alignment processing, if there is semantic ambiguity in the natural language interactive instruction, generating a counter-question sentence based on a generative interaction mechanism, receiving and analyzing the natural language feedback of the user, updating the semantic label of the candidate target, until there is no ambiguity in the natural language interactive instruction, and determining a unique target;

[0008] collecting multi-view two-dimensional images of the target;

[0009] fusing the multi-view two-dimensional images to obtain a complete three-dimensional point cloud model of the target;

[0010] input the three-dimensional point cloud model into a deep learning-based spatial pose generation network to obtain a plurality of candidate six-degree-of-freedom spatial poses;

[0011] the plurality of candidate six-degree-of-freedom spatial poses are sequentially subjected to semantic common sense filtering and geometric interference filtering;

[0012] a pose score of each filtered candidate six-degree-of-freedom spatial pose is calculated, and a candidate six-degree-of-freedom spatial pose with the highest pose score is selected as an optimal target three-dimensional pose;

[0013] a grasping operation parameter sequence corresponding to the optimal target three-dimensional pose is generated;

[0014] the grasping operation parameter sequence is executed to achieve grasping of the target.

[0015] In another aspect, the embodiments of the present application also provide a target positioning and grasping system based on multi-modal semantics, comprising:

[0016] a data input layer configured to acquire a two-dimensional image and a natural language interaction instruction input by a user;

[0017] a multi-modal computing core, comprising:

[0018] a semantic understanding engine configured to perform visual-linguistic feature alignment processing on the two-dimensional image and the natural language interaction instruction, detect confidence and quantity of candidate targets in the alignment processing, generate a counter-question statement based on a generative interaction mechanism if there is semantic ambiguity in the natural language interaction instruction, receive and analyze natural language feedback of the user, update semantic labels of the candidate targets, and determine a unique target until there is no ambiguity in the natural language interaction instruction;

[0019] a spatial computing engine configured to acquire multi-view two-dimensional images of the target collected by the data input layer, fuse the multi-view two-dimensional images to obtain a complete three-dimensional point cloud model of the target, input the three-dimensional point cloud model into a deep learning-based spatial pose generation network to obtain a plurality of candidate six-degree-of-freedom spatial poses, sequentially subject the plurality of candidate six-degree-of-freedom spatial poses to semantic common sense filtering and geometric interference filtering, calculate a pose score of each filtered candidate six-degree-of-freedom spatial pose, and select a candidate six-degree-of-freedom spatial pose with the highest pose score as an optimal target three-dimensional pose;

[0020] a result output layer configured to generate a grasping operation parameter sequence corresponding to the optimal target three-dimensional pose;

[0021] an action execution module configured to execute the grasping operation parameter sequence to achieve grasping of the target.

[0022] The target positioning and grasping method and system based on multi-modal semantics in the present application have the following advantages:

[0023] (1) Solving the problem of semantic ambiguity, improving the positioning accuracy. The introduction of visual-linguistic large model can not only handle explicit instructions, but also understand implicit operation intention; through multi-round natural language interaction mechanism, it actively eliminates the semantic ambiguity caused by object attributes (color, type, position) in the scene, ensuring the uniqueness and accuracy of target positioning.

[0024] (2) Overcoming the problem of occlusion, improving the integrity of three-dimensional modeling. The fusion strategy of multi-view RGB-D image and point cloud effectively solves the problem of incomplete geometric information caused by missing point cloud in single view, and constructs a more complete and high-precision three-dimensional point cloud model of the target object, providing a reliable data basis for subsequent six-degree-of-freedom pose estimation.

[0025] (3) Optimizing the pose evaluation mechanism and improving the robustness of estimation. The hierarchical evaluation mechanism of semantic common sense filtering and three-dimensional geometric constraint analysis is adopted. Semantic filtering uses the common sense knowledge of large model for preliminary screening; geometric constraint analysis detects spatial interference through three-dimensional bounding box model, filters out the poses with potential interference risk, greatly improves the accuracy of the final output pose parameters and the robustness in complex scenes, and thus improves the success rate of grasping. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0027] Figure 1 A flowchart of a target positioning and grasping method based on multi-modal semantics provided by an embodiment of the present application.

[0028] Figure 2 A composition diagram of a target positioning and grasping system based on multi-modal semantics provided by an embodiment of the present application.

[0029] Figure 3 A general framework diagram of a target positioning and grasping method based on multi-modal semantics provided by an embodiment of the present application.

[0030] Figure 4 An interactive strategy generation schematic diagram of a target positioning and grasping method based on multi-modal semantics provided by an embodiment of the present application.

[0031] Figure 5A grasp pose evaluation method of a target positioning and grasping method based on multi-modal semantics provided by an embodiment of the present application.

[0032] Figure 6 A simple experimental scene diagram of a target positioning and grasping method based on multi-modal semantics provided by an embodiment of the present application.

[0033] Figure 7 A complex experimental scene diagram of a target positioning and grasping method based on multi-modal semantics provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0035] Figure 1 A flowchart of a target positioning and grasping method based on multi-modal semantics provided by an embodiment of the present application. The embodiment of the present application provides a target positioning and grasping method based on multi-modal semantics, which comprises the following steps:

[0036] S100, acquiring a two-dimensional image and a natural language interaction instruction input by a user.

[0037] Exemplarily, a two-dimensional image containing a target in a current scene can be acquired by a visual acquisition device, and the two-dimensional image is specifically an RGB-D image. The natural language interaction instruction can be input in the form of voice or text.

[0038] S110, performing visual-language feature alignment processing on the two-dimensional image and the natural language interaction instruction, detecting the confidence and the number of candidate targets in the alignment processing process, generating a rhetorical question based on a generative interaction mechanism if there is semantic ambiguity in the natural language interaction instruction, receiving and analyzing the natural language feedback of the user, updating the semantic label of the candidate target, until there is no ambiguity in the natural language interaction instruction, and determining a unique target.

[0039] Exemplarily, the embodiment of the present application utilizes a pre-trained visual language large model for alignment processing, semantic ambiguity detection and generative interaction, and the visual language large model adopts GPT-4o (generative pre-trained transformer 4o). GPT-4o is designed using a single Transformer architecture, which unifies data of different modalities such as text, images and audio into a neural network for processing. The core of this architecture is the Transformer, which processes input sequence data through a self-attention mechanism (Self-Attention).

[0040] Specifically, the alignment processing is a deep alignment in a high-dimensional space between the features of the input two-dimensional image and the semantic features of the natural language interaction instruction.

[0041] The semantic ambiguity of the natural language interaction instruction means that multiple candidate targets meet the requirements of the natural language interaction instruction. At this time, a targeted counter-question sentence will be generated by GPT-4o, such as "Do you mean the red apple on the left or the green apple on the right?" After the user receives the counter-question sentence, he / she will input his / her choice in the form of voice or text, and then the system receives and analyzes the user's natural language feedback, dynamically updates the semantic label of the target, and locks the ID of the unique target and its corresponding two-dimensional pixel-level mask (Mask).

[0042] As can be seen, in the embodiment of the present application, GPT-4o takes a visual scene and a natural language instruction g as input, based on the multi-modal features of text and images, combined with instruction semantics and task context, through one or more rounds of language interaction with the user, it can be ensured that the unique target object can be accurately located. After the target is locked, the system uses the semantic segmentation network LANG-SAM to perform pixel-level segmentation on the target to obtain the boundary box and two-dimensional pixel-level mask of the target.

[0043] LANG-SAM uses an image encoder (usually a visual transformer ViT) to extract image embeddings as the basis for mask prediction. The model also contains a prompt encoder that encodes various types of input prompts, such as point coordinates, bounding boxes, and low-resolution mask inputs. These encoded prompts are then input into the mask decoder along with the image embeddings to generate the final object mask.

[0044] LANG-SAM aims to handle various prompts, including:

[0045] Mask: A coarse, low-resolution binary mask can be provided as an initial input to guide the model.

[0046] Point: The user can input [x, y] coordinates and their type (foreground or background) to help define the object boundary.

[0047] Box: A bounding box can be specified using coordinates [x1, y1, x2, y2] to inform the model of the object's location and size.

[0048] Text: Textual cues can also be used to provide additional context or specify objects of interest.

[0049] S120, collect multi-view two-dimensional images of the target.

[0050] Illustratively, after the two-dimensional pixel-level mask of the target is collected, multi-view two-dimensional images corresponding to the position of the two-dimensional pixel-level mask are collected, and at least four views of RGB-D images of the front, back, left, and right of the target are collected.

[0051] S130, fuse the multi-view two-dimensional images to obtain a complete three-dimensional point cloud model of the target.

[0052] Illustratively, using a point cloud registration algorithm, the embodiment of the present application preferably uses an Iterative Closest Point (ICP) algorithm to unify the local point cloud data in the multi-view two-dimensional images to the world coordinate system for fusion. This step effectively complements the missing geometric information caused by self-occlusion of the object under single-view, generating a high-fidelity complete three-dimensional point cloud model of the target.

[0053] S140, input the three-dimensional point cloud model into a spatial pose generation network based on deep learning to obtain a plurality of candidate six-degree-of-freedom spatial poses.

[0054] Illustratively, the spatial pose generation network based on deep learning in the embodiment of the present application preferably uses GraspNet or similar 6-DoF generation network, and the candidate six-degree-of-freedom spatial poses output by the network include position coordinates (x, y, z) and attitude direction (roll, pitch, yaw).

[0055] GraspNet is a deep learning model specifically designed to predict 6-DoF (six degrees of freedom) grasping poses. It plays an important role in the field of robotic grasping, selecting the optimal grasping pose by simulating human grasping behavior. GraspNet model uses a deep neural network architecture, integrating multiple data sources such as visual images of objects, physical properties, etc. Through learning these data, it can output grasping schemes for different objects.

[0056] The network structure of this model usually includes a convolutional neural network (CNN) as the main feature extraction, combined with a recurrent neural network (RNN) or attention mechanism to process sequential data, ensuring the coherence and adaptability of grasping actions. For complex three-dimensional space grasping, GraspNet may also integrate additional network components such as Transformer architecture to handle long-range dependencies and complex three-dimensional spatial information.

[0057] S150, sequentially filtering the plurality of candidate six-degree-of-freedom spatial poses through semantic common sense filtering and geometric interference filtering.

[0058] Exemplarily, the semantic common sense filtering comprises: eliminating, by using a common sense knowledge base, candidate six-degree-of-freedom spatial poses that do not conform to object operation logic, for example, eliminating unstable poses that only contact the edge of the fruit stem for fruit objects. The common sense knowledge base is set in the visual language large model.

[0059] Specifically, the method of eliminating candidate six-degree-of-freedom spatial poses by using the common sense knowledge base comprises:

[0060] inputting the preliminarily generated candidate six-degree-of-freedom spatial poses and the category information of the target into the visual language large model;

[0061] the visual language large model generates corresponding filtering criteria according to the common sense knowledge base;

[0062] eliminating, according to the filtering criteria, candidate six-degree-of-freedom spatial poses that do not conform to object operation logic.

[0063] Further, the geometric interference filtering comprises: constructing a three-dimensional bounding box geometric model of the gripper, calculating the spatial overlap amount of the three-dimensional bounding box geometric model with the scene background point cloud obtained by shooting through the built-in depth camera under each candidate six-degree-of-freedom spatial pose, and eliminating candidate six-degree-of-freedom spatial poses whose spatial overlap amount exceeds a threshold.

[0064] Specifically, the method of calculating the spatial overlap amount of the three-dimensional bounding box geometric model with the scene background point cloud comprises:

[0065] transforming the scene background point cloud from the world coordinate system to the local coordinate system of the gripper through coordinate transformation;

[0066] in the local coordinate system of the gripper, the number of points belonging to the scene background point cloud in the three-dimensional bounding box geometric model, i.e., the spatial overlap amount, is detected by using an axis-aligned bounding box determination algorithm.

[0067] S160, calculating the pose score of each filtered candidate six-degree-of-freedom spatial pose, and selecting the candidate six-degree-of-freedom spatial pose with the highest pose score as the optimal target three-dimensional pose.

[0068] S170, generating a grasping operation parameter sequence corresponding to the optimal target three-dimensional pose.

[0069] S180, executing the grasping operation parameter sequence to realize grasping of the target.

[0070] The embodiment of the present application also provides a target positioning and grasping system based on multi-modal semantics, as shown in the accompanying drawings. Figure 2 The system comprises:

[0071] a data input layer configured to acquire a two-dimensional image and a natural language interaction instruction input by a user;

[0072] a multi-modal computing core comprising:

[0073] a semantic understanding engine configured to perform visual-linguistic feature alignment on the two-dimensional image and the natural language interaction instruction, detect confidence and quantity of candidate targets during the alignment process, generate a counter-question statement based on a generative interaction mechanism if the natural language interaction instruction has semantic ambiguity, receive and parse natural language feedback from the user, update semantic labels of the candidate targets until there is no ambiguity in the natural language interaction instruction, and determine a unique target;

[0074] a spatial computing engine configured to acquire multi-view two-dimensional images of the target collected by the data input layer, fuse the multi-view two-dimensional images to obtain a complete three-dimensional point cloud model of the target, input the three-dimensional point cloud model into a spatial pose generation network based on deep learning to obtain a plurality of candidate six-degree-of-freedom spatial poses, sequentially perform semantic common sense filtering and geometric interference filtering on the plurality of candidate six-degree-of-freedom spatial poses, calculate a pose score of each filtered candidate six-degree-of-freedom spatial pose, and select a candidate six-degree-of-freedom spatial pose with the highest pose score as an optimal target three-dimensional pose;

[0075] a result output layer configured to generate a grasping operation parameter sequence corresponding to the optimal target three-dimensional pose;

[0076] an action execution module configured to execute the grasping operation parameter sequence to achieve grasping of the target.

[0077] By way of example, the data input layer serves as a perception interface of the system and is responsible for receiving raw data streams. Specifically, the data input layer includes:

[0078] an instruction interface configured to receive natural language interaction instructions input by a user through voice or text; and a vision interface configured to connect an RGB-D sensor group and receive two-dimensional images and depth point cloud streams of a scene in real time.

[0079] The multi-modal computing core adopts a dual-engine architecture with parallel semantic understanding and spatial computing, and is built-in with a pre-trained visual-linguistic large model for performing feature alignment and intent reasoning. The multi-modal computing core includes an interactive disambiguation controller for monitoring semantic ambiguity. When ambiguity is detected, the controller takes over the dialogue flow, generates a query statement, and updates semantic labels according to user feedback, thereby solving the "long tail" recognition problem in complex unstructured scenarios.

[0080] The spatial computing engine contains a three-dimensional point cloud reconstruction module and a pose generation and geometry analyzer. The three-dimensional point cloud reconstruction module receives the locked target area information, schedules the multi-view registration algorithm to generate a high-precision three-dimensional geometric model, and overcomes the single-view occlusion defect. The pose generation and geometry analyzer is a cascaded computing unit that first uses a deep network to generate a high-dimensional pose space, and then uses a virtual geometric bounding box technique to perform strict spatial interference detection. This analyzer ensures that the output pose of the system is “collision-free” and “executable” in physical space.

[0081] The result output layer converts the data obtained by the computing core into standardized control signals or visualized data, outputs the coordinate matrix and quaternion after geometric optimization, and provides them for downstream execution modules or rendering engines; the bounding box of the target, the recommended operation center point, or the interference warning area can be rendered on the user interface.

[0082] As shown in Figure 3 , 4 , the multi-modal computing core proposes a multi-modal human-computer interaction system architecture that integrates visual perception and natural language understanding capabilities to achieve intuitive and efficient object operation tasks. The core consists of three parts: 1. Visual-linguistic processing; 2. Interactive target selection; 3. Dynamic, context-aware strategy generation and execution. Adopting a hierarchical design principle, each part exchanges data through standardized interfaces to ensure the scalability and robustness of the system. The system architecture can be expressed as a multi-modal mapping function :

[0083]

[0084] where represents the visual input, i.e., a two-dimensional image, represents the user's initial natural language instruction, represents a sequence of grasp operation parameters that may contain multiple steps and ultimately lead to the target.

[0085] First, the visual-linguistic large model in visual-linguistic processing performs preliminary analysis on the visual and instruction , and identifies potential candidate targets. If there are multiple candidate targets causing ambiguity, the interactive disambiguation controller will be activated to clarify the user's final intention to grasp the object through natural language interaction with the user, called .

[0086] After the unique target is confirmed, the strategy generation does not directly plan to grasp . Instead, it will use GPT-4o to plan a sequence of actions to achieve the goal, called Detailed contextual understanding and strategic planning for the central local scene.

[0087] The pose generation and geometry analyzer proposes a hierarchical grasp pose evaluation method based on large models, aiming to filter out the optimal and safest one from the numerous candidate six-degree-of-freedom spatial poses generated by GraspNet. As shown in Figure 5 , the entire process is divided into two stages: first, GPT-4o uses prior knowledge for preliminary heuristic filtering, and then performs fine collision detection and multi-dimensional scoring on the filtered poses.

[0088] First stage: Before the heuristic pose filtering, the GPT-4o's common sense reasoning ability is used to perform a round of rapid semantic filtering on the candidate six-degree-of-freedom spatial poses before performing accurate geometric calculations. As shown in Figure 5 , the system submits the set of initially generated candidate six-degree-of-freedom spatial poses and the category information of the target object (e.g., "mango") to GPT-4o. The large model generates a filtering criterion based on its built-in prior knowledge. For example, it may judge: "For fruits like mangoes, the grasp position should be as close to the center as possible, and grasp points at the edge should be filtered out." Based on this high-level instruction, the system immediately eliminates grasp poses that are obviously not common sense, such as those located at the edge of the object, greatly reducing the search space for subsequent processing.

[0089] Second stage: Multi-dimensional pose scoring based on collision detection The candidate six-degree-of-freedom spatial poses filtered by the first stage will enter a more rigorous quantitative evaluation link. The core of this link is efficient collision detection based on bounding boxes and a comprehensive scoring system. An efficient collision detection algorithm based on bounding boxes is used, which has the advantages of low computational overhead and easy implementation. The core idea of collision detection is to construct a virtual bounding box model of the gripper and detect whether there are enough points in the environment point cloud falling into the bounding box, thereby determining the potential collision risk.

[0090] In specific implementation, the system first constructs an oriented bounding box (OBB) model of the gripper. Given the gripper width (default value is 0.08m), thickness (0.04m) and height (0.12m), the system establishes the corresponding bounding box geometric representation for each candidate six-degree-of-freedom spatial pose, called candidate grasp pose. Subsequently, the scene background point cloud is transformed from the world coordinate system to the local coordinate system of the gripper through coordinate transformation:

[0091]

[0092] where is the homogeneous transformation matrix of the candidate grasp pose, containing position and rotation information, is the local coordinate, World coordinates.

[0093] In the local coordinate system of the gripper, the system uses an axis-aligned bounding box collision detection algorithm to detect whether the point cloud interferes with the gripper. When the number of points in the bounding box exceeds the preset threshold (10 points), it is determined that there is a collision risk. Compared with complex geometric model collision detection, the calculation efficiency is improved by about 70%, while maintaining acceptable detection accuracy.

[0094] To select the optimal solution from the generated candidate grasping poses, the system designs a multi-dimensional scoring mechanism, which considers four key factors: workspace constraints, pose angle, collision risk, and target proximity. The pose score is defined as:

[0095]

[0096] wherein, represents the grasping pose, represents the pose score of, , , , are the weight coefficients of each item, which are set to 0.2, 0.15, 0.15, and 0.5 respectively in the experiment, is the workspace constraint term, is the pose angle score term, is the target distance score term, is the collision risk score term. Each scoring term is as follows:

[0097] 1. Workspace constraint term : Ensure that the grasping position is within the robot's workspace. The system checks whether the grasping position meets the lower and upper bounds of the workspace.

[0098] 2. Pose angle score term : Preferentially select a vertically downward grasping pose to reduce lateral force. The system calculates the angle between the grasping direction and the vertical direction.

[0099] 3. Target distance score term : Evaluate the proximity of the grasping position to the center of the target object. The system calculates the Euclidean distance between the two.

[0100] 4. Collision risk score term : Through the aforementioned bounding box collision detection algorithm, evaluate the collision risk during grasping execution. Poses with no collision risk receive additional score weighting.

[0101] The system calculates a normalized score for each candidate grasping pose and ranks them in descending order of score. The candidate grasping pose with the highest score is finally selected as the execution scheme, and in addition, the system also retains the suboptimal poses with scores above 80% of the highest score as backup schemes, thereby enhancing the robustness of the system.

[0102] To verify the effectiveness of the present application, the present application selects different scenes for experimental verification:

[0103] As shown in Figure 6 , the experimental scenarios and corresponding instructions are set, as shown in Table 1. In this scenario, the benchmark performance of each method in an interference-free and clear target environment is evaluated. As shown in Table 1, the performance of the IMRG (interactive multi-modal robot grasping framework for complex environments) framework proposed in the present application in a simple scenario has already shown an advantage, with an average success rate of 86%, which is significantly better than VLG (visual language action joint modeling for target grasping in cluttered environments) (65.2%) and ThinkGrasp (68.3%). More importantly, the average number of actions for completing the task by IMRG is only 2.22 steps, which is much lower than 4.6 steps of VLG and 3.01 steps of ThinkGrasp. This result shows that IMRG not only has a higher success rate of grasping, but also has a leading decision-making efficiency.

[0104] Table 1 Comparison of the method provided in the present application with other methods

[0105]

[0106] After in-depth analysis of the results, it is found that for explicit instructions such as "grasp the mango", all methods can be well completed. The advantage of IMRG mainly lies in dealing with implicit instructions such as "I need a fruit". VLG and ThinkGrasp tend to select a preset or random target when facing such instructions, which increases the possibility of selecting errors. In contrast, the GPT-4o module of IMRG can accurately understand the category intention of the instruction, and even if there are multiple "fruit" options in the scene, it can also lock the correct object through interaction with the user (or directly select the most reasonable target when there is no ambiguity), thereby achieving one-step success, which directly explains why it can complete the task with fewer average steps.

[0107] As shown in Figure 7 , more complex scenarios are set, as shown in Table 2. Under extremely complex conditions, the performance of IMRG is still robust, achieving an average success rate of 82.67%, while the success rates of the comparative methods VLG and ThinkGrasp are reduced to 71.1% and 74.7%, respectively. This proves the comprehensive advantage of the IMRG framework. Through analysis of failed cases, the success of IMRG is mainly due to the synergistic effect of two key modules:

[0108] Multi-view point cloud reconstruction: When an object is heavily occluded, single-view observation will lose a large amount of geometric information, resulting in poor or completely wrong grasp poses generated by GraspNet. IMRG builds a more complete 3D model of the target object by fusing point clouds from four views, providing a reliable data foundation for generating high-quality grasp poses.

[0109] Collision detection and grasp pose scoring system: In cluttered environments, many theoretically feasible grasp poses will collide with surrounding obstacles in actual execution. VLG and ThinkGrasp do not take this into account, resulting in failed grasps. The collision detection module of IMRG can filter out these high-risk actions, and the scoring system can select the optimal solution from the remaining safe poses, greatly improving the success rate of grasping in narrow spaces.

[0110] In particular, IMRG can also plan multi-step operations, such as moving the occluder before grasping the target, which further explains why its average number of actions, 3.5, is higher than that of simple scenes, but still significantly better than other methods.

[0111] Table 2 Comparison of the method provided by the application with other methods

[0112]

[0113] As shown in Tables 3 and 4, ablation experiments were performed on the complex scene dataset. In order to quantitatively analyze the independent contribution and synergistic effect of each core component (interactive strategy generation I-GPT 4o, multi-view point cloud fusion M-point cloud, collision detection cd) in the IMRG framework, detailed ablation experiments were conducted under the "scene with duplicate objects". The experiment takes the complete IMRG framework integrating all modules as the benchmark, and measures the impact of each key module on the overall performance of the system by removing each key module one by one. The experimental results clearly prove the indispensability of each module and the close synergistic relationship between them.

[0114] Table 3 Baseline indicators provided by the application

[0115]

[0116] Table 4 Ablation experiment results of the method provided by the application

[0117]

[0118] Although the preferred embodiments of the application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they understand the basic inventive concept. Therefore, the appended claims are intended to include the preferred embodiments and all changes and modifications falling within the scope of the application.

[0119] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A multi-modal semantic based target positioning and grasping method, characterized in that, The method comprises the following steps: acquiring a two-dimensional image and a natural language interaction instruction input by a user; performing visual-language feature alignment processing on the two-dimensional image and the natural language interaction instruction, detecting confidence and quantity of candidate targets in the alignment processing, generating a rhetorical question based on a generative interaction mechanism if there is semantic ambiguity in the natural language interaction instruction, receiving and analyzing natural language feedback of the user, updating semantic labels of the candidate targets until there is no ambiguity in the natural language interaction instruction, and determining a unique target; acquiring multi-view two-dimensional images of the target; fusing the multi-view two-dimensional images to obtain a complete three-dimensional point cloud model of the target; inputting the three-dimensional point cloud model into a spatial pose generation network based on deep learning to obtain a plurality of candidate six-degree-of-freedom spatial poses; performing semantic common sense filtering and geometric interference filtering on the plurality of candidate six-degree-of-freedom spatial poses in sequence; calculating a pose score of each filtered candidate six-degree-of-freedom spatial pose, and selecting the candidate six-degree-of-freedom spatial pose with the highest pose score as an optimal target three-dimensional pose; generating a grasping operation parameter sequence corresponding to the optimal target three-dimensional pose; executing the grasping operation parameter sequence to realize grasping of the target; the semantic common sense filtering comprises: using a common sense knowledge base to eliminate the candidate six-degree-of-freedom spatial poses that do not conform to object operation logic; the method for eliminating the candidate six-degree-of-freedom spatial poses using the common sense knowledge base comprises: inputting the preliminarily generated candidate six-degree-of-freedom spatial poses and category information of the target into a visual language large model; the visual language large model generates corresponding filtering criteria according to the common sense knowledge base; eliminating the candidate six-degree-of-freedom spatial poses that do not conform to object operation logic according to the filtering criteria; the geometric interference filtering comprises: constructing a three-dimensional bounding box geometric model of a gripper, calculating a spatial overlap amount of the three-dimensional bounding box geometric model with a scene background point cloud under each candidate six-degree-of-freedom spatial pose, and eliminating the candidate six-degree-of-freedom spatial poses whose spatial overlap amount exceeds a threshold; the method for calculating the spatial overlap amount of the three-dimensional bounding box geometric model with the scene background point cloud comprises: transforming the scene background point cloud from a world coordinate system to a local coordinate system of the gripper; in the local coordinate system of the gripper, the number of points belonging to the scene background point cloud in the three-dimensional bounding box geometric model is detected, i.e., the spatial overlap amount, by using an axis-aligned bounding box determination algorithm.

2. The target positioning and grasping method based on multi-modal semantics according to claim 1, characterized in that, While the target is determined, a two-dimensional pixel-level mask of the target is also determined, and then the multi-view two-dimensional images corresponding to the positions of the two-dimensional pixel-level mask are acquired.

3. The method of claim 1, wherein, The pre-trained visual language large model is used for alignment processing, semantic ambiguity detection and generative interaction, and the common sense knowledge base is set in the visual language large model.

4. The target positioning and grasping method based on multi-modal semantics according to claim 3, characterized in that, The visual language large model adopts GPT-4o.

5. The multi-modal semantic based target positioning and grasping method according to claim 1, wherein, The pose score is calculated according to the following formula: wherein, represents a grasp pose, represents a pose score, , , , are weight coefficients for each term, is a workspace constraint term, is a pose angle score term, is a target distance score term, is a collision risk score term.

6. The system for object localization and grasping based on multi-modal semantics according to any one of claims 1-5, characterized in that, The method comprises the following steps: a data input layer is used to acquire a two-dimensional image and a natural language interaction instruction input by a user; a multi-modal calculation core comprises: The semantic understanding engine is configured to perform visual-linguistic feature alignment on the two-dimensional image and the natural language interaction instruction, detect confidence and quantity of candidate targets during the alignment, generate a counter-question statement based on a generative interaction mechanism if there is semantic ambiguity in the natural language interaction instruction, receive and analyze natural language feedback from a user, update semantic labels of the candidate targets, and determine a unique target until there is no ambiguity in the natural language interaction instruction; The spatial computing engine is configured to acquire the multi-view two-dimensional image of the target collected by the data input layer, fuse the multi-view two-dimensional image to obtain a complete three-dimensional point cloud model of the target, input the three-dimensional point cloud model into a spatial pose generation network based on deep learning to obtain a plurality of candidate six-degree-of-freedom spatial poses, sequentially perform semantic common sense filtering and geometric interference filtering on the plurality of candidate six-degree-of-freedom spatial poses, calculate a pose score of each filtered candidate six-degree-of-freedom spatial pose, and select the candidate six-degree-of-freedom spatial pose with the highest pose score as an optimal target three-dimensional pose; The result output layer is configured to generate a grasping operation parameter sequence corresponding to the optimal target three-dimensional pose. The action execution module is configured to execute the grasping operation parameter sequence to achieve grasping of the target.

Images