Robot grasping pose detection method and system based on multi-modal semantic consistency verification and functional region guidance

Abstract

The application discloses a robot grasping pose detection method and system based on multi-modal semantic consistency verification and functional area guidance, and the method comprises the following steps: S1, inputting an RGB image and a user instruction into a visual language large model to output a boundary box of a target object and a functional area description; S2, inputting the boundary box into a segmentation large model to obtain an initial segmentation mask; S3, calculating the semantic similarity between the image area corresponding to the current segmentation mask and the functional area description by using a pre-trained visual-language alignment model; if a preset threshold or a preset iteration upper limit is reached, jumping to S5; otherwise, entering S4; S4, generating a correction prompt point according to the difference type of the current segmentation mask and the boundary box and inputting the correction prompt point and the boundary box into the segmentation large model for re-segmentation, and repeatedly executing step S3; and S5, generating a grasping pose based on the final output segmentation mask and depth data. The application effectively solves the problems of inaccurate large model output and segmentation model edge adhesion.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot technology, specifically, it relates to a robot grasping pose detection method and system based on multimodal semantic consistency verification and functional region guidance. Background Technology

[0002] Autonomous grasping is one of the core capabilities of intelligent robots. Current technologies suffer from the following problems:

[0003] (1) Traditional segmentation models have poor generalization: Segmentation models based on convolutional neural networks have low segmentation accuracy for unseen objects outside the training set and have difficulty understanding semantic-level grasping instructions, such as users requesting to grasp "semi-transparent objects" or objects with specific attributes (such as "metallic luster").

[0004] (2) The large visual language model lacks pixel-level precision: Although the large visual language model (VLM) can understand natural language and output a rough bounding box of an object, its output is only a bounding box and cannot provide a fine pixel-level mask, which is insufficient for direct use in grasping pose calculation.

[0005] (3) Robots do not understand the functional areas of objects: Existing methods often only recognize the object as a whole and do not understand the functional semantics of different areas of the object. For example, when performing the task of pouring water, the system should grasp the handle of the cup instead of the rim; when performing the task of picking up the cup, it should grasp the body of the cup. Existing technology cannot automatically select the correct grasping area according to the task instructions, resulting in random grasping positions and possible violation of functional semantics.

[0006] In recent years, methods combining VLM with large segmentation models have emerged for robotic grasping. However, these methods still lack mechanisms for verifying and correcting segmentation results, making it difficult to guarantee segmentation quality. Therefore, there is an urgent need for a robotic grasping method that can automatically verify and correct segmentation results. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings and deficiencies of the existing technology and provide a robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance. By introducing a semantic verification module between the visual language model and the segmentation model, adaptive feedback and prompt correction are performed based on the verification results, which effectively solves the problems of inaccurate output of the visual language model and incomplete segmentation or inclusion of background regions in the segmentation model.

[0008] The second objective of this invention is to provide a robot grasping pose detection system based on multimodal semantic consistency verification and functional region guidance.

[0009] The objective of this invention is achieved through the following technical solution: a robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance, comprising the following steps:

[0010] S1. Obtain the RGB image of the target object, and input the RGB image and user instructions into the visual language large model, and output the bounding box and functional area description of the target object;

[0011] S2. Input the bounding box into the segmentation model to obtain the initial segmentation mask;

[0012] S3. Calculate the semantic similarity between the image region corresponding to the current segmentation mask and the functional region description using a pre-trained visual-language alignment model; if the semantic similarity reaches a preset threshold or the current iteration count reaches a preset iteration limit, proceed to step S5; otherwise, proceed to step S4.

[0013] S4. Generate correction hints based on the difference type between the current segmentation mask and the bounding box. Input the correction hints and the bounding box into the large segmentation model for re-segmentation to obtain the updated segmentation mask. Increment the iteration count by 1, and then return to step S3.

[0014] S5. Generate the grasping pose based on the final output segmentation mask and depth data.

[0015] Preferably, in step S3, the semantic similarity is calculated using cosine similarity, and the formula for calculating cosine similarity S is:

[0016] ,

[0017] Among them, V img V is the image feature vector extracted by the visual-language alignment model for the image region corresponding to the segmentation mask. text The functional region is described by the text feature vector extracted by the visual-language alignment model.

[0018] Preferably, the difference types between the segmentation mask and the bounding box include oversegmentation, undersegmentation, and multiple candidate box ambiguity; wherein,

[0019] When the difference type is oversegmentation, a negative cue point is generated to suppress the expansion of the segmentation mask into the background region;

[0020] When the difference type is undersegmentation, a positive cue point is generated to guide the segmentation mask to grow towards the missing target region;

[0021] When the difference type is multi-candidate box ambiguity, perform parallel segmentation and semantic reordering of multi-candidate boxes.

[0022] Preferably, generating negative prompts includes the following steps:

[0023] Calculate the difference region between the current segmentation mask M and the bounding box B: R diff =MB;

[0024] If the difference region R diff If it contains pixels, then for the difference set region R diff Perform connected component analysis and extract the geometric centroid of the largest connected component as the negative cue point;

[0025] If the difference region R diff If the current segmentation mask M contains no pixels, then a morphological dilation operation is performed on the current segmentation mask M to obtain the annular region at the edge of the current segmentation mask as the new difference region R. diff Then, the geometric centroid of the annular region is extracted as the negative prompt point.

[0026] Preferably, generating positive cue points includes the following steps:

[0027] Calculate the region R inside the bounding box B that is not covered by the current segmentation mask M. miss =BM;

[0028] In the region R miss Identify the connected component with the largest area and calculate the distance transformation graph of the connected component;

[0029] The local maxima of the distance transformation graph are selected as the positive prompts.

[0030] Preferably, the parallel segmentation and semantic reordering of multiple candidate boxes includes the following steps:

[0031] When the visual language big model outputs multiple candidate bounding boxes, the center of each candidate bounding box is used as the initial cue, and independent masks corresponding to each candidate bounding box are generated in parallel to obtain an independent mask set.

[0032] The image region corresponding to each independent mask in the set of independent masks and the functional region description are input into the visual-language alignment model to calculate semantic similarity.

[0033] Select the mask with the highest semantic similarity as the final output and proceed to step S5.

[0034] Preferably, the preset threshold value ranges from 0.6 to 0.9, and the preset number of iterations ranges from 3 to 5.

[0035] Preferably, in step S5, generating the grasping pose specifically includes: generating a 3D point cloud of the target object based on the final output segmentation mask and combining it with the depth image; and calculating a six-degree-of-freedom grasping pose from the 3D point cloud using a grasping pose detection network.

[0036] A robot grasping pose detection system based on multimodal semantic consistency verification and functional region guidance includes:

[0037] The visual language large model unit is used to acquire the RGB image of the target object, input the RGB image and user instructions into the visual language large model, and output the bounding box and functional region description of the target object.

[0038] A segmentation large model unit is used to input the bounding box into the segmentation large model to obtain an initial segmentation mask;

[0039] The semantic verification unit is used to calculate the semantic similarity between the image region corresponding to the current segmentation mask and the functional region description using a pre-trained visual-language alignment model; if the semantic similarity reaches a preset threshold or the current iteration number has reached a preset iteration limit, the capture pose output unit is triggered; otherwise, the correction prompt generation unit is triggered.

[0040] The correction suggestion generation unit is used to generate correction suggestion points based on the difference type between the current segmentation mask and the bounding box, input the correction suggestion points and the bounding box into the segmentation model for re-segmentation, obtain the updated segmentation mask, increment the iteration count by 1, and then return to the semantic verification unit.

[0041] The grasping pose output unit is used to generate the grasping pose based on the final output segmentation mask combined with depth data.

[0042] The present invention has the following advantages and effects compared with the prior art:

[0043] (1) This invention combines the semantic understanding capability of the visual language big model with the pixel-level segmentation capability of the segmentation big model. By calculating semantic similarity to perform multimodal semantic consistency verification, it can promptly detect and intercept erroneous masks caused by the output deviation of the visual language big model or the inaccurate segmentation of the segmentation big model, and prevent them from entering the grasping pose generation stage, thereby greatly improving the reliability of the grasping system and the accuracy of the segmentation results.

[0044] (2) The present invention adopts an adaptive correction mechanism when the verification fails: when the semantic similarity fails the verification, the corresponding correction prompt point is generated according to the difference type between the current segmentation mask and the bounding box (oversegmentation, undersegmentation or multiple candidate box ambiguity), which drives the segmentation large model to re-segment, forming a closed loop iterative optimization. High-quality masks can be obtained automatically without manual intervention, which improves the adaptability of the capture system to complex scenes.

[0045] (3) This invention compares the semantic similarity between the functional region description and the segmentation mask image region during semantic consistency verification. Based on whether the semantic similarity reaches a preset threshold, it determines whether to directly output the current segmentation mask or trigger a correction: when the semantic similarity meets the threshold, the current segmentation mask is considered aligned with the functional region and is directly used for grasping pose generation; when the semantic similarity does not meet the threshold, correction prompts are generated to drive the large segmentation model to re-segment, gradually bringing the new segmentation mask closer to the functional region description until the semantic similarity meets the threshold. This mechanism automatically aligns the segmentation results with the task-related functional components, thereby guiding the grasping pose generation within that functional region, avoiding subsequent operation failures due to improper grasping positions, and achieving accurate matching between task semantics and grasping actions. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance according to the present invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0048] Example 1

[0049] like Figure 1 As shown, this embodiment details the overall process of the robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance of the present invention. The method includes the following steps:

[0050] Step S1: Multimodal perception and initial inference

[0051] First, an RGB image of the target object is acquired using an RGB camera, while user natural language commands are captured using a microphone. The audio is then transcribed into text using a recognition model. The RGB image and the transcribed text commands are then packaged and input into a Vision-Language Model. This invention directly utilizes existing pre-trained Vision-Language Models (such as GPT-4V, LLaVA, etc.) and drives the Vision-Language Model to complete semantic understanding and visual reasoning through cue word engineering.

[0052] Specifically, the system adds a preset system prompt template, such as: "Please locate the object in the image that matches the 'user text instruction' and output its bounding box in the format [x_min, y_min, x_max, y_max], along with a brief description of the object's functional region." Based on its cross-modal alignment capability (i.e., joint understanding of images and text), the visual language large model outputs: a coarse bounding box B of the target object and a description of its functional region. This description guides subsequent segmentation to focus on task-related object parts.

[0053] Step S2, Initial Segmentation

[0054] The RGB image and bounding box B obtained in step S1 are input into the Segment Anything Model. This invention directly uses a pre-trained version of the Segment Anything Model (e.g., SAM). The Segment Anything Model uses the bounding box B as a box cue to extract the edges of the most salient object within the box, obtaining an initial segmentation mask, denoted as M0.

[0055] Specifically, the visual language big model excels at high-level semantic understanding, overcoming the limitations of traditional visual models in recognizing complex semantics and providing a rough location of the target (boundary box B); while the segmentation big model possesses extremely strong pixel-level low-level edge perception capabilities. Relying on the region defined by the visual language big model, the segmentation big model only needs to extract edges within the bounding box. The combination of the two solves the problems of insufficient semantic understanding or inaccurate edge segmentation when used alone.

[0056] Step S3: Multimodal semantic consistency verification

[0057] Performing multimodal semantic consistency verification includes the following sub-steps:

[0058] S3.1 Masking and Region of Interest Extraction:

[0059] The current segmentation mask (initially M0, and subsequently the segmentation mask updated after each iteration) is multiplied pixel by pixel with the original RGB image (i.e., Hadamard product). All background pixels outside the current segmentation mask are set to zero, retaining only the object region. Then, the region is rectangularly cropped to generate candidate object images.

[0060] S3.2 Spatial projection of multimodal features:

[0061] Spatial projection of multimodal features is performed using pre-trained visual-language alignment models (such as contrastive language-image pre-training), which maps candidate object images and functional region descriptions to the same high-dimensional feature space.

[0062] The candidate object image is input into the image encoder of the vision-language alignment model to obtain the image feature vector V. img ;

[0063] The functional region description generated by the large visual language model in step S1 is input into the text encoder of the same model to obtain the text feature vector V. text ;

[0064] S3.3, Semantic Similarity Calculation:

[0065] Calculate the image feature vector V img With text feature vector V text Cosine similarity in high-dimensional feature space:

[0066] ,

[0067] Among them, V img V is the image feature vector extracted by the visual-language alignment model for the image region corresponding to the segmentation mask. text The functional region is described by the text feature vector extracted by the visual-language alignment model.

[0068] S3.4 Conditional Judgment

[0069] If the semantic similarity S ≥ the preset threshold θ (θ is preferably 0.6 to 0.9, and 0.75 in this embodiment), or the current iteration count has reached the preset iteration limit (the iteration limit is preferably 3 to 5 times in this embodiment), then the verification is determined to be passed or the iteration is terminated, the current segmentation mask is output as the final mask, and the process jumps to step S5; otherwise, the verification is determined to be failed, and the process proceeds to step S4.

[0070] Step S4: Difference Analysis and Adaptive Hint Correction

[0071] When semantic consistency verification fails, the system enters a difference analysis and adaptive prompt correction phase. Based on the geometric topological differences between the current segmentation mask M and the predicted bounding box B of the visual-language large model, the following strategies are adopted.

[0072] (1) Oversegmentation suppression strategy (when the segmentation mask is determined to contain background noise)

[0073] Decision criterion: Calculate the difference region R diff =MB. If the difference set region R diff If the pixel area is greater than the preset area threshold, it is judged as oversegmentation.

[0074] Negative prompt point generation:

[0075] If the difference region R diff If it contains pixels, then for the difference set region R diffPerform connected component analysis. Use a connected component labeling algorithm (such as Two-Pass or seed filling) to count all discrete blocks, selecting the block with the most pixels as the largest connected component. Calculate the geometric centroid of this connected component: first calculate the zeroth moment M. 00 and first moment M 10 M 01 Then, using the formula x=M 10 / M 00 y=M 01 / M 00 Obtain the centroid coordinates and use this centroid as the negative cue point.

[0076] If the difference region R diff If the current segmentation mask M contains no pixels, then perform morphological dilation on it (e.g., call OpenCV's cv2.dilate function, using a 5×5 elliptic kernel). Subtract the dilated mask from the original mask to obtain the ring-shaped region at the mask's edge as the new R. diff Then, the geometric centroid of the annular region is extracted as a negative cue point.

[0077] Feedback correction: Input the negative cue point together with the original bounding box B into the segmentation model to suppress the outward expansion of the mask boundary.

[0078] (2) Undersegmentation expansion strategy (when the segmentation mask is determined to be missing part of the object)

[0079] Decision criterion: Calculate the region R inside bounding box B that is not covered by mask M. miss =BM. If region R miss If the area of ​​a segment is greater than the preset threshold for the total area of ​​segment B, it is considered undersegmented.

[0080] Positive cue point generation: Recognition region R miss Find the largest connected component in the region and calculate its distance transformation graph (using OpenCV's `cv2.distanceTransform` function). Select a local maximum point of the distance transformation as a positive cue point. This point is located at the very center of the missing region and can effectively guide the segmentation model to grow towards that region.

[0081] Feedback correction: Use the positive cue point and the original bounding box B together as new cue input to segment the large model.

[0082] (3) Multi-candidate box ambiguity resolution strategy (when the visual language large model outputs multiple candidate boxes)

[0083] Judgment condition: If the output of the visual language big model in step S1 is parsed and more than one bounding box is detected in the array (number of candidate boxes > 1), then the parallel segmentation of multiple candidate boxes and semantic reordering will be automatically performed.

[0084] Parallel inference: Instead of point sampling, it obtains the center coordinates of all candidate bounding boxes, i.e., the geometric center (x, y) of each candidate box. center ,y center These are integrated into a batch tensor and input into the large segmentation model all at once. The large segmentation model utilizes the parallel computing capabilities of the GPU to simultaneously output multiple independent masks corresponding to multiple center points, forming an independent mask set.

[0085] Semantic reordering: Input the image region (after cropping and scaling) corresponding to each independent mask in the set into the image encoder of the vision-language alignment model, and input the functional region description into the text encoder. Calculate the semantic similarity between each mask and the functional region description. Select the mask with the highest similarity as the final output and proceed to step S5.

[0086] (4) Iteration mechanism

[0087] The generated correction cue point (negative or positive) is used together with the original bounding box B as a new cue. Return to step S2 to re-segment, and increment the iteration count by 1. Then, perform the verification in step S3 again. Repeat this process until the jump condition in step S3.4 is met.

[0088] Step S5: Capture and generate pose

[0089] Based on the final output segmentation mask, combined with depth image data (obtained from an RGB-D camera or stereo matching), perform the following operations:

[0090] 2D mask to 3D point cloud mapping: The final segmentation mask is mapped back to the original image coordinates, and the local 3D point cloud of the target object is generated by combining the depth image.

[0091] Six-DOF grasping pose calculation: The six-DOF grasping pose (i.e., three degrees of freedom for position plus three degrees of freedom for attitude) is calculated from the 3D point cloud using a grasping pose detection network (such as GraspNet, GG-CNN, etc.).

[0092] Perform the grasping action: Send the calculated grasping pose to the robotic arm controller to control the robotic arm to perform the grasping operation.

[0093] The following example, using the scenario of a stapler sticking to a mouse pad, will be used to specifically illustrate the method of the present invention.

[0094] Scenario description: A black stapler is placed on the desktop, with its bottom resting on a black mousepad. The user issues a voice command through the microphone: "Give me that stapler."

[0095] Step 1: Initial Inference of the Multimodal Perception and Visual Language Large Model

[0096] The system receives a voice command, converts it into the text "Hand me that stapler," and simultaneously, an RGB camera captures a global image. The image and text are then input into a large visual language model, along with a preset system prompt template.

[0097] Because both the stapler and the mousepad are black and stacked together, the visual language model is prone to output bias, incorrectly treating them as a single unit and mistaking the mousepad for the stapler's base. Consequently, the visual language model outputs a huge bounding box B0 that not only includes the stapler but also the entire mousepad below it. Simultaneously, based on the task "hand me," the visual language model infers the functional area description as "the main body of the stapler (excluding the base)."

[0098] Step 2: Initial Segmentation

[0099] The bounding box B0 is used as input for segmenting the large model. After receiving the bounding box, the large model generates the first segmentation mask M1. Since the bounding box includes the mousepad, and the large model has difficulty distinguishing connected regions with similar colors and textures, the segmentation mask M1 covers both the stapler and most of the mousepad, which is considered oversegmentation.

[0100] Step 3: Multimodal semantic consistency verification

[0101] Extract the image region corresponding to M1 (including the stapler and mouse pad), and perform spatial projection of multimodal features using a pre-trained vision-language alignment model: input the candidate object image into the image encoder to obtain the image feature vector V. img Input the functional area description "the main body of the stapler" into the text encoder to obtain the text feature vector V. text At this point, the current iteration count is 1, and the preset iteration limit is 3 to 5 times. Since S=0.5 is lower than the preset threshold of 0.75 and has not reached the iteration limit, the verification fails, and the correction step is initiated.

[0102] Step 4: Difference Analysis and Adaptive Hint Correction

[0103] Analysis revealed that the segmentation mask M1 significantly exceeded the actual area of ​​the target object (stapler), and the area exceeding the mask (mouse pad) had a regular rectangular shape. A large rectangular protruding area (mouse pad portion) was identified at the bottom of the segmentation mask, indicating oversegmentation.

[0104] Furthermore, to confirm whether the protruding area is indeed background noise rather than part of the stapler, the system crops the area and inputs it into a large visual-language model or a visual-language alignment model, providing query instructions (e.g., "Does this area have a metallic texture or mechanical structure?") or calculating the similarity score between the image of the area and preset feature text (e.g., "metallic texture"). Since the mousepad is made of cloth and lacks metallic or mechanical features, the similarity score is below the set threshold, confirming that the area does not belong to the stapler body and is identified as background noise.

[0105] Next, calculate the geometric centroid of the convex region and sample a negative cue point P at that centroid. neg (x, y). Then the original bounding box B0 and the negative cue point P. neg (x,y) are re-input into the large segmentation model, and the iteration count is incremented by 1 (the iteration count is now 2).

[0106] Step 5: Iterative Segmentation

[0107] After receiving a negative cue point, the segmentation model's internal attention mechanism moves away from the local region where that point is located and re-executes inference. The output mask undergoes inward boundary contraction in the corresponding region, automatically severing the erroneous adhesion between the stapler and the mousepad, generating a second-round segmentation mask M2. Segmentation mask M2 only covers the stapler body and no longer includes the mousepad.

[0108] Step Six: Secondary Verification

[0109] The image region corresponding to M2 and the functional region description "the main body of the stapler" are input again into the visual-language alignment model, and the cosine similarity S is calculated to be 0.9, which is greater than the threshold of 0.75, so the verification passes. M2 is output as the final capture mask.

[0110] Step 7: Capture and Generate Pose

[0111] The segmentation mask M2 is mapped back to the original image coordinates, and combined with the depth image to generate a 3D point cloud of the stapler. A grasping pose detection network (such as GraspNet) is used to calculate the six-degree-of-freedom grasping pose from the point cloud, ultimately controlling the robotic arm to accurately grasp the stapler.

[0112] Through a closed-loop iteration (verification-correction-re-segmentation), the semantic similarity was improved from 0.5 to 0.9, effectively improving the output bias of the visual language model and the inaccurate segmentation of the segmentation model, and achieving accurate grasping of functional areas (stapler body, excluding mouse pad).

[0113] Example 2

[0114] A robot grasping pose detection system based on multimodal semantic consistency verification and functional region guidance includes:

[0115] The visual language large model unit is used to acquire the RGB image of the target object, input the RGB image and user instructions into the visual language large model, and output the bounding box and functional region description of the target object.

[0116] A segmentation large model unit is used to input the bounding box into the segmentation large model to obtain an initial segmentation mask;

[0117] The semantic verification unit is used to calculate the semantic similarity between the image region corresponding to the current segmentation mask and the functional region description using a pre-trained visual-language alignment model; if the semantic similarity reaches a preset threshold or the current iteration number has reached a preset iteration limit, the capture pose output unit is triggered; otherwise, the correction prompt generation unit is triggered.

[0118] The correction suggestion generation unit is used to generate correction suggestion points based on the difference type between the current segmentation mask and the bounding box, input the correction suggestion points and the bounding box into the segmentation model for re-segmentation, obtain the updated segmentation mask, increment the iteration count by 1, and then return to the semantic verification unit.

[0119] The grasping pose output unit is used to generate the grasping pose based on the final output segmentation mask combined with depth data.

[0120] The above embodiments are preferred embodiments of the present invention and are not intended to limit the present invention. Any changes or other equivalent substitutions made without departing from the technical solution of the present invention are included within the protection scope of the present invention.

Claims

1. A robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance, characterized in that, Includes the following steps: S1. Obtain the RGB image of the target object, and input the RGB image and user instructions into the visual language large model, and output the bounding box and functional area description of the target object; S2. Input the bounding box into the segmentation model to obtain the initial segmentation mask; S3. Calculate the semantic similarity between the image region corresponding to the current segmentation mask and the functional region description using a pre-trained visual-language alignment model; If the semantic similarity reaches a preset threshold or the current iteration count reaches a preset iteration limit, then proceed to step S5; Otherwise, proceed to step S4; S4. Generate correction hints based on the difference type between the current segmentation mask and the bounding box. Input the correction hints and the bounding box into the large segmentation model for re-segmentation to obtain the updated segmentation mask. Increment the iteration count by 1, and then return to step S3. S5. Generate the grasping pose based on the final output segmentation mask and depth data.

2. The robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance according to claim 1, characterized in that, In step S3, the semantic similarity is calculated using cosine similarity, and the formula for calculating cosine similarity S is: ， Among them, V img V is the image feature vector extracted by the visual-language alignment model for the image region corresponding to the segmentation mask. text The functional region is described by the text feature vector extracted by the visual-language alignment model.

3. The robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance according to claim 1, characterized in that, The difference types between the segmentation mask and the bounding box include oversegmentation, undersegmentation, and multiple candidate box ambiguity; wherein... When the difference type is oversegmentation, a negative cue point is generated to suppress the expansion of the segmentation mask into the background region; When the difference type is undersegmentation, a positive cue point is generated to guide the segmentation mask to grow towards the missing target region; When the difference type is multi-candidate box ambiguity, perform parallel segmentation and semantic reordering of multi-candidate boxes.

4. The robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance according to claim 3, characterized in that, The generation of negative prompts includes the following steps: Calculate the difference region between the current segmentation mask M and the bounding box B: R diff =MB; If the difference region R diff If it contains pixels, then for the difference set region R diff Perform connected component analysis and extract the geometric centroid of the largest connected component as the negative cue point; If the difference region R diff If the current segmentation mask M contains no pixels, then a morphological dilation operation is performed on the current segmentation mask M to obtain the annular region at the edge of the current segmentation mask as the new difference region R. diff Then, the geometric centroid of the annular region is extracted as the negative prompt point.

5. The robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance according to claim 3, characterized in that, The generation of positive prompts includes the following steps: Calculate the region R inside the bounding box B that is not covered by the current segmentation mask M. miss =BM; In the region R miss Identify the connected component with the largest area and calculate the distance transformation graph of the connected component; The local maxima of the distance transformation graph are selected as the positive prompts.

6. The robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance according to claim 3, characterized in that, The parallel segmentation and semantic reordering of multiple candidate boxes includes the following steps: When the visual language big model outputs multiple candidate bounding boxes, the center of each candidate bounding box is used as the initial cue, and independent masks corresponding to each candidate bounding box are generated in parallel to obtain an independent mask set. The image region corresponding to each independent mask in the set of independent masks and the functional region description are input into the visual-language alignment model to calculate semantic similarity. Select the mask with the highest semantic similarity as the final output and proceed to step S5.

7. The robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance according to claim 1, characterized in that, The preset threshold value ranges from 0.6 to 0.9, and the preset number of iterations ranges from 3 to 5.

8. The robot grasping pose detection method based on multimodal semantic consistency verification and functional region guidance according to claim 1, characterized in that, In step S5, generating the grasping pose specifically includes: generating a 3D point cloud of the target object based on the final output segmentation mask and combining it with the depth image; and calculating a six-degree-of-freedom grasping pose from the 3D point cloud using a grasping pose detection network.

9. A robot grasping pose detection system based on multimodal semantic consistency verification and functional region guidance, characterized in that, include: The visual language large model unit is used to acquire the RGB image of the target object, input the RGB image and user instructions into the visual language large model, and output the bounding box and functional region description of the target object. A segmentation large model unit is used to input the bounding box into the segmentation large model to obtain an initial segmentation mask; The semantic verification unit is used to calculate the semantic similarity between the image region corresponding to the current segmentation mask and the functional region description using a pre-trained visual-language alignment model. If the semantic similarity reaches a preset threshold or the current iteration count has reached a preset iteration limit, the pose capture output unit is triggered; otherwise, the correction prompt generation unit is triggered. The correction suggestion generation unit is used to generate correction suggestion points based on the difference type between the current segmentation mask and the bounding box, input the correction suggestion points and the bounding box into the segmentation model for re-segmentation, obtain the updated segmentation mask, increment the iteration count by 1, and then return to the semantic verification unit. The grasping pose output unit is used to generate the grasping pose based on the final output segmentation mask combined with depth data.

Images