Target detection model training method based on dialogue guidance, and grabbing pose generation method and device

Abstract

The application provides a target detection model training method based on dialogue guidance, a grasping pose generation method and device. A sample set containing scene images and multi-round dialogues is constructed, and feedback categories are labeled for the guidance prompts and target detection box coordinates in the multi-round dialogues. In the fine-tuning stage, the cross-entropy loss of the generated sentence and the guidance prompt is used to optimize the dialogue generation capability of the model. The classification loss of the feedback as the guidance prompt or the detection box coordinates is used to make the model learn the decision-making capability of the output type. The detection box prediction loss is used to improve the positioning accuracy of the detection box, so as to obtain a visual language model that can understand vague intentions, actively clarify and accurately position targets. Based on the detection box output by the model, point cloud reconstruction and occlusion completion are performed in combination with the depth image information to generate complete object point clouds, and the optimal grasping pose is generated through a grasping strategy network. The application solves the problem that the existing model cannot handle vague instructions, and improves the grasping success rate of the mechanical arm in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of robot control and embodied intelligence technology, and in particular to a dialogue-guided target detection model training method, a grasping pose generation method, and an apparatus. Background Technology

[0002] In the fields of robotics and embodied intelligence, object recognition and grasping are key technologies for enabling robots to interact with their environment, and their performance directly affects the practicality of robots in scenarios such as home services and industrial collaboration. Interactive object grasping (IOG), as an important technology in this field, combines limited user input, such as clicks and bounding boxes, with computer vision technology to guide models to quickly generate accurate object segmentation masks, thereby achieving the localization and grasping of target objects. IOG technology demonstrates good performance in scenarios where users clearly know the specific item they need and can clearly express it.

[0003] However, with technological advancements and increasing practical application demands, user-robot interaction scenarios are becoming increasingly complex. In many cases, users may not know for sure whether specific items in their surroundings meet their needs, and can only express their underlying intentions through implicit natural language. For example, a user might simply say "I want to drink water," without knowing if there's a water cup nearby. Furthermore, objects are often partially obscured in the scene, which places higher demands on the robot's recognition and grasping capabilities.

[0004] Existing IOG (In-App Purchase) technologies typically require a defined context to function effectively; users must know what specific item they need. If users lack this knowledge and can only express vague linguistic intentions, such as "I want water" without knowing if a cup is available, the effectiveness of existing IOG methods is significantly reduced. In such cases, the robot needs to understand the user's intent, inferring the need for water from semantics, and identifying items within its field of vision that fulfill this need, such as a cup or beverage.

[0005] Furthermore, existing technologies have significant shortcomings when dealing with occluded objects. When an object is partially occluded, models based on fixed-category recognition struggle to accurately identify the target object, leading to a higher grasping failure rate. For example, in complex environments, if the target object is occluded by other objects, existing models may fail to accurately identify the complete outline and position of the target object, thus failing to generate an effective grasping pose.

[0006] Therefore, existing IOG technology has obvious limitations in handling users' implicit intentions and occluded objects, and cannot meet the high requirements of robots for object recognition and grasping in complex interaction scenarios. This limits the intelligence level and practicality of robots in practical applications, and there is an urgent need for a new technology that can effectively solve these problems to improve the robot's interaction capabilities and grasping success rate. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a dialogue-guided target detection model training method, a grasping pose generation method and apparatus, which solve the problems of low understanding of user implicit intent and insufficient recognition of occluded objects in existing IOG technology.

[0008] One aspect of the present invention provides a method for training a dialogue-guided object detection model, the method comprising the following steps:

[0009] A training sample set is constructed, wherein each sample in the training sample set contains a multi-turn dialogue for recognizing a specified target in a sample scene image. The multi-turn dialogue contains user instructions and guidance prompts in the form of alternating natural language questions and answers, as well as the coordinates of the target detection box as the final feedback result. Corresponding feedback categories are added as labels to the guidance prompts and the final feedback result in the multi-turn dialogue.

[0010] The training sample set is used to fine-tune a multimodal visual language model comprising a language layer, an intermediate layer, and a visual layer. The multimodal visual language model is a Qwenvl series model. During fine-tuning, the user command is used as input, and a first prediction result for the guidance prompt or a second prediction result for the specified target detection box position is output. The visual layer is frozen to fine-tune the language layer and the intermediate layer. A cross-entropy loss is constructed based on the first prediction result and the corresponding guidance prompt. The prediction feedback category of the output first or second prediction result is determined, and a feedback category loss is constructed based on the deviation between the prediction feedback category and the feedback category of the next level dialogue for the current user command in the samples. The cross-entropy loss and the feedback category loss are minimized to fine-tune the parameters of the language layer and the intermediate layer for a first set number of rounds. The visual layer is unfrozen, and a target detection box recognition loss is constructed based on the deviation between the second prediction result and the target detection box coordinates. The cross-entropy loss, the feedback category loss, and the target detection box recognition loss are minimized to fine-tune the parameters of all layers of the multimodal visual language model for a second set number of rounds, resulting in a dialogue-guided target detection model.

[0011] In some embodiments of the present invention, in the multi-turn dialogue of each sample, the alternating question-and-answer of the user instruction and the guidance prompt is configured to progressively guide the clear target through at least three levels of dialogue turns, the three levels including: a first-turn dialogue for determining the target type, an intermediate-turn dialogue for determining the target attributes, and a final-turn dialogue for finally confirming the target object.

[0012] In some embodiments of the present invention, the process of unfreezing the visual layer adopts a back-to-forward progressive unfreezing strategy, first unfreezing the network layers in the visual layer that are close to the output, and then unfreezing them sequentially in the gradient backpropagation direction until all visual layers participate in training.

[0013] In some embodiments of the present invention, the target detection box recognition loss adopts the intersection-union loss function, and the calculation formula is:

[0014] ;

[0015] in, Represents the intersection and union loss value; This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. This represents the target detection box corresponding to the second prediction result; This represents the coordinates of the target detection box in the sample.

[0016] In some embodiments of the present invention, the method further includes, during the process of unfreezing the visual layer and performing a second set round of parameter fine-tuning on all layers of the multimodal visual language model, the following:

[0017] Introducing a loss based on bounding box deviation, the calculation formula is as follows:

[0018] ;

[0019] in, This represents the loss value indicating the degree of deviation of the detection box; Indicates batch size; This represents the set of coordinates of the target detection box. This represents the coordinates of the top-left corner of the target detection box. This represents the coordinates of the lower right corner of the target detection box; Indicates the first Coordinates in the second prediction result of each sample The predicted value; Indicates the first The coordinates of the target detection box in each sample The true value; Indicates smoothness The loss function is calculated as follows:

[0020] ;

[0021] in, For the The input variable of the function represents the difference between the predicted and true values ​​of the target detection box coordinates; This indicates configurable hyperparameters.

[0022] On the other hand, the present invention also provides a dialogue-guided method for generating object grasping poses for a robotic arm, the method comprising the following steps:

[0023] Obtain the user's natural language expression of the capture command, and simultaneously acquire color and depth images of the target scene through visual sensing devices;

[0024] Using the grabbing command and the color image as input, the target detection model in the dialogue-guided target detection model training method responds to the grabbing command in a multi-level manner to guide the clear grabbing target, and then identifies the grabbing target in the color image and outputs the corresponding detection box coordinates.

[0025] Based on the detection box coordinates, the target region is extracted from the depth image and a mask is generated; based on the color image, the depth image, and the operating parameters of the visual sensing device, an initial 3D point cloud is reconstructed from the target region through back projection; the operating parameters include the camera's focal length and principal point coordinates; a supporting plane is fitted from the initial 3D point cloud using a planar detection algorithm, and the supporting plane normal vector is obtained; based on the mask, the depth image, and the supporting plane, the initial 3D point cloud is filtered to obtain an initial object point cloud with the background point cloud removed;

[0026] By analyzing the depth abruptness features in the depth image information, the occlusion contours in the initial object point cloud are identified; based on the occlusion contours, new point cloud data is generated by outward expansion to complete the volume and generate a complete object point cloud.

[0027] The complete object point cloud is input into the grasping strategy network to generate multiple candidate grasping poses of the preset grasping component; the spatial angle between the gripper approach direction and the upward-facing normal vector of the support plane for each candidate grasping pose is calculated; based on the spatial angle, the multiple candidate grasping poses are filtered according to preset rules to output the optimal grasping pose.

[0028] In some embodiments of the present invention, the grasping strategy network also outputs the original score of the candidate grasping pose;

[0029] Based on the spatial angle, the multiple candidate grasping poses are filtered according to preset rules, including:

[0030] Candidate grasping poses whose spatial angle exceeds a preset range are eliminated; for the selected candidate grasping poses, a preset penalty coefficient is introduced to calculate a penalty value based on the size of the spatial angle, and the penalty value is deducted from the corresponding original score to obtain an optimized score; wherein, the penalty value is the product of the preset penalty coefficient and the spatial angle; the grasping pose with the highest optimized score is selected as the output.

[0031] In some embodiments of the present invention, filtering the initial 3D point cloud based on the mask, the depth image, and the supporting plane to obtain an initial object point cloud with background point cloud removed includes:

[0032] A first mask for localization is generated based on a portion of the detection box coordinates; a second mask for filtering valid depth points is generated for the depth image; a third mask for filtering desktop point clouds is generated based on the supporting plane; the initial 3D point cloud is filtered based on the first mask, the second mask, and the third mask to obtain the initial object point cloud.

[0033] In some embodiments of the present invention, occlusion contours in the initial object point cloud are identified by analyzing depth abrupt change features in the depth image information, including:

[0034] Traverse the valid pixels in the depth image whose depth values ​​are within a preset range, and determine the pixels that meet the following conditions as edge points on the occlusion contour: the depth value of the adjacent pixel directly above the pixel in the image coordinate system is zero, and there is at least one other pixel with a non-zero depth value among the neighboring pixels within a preset range around the pixel.

[0035] In some embodiments of the present invention, the spatial angle between the approach direction of the gripper of each candidate grasping pose and the upward-facing normal vector of the support plane is calculated, wherein the approach direction of the gripper is defined by the first column vector of the grasping pose rotation matrix output by the grasping strategy network, and before the step of calculating the spatial angle, the direction of the normal vector of the support plane is determined, and if the normal vector of the support plane is not upward-facing, a flipping operation is performed.

[0036] On the other hand, the present invention also provides a robotic arm object grasping pose generation device based on dialogue guidance and visual reasoning, including a processor, a memory, and a computer program or instructions stored in the memory. The processor is used to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the above method.

[0037] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0038] This invention provides a dialogue-guided object detection model training method, a grasping pose generation method, and an apparatus. By constructing a training sample set containing multi-turn dialogues and bounding box annotations, category labels are added to the user's guidance prompts and the final object detection box coordinates in the multi-turn dialogues. A phased fine-tuning strategy is adopted: first, the visual layer is frozen to optimize the language layer and intermediate layer; then, the visual layer is unfrozen for overall fine-tuning. On the one hand, the cross-entropy loss of generated sentences and guidance prompts enables the visual language model to learn how to provide prompts; on the other hand, the classification loss of whether the feedback belongs to guidance prompts or bounding box coordinates enables the visual language model to learn when further guidance prompts are needed or when to output a bounding box after confirming the grasping object. Finally, the prediction loss of the bounding boxes guides the visual language model to improve object detection accuracy, ultimately resulting in a visual language model that can understand the user's ambiguous intentions, proactively initiate multi-turn dialogues to clarify, and accurately locate the target object. This effectively solves the fundamental defects of existing interactive grasping systems that rely on explicit references and cannot handle semantically ambiguous commands.

[0039] Furthermore, a complete scheme for generating poses for pre-defined component grasping is provided. This scheme acquires target scene information through a visual sensing device, uses a trained target detection model to identify the grasping target and outputs the coordinates of the detection box; then, based on the detection box coordinates, it extracts the target region from the depth image, reconstructs the initial 3D point cloud using back projection and mask filtering techniques, and uses a planar detection algorithm to fit the supporting plane and normal vector; specifically, it identifies occlusion contours by analyzing abrupt changes in the depth image and performs point cloud expansion and completion on the occluded region to generate a complete object point cloud, effectively solving the problem of missing point clouds caused by object occlusion in a single viewpoint.

[0040] Furthermore, in the grasping pose generation stage, physical constraints are introduced to optimize and filter grasping strategies. By calculating the spatial angle between the approach direction of the grasping gripper and the normal vector of the supporting plane, unreasonable grasping poses are filtered out, and candidate strategies are penalized and reordered according to the size of the angle, ensuring that the output optimal grasping pose conforms to the constraints of the real physical scene, thereby significantly improving the grasping success rate and system usability.

[0041] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0042] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0043] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0044] Figure 1 This is a flowchart illustrating the dialogue-guided target detection model training method according to an embodiment of the present invention.

[0045] Figure 2 This is a flowchart illustrating the dialogue-guided robotic arm object grasping pose generation method according to another embodiment of the present invention.

[0046] Figure 3 This is a technical flowchart illustrating the method for generating object grasping pose of a robotic arm based on dialogue guidance and its implementation method, as described in another embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0048] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0049] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0050] Existing interactive object grasping (IOG) technology allows users to guide robots to grasp specific objects through explicit commands such as clicking and selecting boxes, achieving significant results in ideal scenarios. However, this technology still faces serious challenges in practical deployment, with limitations mainly manifested in the following three aspects: 1) Rigid intent understanding, unable to handle ambiguous commands: Existing IOG technology heavily relies on explicit referential information about the target object provided by the user. When the user can only express implicit, task-level intents without specifying a concrete object, the model, lacking dialogue reasoning and contextual understanding capabilities, cannot associate abstract needs with physical objects in the scene, leading to interaction interruption and task failure. 2) Fragile visual perception models, insufficient ability to handle occlusion: Existing models struggle to infer the geometric features of occluded areas from visible parts, resulting in inaccurate target recognition, poor segmentation mask quality, and consequently, increased grasping planning failure rates. 3) One-way passive interaction mechanism, lacking collaborative confirmation: Existing technologies mostly adopt a single interaction mode of "one-time command-execution." The model lacks the ability to proactively initiate multi-turn dialogues to clarify, confirm, or obtain additional information. This makes the entire system less tolerant of faults and unable to reach a consensus gradually through cyclical interaction, as is the case with human collaboration.

[0051] In view of this, one aspect of the present invention provides a method for training a dialogue-guided object detection model, such as... Figure 1 As shown, the method includes the following steps S101~S102:

[0052] S101: Construct a training sample set. Each sample in the training sample set contains a multi-turn dialogue for recognizing a specified target in a sample scene image. The multi-turn dialogue contains user instructions and guidance prompts in the form of alternating natural language questions and answers, as well as the coordinates of the target detection box as the final feedback result. Add corresponding feedback categories as labels to the guidance prompts and the final feedback result in the multi-turn dialogue.

[0053] S102: Fine-tuning a multimodal visual language model containing a language layer, intermediate layer, and visual layer using a training sample set. The multimodal visual language model is a Qwenvl series model. During fine-tuning, user commands are used as input, and the output is either a first prediction result for the guidance prompt or a second prediction result for the specified target detection box position. The language layer and intermediate layer are fine-tuned by freezing the visual layer. A cross-entropy loss is constructed based on the first prediction result and the corresponding guidance prompt. The prediction feedback category of the output first or second prediction result is determined, and a feedback category loss is constructed based on the deviation between the prediction feedback category and the feedback category of the next level dialogue for the current user command in the sample. The parameters of the language layer and intermediate layer are fine-tuned for a first set number of rounds by minimizing the cross-entropy loss and the feedback category loss. The visual layer is unfrozen, and a target detection box recognition loss is constructed based on the deviation between the second prediction result and the target detection box coordinates. The parameters of all layers of the multimodal visual language model are fine-tuned for a second set number of rounds by minimizing the cross-entropy loss, the feedback category loss, and the target detection box recognition loss, resulting in a dialogue-guided target detection model.

[0054] In step S101, the core task is to build structured training samples for subsequent model fine-tuning. The purpose is to teach the model how to reason and locate the target in the sample scene image through multi-round dialogue interaction.

[0055] Specifically, each training sample in the training sample set contains three core elements: a sample scene image, a multi-turn dialogue corresponding to a specified target in the image, and the coordinates of the specified target in the image.

[0056] The above sample scene images serve as the visual background for multi-turn dialogues and specified target recognition actions, and contain one or more targets to be identified.

[0057] In some embodiments, the above sample scene images can be captured using a depth camera and saved as RGB type images.

[0058] The multi-turn dialogues corresponding to the specified targets in the images are the core of the training sample set. They include user instructions and guidance prompts in the form of alternating natural language questions and answers. The purpose is to simulate the complete human-computer collaboration process, including: the user makes a vague request - the model actively asks to clarify the intent - the user supplements information - the model finally confirms and executes the task.

[0059] In some embodiments, the alternating question-and-answer format, which includes user instructions and guidance prompts, is configured to progressively guide the user toward a specific goal through at least three levels of dialogue rounds, including: an initial dialogue round for determining the type of the goal, intermediate dialogue rounds for refining the attributes of the goal, and a final dialogue round for finally confirming the goal object.

[0060] In the initial dialogue, the training samples provided vague instructions containing the user's initial intent, indicating the general type of the target. For example, a user instruction might be: "I need a tool."

[0061] In the intermediate rounds of dialogue, the training samples can include multiple rounds of question-and-answer interactions. Each round of interaction pre-constructs a guiding response from the model to the ambiguous instructions in the first round of dialogue. These responses progress step by step, simulating a proactive inquiry process, aiming to guide the user to supplement more discriminative attribute information to continuously narrow down the candidate target range. For example, the model's guiding prompt might be: "Do you want the red screwdriver, the yellow pliers, or the black wrench?" It should be noted that the options mentioned in the model's guiding prompt are all pre-set based on the attributes of real objects existing in the corresponding sample scene images.

[0062] In the final round of dialogue, the training samples provide explicit selection instructions from the user based on the aforementioned multi-round options, used to confirm the final target object. For example, the user instruction might be: "Yellow pliers."

[0063] After the final round of dialogue is completed, the training samples provide the coordinates of the detection boxes corresponding to the final target object, which serve as the final output of the entire multi-round dialogue process.

[0064] To clearly guide the model in generating appropriate responses at different stages of a dialogue, a corresponding feedback category label was added to each response made by the model in the training samples. Adding feedback category labels decomposes the complex dialogue reasoning task into two distinct sub-tasks: response type determination and content generation. This approach of adding feedback category labels allows the model to clearly grasp the rhythm and structure of the dialogue, significantly improving training efficiency and output accuracy.

[0065] In some embodiments, when the model's response is a guided dialogue, an extension is added before the response text. <ask>Tags; when the model's response is the final feedback result, add a tag before the coordinates of the output bounding box. <bbox>Label.

[0066] In step S102, the multimodal visual language model is fine-tuned using the training sample set constructed in step S101. The fine-tuning process employs a phased strategy, aiming to enable the model to learn to reason step by step through multiple rounds of dialogue and accurately locate the target. This invention uses the Qwenvl series model as the basic model for training, which includes a visual layer, a language layer, and an intermediate layer connecting the two.

[0067] The fine-tuning process is divided into two stages:

[0068] In the first stage, the visual layer parameters of the model are frozen, and only the parameters of the language layer and intermediate layers are optimized. The user commands and corresponding sample scene images from the training samples are used as input to the model, which outputs two types of prediction results: the first prediction result is the prediction of the content of the guidance prompt text, and the second prediction result is the prediction of the coordinates of the detection box.

[0069] The cross-entropy loss is constructed by comparing the deviation between the first prediction output of the model and the corresponding real guidance prompt text in the training samples. By minimizing this cross-entropy loss, the model is trained to learn the ability to generate more reasonable and accurate dialogue content.

[0070] In some embodiments, the calculation principle of cross-entropy loss is as follows:

[0071] ;

[0072] in, Indicates batch size; Indicates the sequence length; Indicates the first The sample at the th The actual labels at each location are encoded using one-hot encoding. Indicates the first The sample at the th The probability distribution of the token predicted at each location.

[0073] In some embodiments, to significantly reduce the optimization difficulty of the cross-entropy loss function and guide the model to quickly grasp the decision logic of dialogue and detection, special identifiers can be introduced during data preprocessing. Specifically, when the model's response content is a guiding dialogue, a special identifier is added before the response text. <ask>Special identifier; when the model's response is the final feedback result, add the following before the coordinates of the output bounding box: <bbox>Special identifiers. (The above) <ask>Special identifiers and <bbox>Special identifiers serve as structured landmarks in the model's output, providing clear pattern guidance for the model's behavior in different dialogue rounds. This approach decomposes the complex dialogue reasoning task into two relatively independent sub-tasks: "pattern judgment" and "content generation." This allows the model to learn more clearly when to ask questions to clarify intent and when to confirm the target and output the location result, thereby effectively improving the convergence speed of the training process and the interaction accuracy and structure of the final model output.

[0074] Based on the deviation between the predicted feedback category corresponding to the first or second prediction result output by the model and the feedback category of the next level dialogue for the current user command in the training samples, a feedback category loss is constructed. By minimizing this feedback category loss, the model is trained to learn the ability to determine whether to output a guiding question or the detection box coordinates in the current dialogue state.

[0075] By minimizing the weighted sum of cross-entropy loss and feedback category loss, the parameters of the language layer and intermediate layer are fine-tuned in the first set round.

[0076] In the second stage, the visual layer parameters of the model are unfrozen, and all layers of the model are fine-tuned as a whole. The object detection box recognition loss is constructed based on the deviation between the second prediction result and the corresponding real object detection box coordinates in the training samples. By minimizing the weighted sum of the object detection box recognition loss, cross-entropy loss, and feedback category loss, the parameters of the visual layer, language layer, and intermediate layer are fine-tuned and updated for the second set number of rounds. Finally, an object detection model that can understand dialogue intent, accurately output guidance prompts, and precisely locate targets is obtained.

[0077] In some embodiments, the process of unfreezing the visual layers adopts a back-to-forward progressive unfreezing strategy, first unfreezing the network layers in the visual layers closest to the output, and then unfreezing them sequentially in the gradient backpropagation direction until all visual layers participate in training.

[0078] In some embodiments, the target detection box recognition loss is composed of a weighted sum of the intersection and union loss function and the detection box deviation loss, in order to collaboratively optimize the model's visual localization capability.

[0079] Specifically, the formula for calculating the intersection-union loss function is:

[0080] ;

[0081] in, Represents the intersection and union loss value; This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. This represents the target detection box corresponding to the second prediction result; This represents the target detection box in the sample.

[0082] In some embodiments, the specific calculation process for the intersection-union ratio is as follows:

[0083] (1) Define the target detection box as Format, in which This represents the coordinates of the top-left corner of the target detection box. Let the coordinates of the lower right corner of the target detection box be denoted as . The corresponding coordinates are represented as , The corresponding coordinates are represented as ;

[0084] (2) Calculate the width and height of the intersection region using the following formula:

[0085] ;

[0086] ;

[0087] in, This represents the width of the horizontal overlap between the predicted bounding box and the ground truth bounding box. This indicates the height of the vertical overlap between the predicted bounding box and the ground truth bounding box.

[0088] (3) Calculate the area of ​​intersection. The formula is:

[0089] ;

[0090] (4) Calculate the areas of the predicted bounding box and the ground truth bounding box respectively. The calculation formula is:

[0091] ;

[0092] ;

[0093] in, Indicates the area of ​​the predicted frame; Represents the actual area of ​​the bounding box.

[0094] (5) Calculate the area of ​​the union of sets. The formula is:

[0095] ;

[0096] (6) Calculate the intersection-union ratio, the formula is:

[0097] ;

[0098] in, The range of values ​​is A value of 1 indicates that the predicted bounding box completely overlaps with the ground truth bounding box; a value of 0 indicates that the predicted bounding box does not overlap with the ground truth bounding box.

[0099] The intersection-union loss function mentioned above directly measures the degree of overlap between the predicted bounding box and the ground truth bounding box, guiding the target detection bounding box model to improve its localization accuracy.

[0100] The formula for calculating the loss due to detection box deviation is:

[0101] ;

[0102] in, This represents the loss value indicating the degree of deviation of the detection box; Indicates batch size; This represents the set of coordinates of the target detection box. This represents the coordinates of the top-left corner of the target detection box. This represents the coordinates of the bottom right corner of the target detection box; Indicates the first Coordinates in the second prediction result of each sample The predicted value; Indicates the first The coordinates of the target detection box for each sample The true value; Indicates smoothness The loss function is calculated as follows:

[0103] ;

[0104] in, for The input variable of the function represents the difference between the predicted and actual values ​​of the target detection box coordinates; This indicates configurable hyperparameters.

[0105] The loss due to the deviation of the detection box mentioned above is achieved by smoothing the coordinate deviation. The calculation effectively improves the model's ability to perform basic regression on the coordinates of the detection box.

[0106] The formula for calculating the object detection box recognition loss is:

[0107] ;

[0108] in, This represents the loss in object detection bounding box recognition; Represents the intersection and union loss function; This indicates the loss due to the deviation of the detection frame; This is the weighting coefficient, and its value can be between 1 and 3.

[0109] To achieve accurate localization and overcome the gradient bottleneck of plain text output in regression tasks, this invention also improves the basic model structure and designs corresponding training and inference methods.

[0110] Specifically, in some embodiments, during model training, a custom loss calculation process is used to coordinate and optimize the dialogue generation and object detection bounding box generation tasks. This process first extracts key features for coordinate regression from the model's forward propagation results before calculating the total model loss. Specifically, this is achieved by locating specific identifiers within the token sequence output by the model. <bbox>The hidden states corresponding to that position are obtained. These hidden states are highly semantic and contextualized feature vectors obtained through multi-layer self-attention computation of the Transformer, providing a rich semantic foundation for subsequent coordinate regression.

[0111] In some embodiments, a dedicated regression prediction head (BBOX-head) can be designed to work in parallel with the model's original text generation head. The regression prediction head consists of two fully connected layers, with an input dimension twice the hidden layer dimension (hidden_size) of the multimodal visual language model, and an output dimension of 4, corresponding to the four coordinate values ​​of the object detection box. This design decouples the continuous coordinate regression task from the discrete text generation task, thereby achieving direct and efficient prediction of numerical coordinates.

[0112] In some embodiments, a multimodal feature fusion method based on an attention mechanism is employed to generate the input features for the regression prediction head. The purpose of this method is to deeply fuse linguistic semantics rich in task intent with visual information containing object details. The specific process includes steps S1021-S1024:

[0113] Step S1021: After forward computation of the model, locate the token in the output token sequence. <bbox>A special identifier is used, and its corresponding hidden layer state vector is extracted as a query semantic vector. This vector encodes the contextual semantics of the current multi-turn dialogue, clearly identifying the target object to be located.

[0114] Step S1022: Extract the hidden layer state set V corresponding to all visual tokens from the visual feature sequence obtained by encoding the sample scene images. The visual feature sequence is usually located between model-specific visual identifiers, such as <|vision_start|> and <|vision_end|>, and carries the structured visual information of the scene.

[0115] Step S1023: Using the semantic vector obtained in step S1021 as the query, and the visual feature set V obtained in step S1022 as the key and value, input it into a learnable attention pooling layer. After calculation, the output is a pooled visual context vector, calculated as follows:

[0116] ;

[0117] in, Represents the query vector The projection vector obtained after linear transformation; Represents the key vector The projection vector obtained after linear transformation; Value vector The projection vector obtained after linear transformation; The implicit dimensions representing the attention mechanism; This represents a learnable scaling factor.

[0118] Step S1024: The semantic vector obtained in step S1021 is concatenated with the visual context vector obtained in step S1023 to form a composite feature vector that integrates linguistic semantics and visual information. This composite feature vector is then input into the regression prediction head to regress the final bounding box coordinates. The length of the composite feature vector is... , consisting of a length of The semantic vector and its length are It is composed of visual context vectors concatenated together.

[0119] In some embodiments, a modular optimization strategy is employed to stabilize the multi-task training process and preserve the model's original language capabilities. During training, a learning rate is set for the newly added regression prediction head and attention pooling layer, independent of and typically higher than the model's main parameters. This allows the gradients of the intersection-union loss function and the bounding box deviation loss to preferentially and efficiently optimize the newly added modules that they directly affect, while avoiding excessive interference and damage to the pre-trained model's learned language understanding and generation capabilities by these location-related loss gradients in the early stages of training.

[0120] In some embodiments, an interruption generation mechanism is employed during the model inference phase to adapt to the workflow of the regression prediction head. Specifically, a custom stopping criterion function is set, which is triggered when the model outputs during the generation process. <bbox>Immediately after the special identifier, subsequent text generation is interrupted. Then, the multimodal feature fusion process described above, such as steps S1021-S1024, is executed. The final bounding box coordinates are calculated using the weighted regression prediction head, and these coordinates are converted into a string and appended to the model output. For example, from the user's perspective, the model output is " <ask>[Guidance and prompts] <bbox>[x1, y1, x2, y2]”, thus providing a response that includes complete dialogue interaction and precise positioning results.

[0121] On the other hand, the present invention also provides a dialogue-guided method for generating the pose of a robotic arm for object grasping, such as... Figure 2 As shown, the method includes the following steps S201~S205:

[0122] S201: Obtain the user's natural language expression of the capture command, and simultaneously acquire the color image and depth image of the target scene through the visual sensing device.

[0123] S202: Taking the grabbing command and color image as input, the target detection model in the dialogue-guided target detection model training method responds to the grabbing command in a multi-level manner to guide the clear grabbing target. Then, it identifies the grabbing target in the color image and outputs the corresponding detection box coordinates.

[0124] S203: Extract the target region from the depth image based on the detection box coordinates and generate a mask; reconstruct the initial 3D point cloud by back projection based on the color image, depth image and operating parameters of the visual sensing device; the operating parameters include the camera's focal length and principal point coordinates; fit the supporting plane from the initial 3D point cloud using a planar detection algorithm and obtain the supporting plane normal vector; filter the initial 3D point cloud based on the mask, depth image and supporting plane to obtain the initial object point cloud after removing the background point cloud.

[0125] S204: By analyzing the depth abrupt change features in the depth image information, the occlusion contours in the initial object point cloud are identified; based on the occlusion contours, new point cloud data is generated by expanding outwards to complete the volume and generate a complete object point cloud.

[0126] S205: Input the complete object point cloud into the grasping strategy network to generate multiple candidate grasping poses of the preset grasping components; calculate the spatial angle between the gripper approach direction and the normal vector of the upward-facing support plane for each candidate grasping pose; filter the multiple candidate grasping poses according to preset rules based on the spatial angle, and output the optimal grasping pose.

[0127] In step S201, the user's natural language grasping instruction is obtained. This instruction is typically a task-oriented, vague description rather than a specific designation of the target object. Simultaneously with obtaining the user's grasping instruction, the on-site deployed visual sensing device immediately captures a color image and a depth image of the current scene.

[0128] The color image, based on the RGB color channels, provides information about the appearance and texture of the target object, serving as a visual reference and recognition basis for the multimodal visual language model in subsequent steps. For example, the information contained in the color image might be color visual information such as yellow wire strippers, red screwdrivers, and blue cups.

[0129] A depth image is a special type of image registered with a color image. The grayscale value of each pixel directly represents the actual distance between the corresponding object point in the scene and the camera lens, usually measured in millimeters. Essentially, a depth image is a two-dimensional intensity map of a three-dimensional scene. Pixels with higher brightness or larger values ​​indicate that the object is farther from the camera, while pixels with lower brightness or smaller values ​​indicate that the object is closer to the camera. By reading the value of each pixel in the depth image, the position of each point on the object's surface in three-dimensional space can be accurately calculated, thereby reconstructing the three-dimensional geometry and structure of the scene. The depth image obtained in this step is mainly used for subsequent steps to perform 3D point cloud reconstruction and volume completion of the target object.

[0130] For example, the depth image stores the precise physical distance from the object surface to the camera lens corresponding to each pixel in the scene. For example, it may include the depth measurement values ​​of all visible objects such as the desktop plane, the surface of the wire stripper, the outline of the screwdriver, and the outer wall of the cup.

[0131] In step S202, the natural language capture command and color image obtained in step S201 are used as joint inputs and fed into the object detection model obtained in step S102. This model performs a target object recognition task based on multi-turn dialogue. It generates guided responses to achieve multi-turn interaction with the user by semantically understanding the ambiguous commands input by the user and combining visual feature analysis of the color image. Through multiple question-and-answer sessions, the user's intent is gradually narrowed down and clarified. Once the model is certain that the target object has been identified, it accurately identifies the target object in the input color image and outputs the coordinates of its corresponding detection box.

[0132] In step S203, based on the detection box coordinates obtained in step S202, depth data of the corresponding region is extracted from the depth map obtained in step S201, and a first mask is generated. This first mask marks the region inside the detection box as the valid region and the region outside as the background region, which is used to initially distinguish the target object from the surrounding environment.

[0133] For each pixel within the first mask range, its corresponding depth value is taken. Combined with the camera focal length and principal point coordinates in the operating parameters of the visual sensing device, the three-dimensional coordinates of the pixel in the camera coordinate system are calculated according to the back projection formula. Then, the two-dimensional image coordinates are transformed into three-dimensional camera coordinates. By traversing all pixels within the first mask, the initial three-dimensional point cloud of the target area is obtained.

[0134] In some embodiments, the formula for calculating back projection is:

[0135] ;

[0136] ;

[0137] ;

[0138] in, These are the two-dimensional coordinates of a pixel in the depth image; For this pixel point The corresponding depth value; Principal coordinates represent the pixel position corresponding to the intersection of the camera's optical axis and the camera's imaging sensor plane; For the camera and Focal length in direction, in pixels; The coordinates of the calculated 3D point in the camera coordinate system.

[0139] After obtaining the initial 3D point cloud through back projection, the supporting plane is fitted from the initial 3D point cloud using a plane detection algorithm.

[0140] In some embodiments, the plane detection algorithm can employ a random sampling consensus algorithm. This algorithm calculates candidate planes by iteratively sampling three points, counts the number of interior points whose distance from a point to the plane is less than a set threshold, and finally selects the plane with the most interior points as the supporting plane, and calculates its normal vector. This method can effectively overcome the interference of abnormal points such as objects and noise in the target scene and fit the supporting plane of the target object.

[0141] The initial 3D point cloud can be filtered based on the generated first mask, depth image, and supporting plane of the target object, specifically including steps S2031~S2033:

[0142] Step S2031: Generate a second mask for the depth image to filter valid depth points. This second mask marks pixels with depth values ​​within the valid range as valid, and removes invalid points and noise points with depth values ​​exceeding the preset range, ensuring that subsequent processing is based on reliable depth data.

[0143] Step S2032: Generate a third mask for filtering out the point cloud of the supporting plane. Specifically, the plane supporting equation is obtained by fitting a plane detection algorithm. By calculating the distance from each point in the initial 3D point cloud to the plane, points with a distance less than a set threshold are marked as desktop point clouds. The desktop point clouds are then filtered out from the initial 3D point cloud to obtain the aforementioned third mask.

[0144] Step S2033: Take the intersection of the first mask, the second mask and the third mask to generate a comprehensive filter mask and filter the initial 3D point cloud to obtain the initial object point cloud after removing the background, invalid points and the supporting plane point cloud.

[0145] In step S204, for any missing parts in the initial object point cloud obtained in step S203, especially the camera's blind spot and occluded areas, occlusion contour recognition and point cloud completion processing are performed to generate a complete object point cloud as input for subsequent grasping strategies. Generating a complete object point cloud includes steps S2041~S2043:

[0146] Step S2041: Analyze the depth abrupt change features in the depth image information to identify the occlusion contours in the initial object point cloud. Specifically, traverse the region in the depth image corresponding to the initial object point cloud and detect edge pixels where the depth value changes significantly. If the depth value of the pixel directly above it in the image coordinate system is zero, and there is at least one other pixel with a non-zero depth value among the neighboring pixels within a preset range around the pixel, then the pixel is determined to be an occlusion boundary point. All these boundary points together constitute the occlusion contours of the object.

[0147] Step S2042: Based on the identified occlusion contours, volume completion is achieved by generating new point cloud data through outward expansion. Specifically, the process involves expanding outward along a preset direction away from the camera with a preset thickness to generate a series of new point cloud data. This process is equivalent to geometrically inferring and filling in the missing parts of the object, thereby constructing a visually complete 3D object model that can be used for grasping and planning.

[0148] Step S2043: Merge the newly generated point cloud data with the initial object point cloud to generate a complete object point cloud.

[0149] In step S205, the complete object point cloud obtained in step S204 is converted into multiple candidate grasping poses executable by the preset grasping component. These candidate poses are then filtered based on preset rules, and the optimal grasping pose is finally output. Specifically, this includes steps S2051 to S2053:

[0150] Step S2051: Input the complete object point cloud into the grasping strategy network. Based on the geometric features of the point cloud, the network automatically generates multiple candidate grasping poses of the preset grasping components and the original scores of the corresponding grasping pose quality.

[0151] Step S2052: Calculate the spatial angle between the approach direction of the gripper and the upward-facing normal vector of the support plane for each candidate grasping pose. The approach direction of the gripper is defined by the first column vector of the rotation matrix of the grasping pose; the normal vector of the support plane is obtained in step S203 and uniformly adjusted to face upwards.

[0152] Step S2053: Filter candidate grasping poses based on spatial angles. By setting a preset range for spatial angles, candidate grasping poses with spatial angles exceeding the preset range are eliminated. For the selected candidate grasping poses, a preset penalty coefficient is introduced to calculate the penalty value based on the size of the spatial angle, and the penalty value is deducted from the corresponding original score to obtain the optimized score. The penalty value is the product of the preset penalty coefficient and the spatial angle. The grasping pose with the highest optimized score is selected as the output.

[0153] In some embodiments, the spatial angle can be preset to a range of 0 to 75 degrees to ensure that the gripper grasps the target object from above, thus avoiding the gripper grasping from below the support plane.

[0154] On the other hand, the present invention also provides a robotic arm object grasping pose generation device based on dialogue guidance and visual reasoning, including a processor, a memory, and a computer program or instructions stored in the memory. The processor is used to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the above method.

[0155] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0156] The present invention will now be described with reference to a specific embodiment:

[0157] This embodiment provides a dialogue-guided method for generating object grasping poses for a robotic arm and its implementation method, aiming to specifically illustrate the implementation process of the technical solution of the present invention.

[0158] The technical process of this invention is as follows: Figure 3 As shown, its core lies in: utilizing a multimodal large model for object recognition, combining the language and image capabilities of the multimodal model, the model deeply mines the textual information hidden in the dialogue, understands the image, and then realizes the recognition of occluded objects. Furthermore, based on the recognition results and the prior knowledge of the large model, it imagines the characteristics of the occluded objects and generates a reasonable grasping strategy.

[0159] I. Model Fine-tuning Training

[0160] First, the dataset is processed, divided into training and testing sets. The training set is used for fine-tuning the model, and the testing set is used to test the model's capabilities. The base model adopts the Qwenvl series model; for ease of deployment, this invention uses a lightweight version with 3 bytes of parameters. The Qwen2.5vl series model architecture includes: a vision layer, a language layer (LLM), and an intermediate layer (MLP). To clearly observe the fine-tuning effect of each layer, the fine-tuning process is divided into two stages:

[0161] In the first stage, the visual layer is frozen so that it does not participate in the fine-tuning process. The language layer and the intermediate layer are fine-tuned first. The user instructions in the training samples and the corresponding sample scene images are used as the input of the model. Two types of prediction results are output: the first prediction result is the prediction of the content of the guidance prompt text, and the second prediction result is the prediction of the coordinates of the detection box.

[0162] The cross-entropy loss is constructed by comparing the deviation between the first prediction output of the model and the corresponding real guidance prompt text in the training samples. By minimizing this cross-entropy loss, the model is trained to learn the ability to generate more reasonable and accurate dialogue content.

[0163] The formula for calculating the cross-entropy loss is:

[0164] ;

[0165] in, Indicates batch size; Indicates the sequence length; Indicates the first The sample at the th The actual labels at each location are encoded using one-hot encoding. Indicates the first The sample at the th The probability distribution of the token predicted at each location.

[0166] Based on the deviation between the predicted feedback category corresponding to the first or second prediction result output by the model and the feedback category of the next level dialogue for the current user command in the training samples, a feedback category loss is constructed. By minimizing this feedback category loss, the model is trained to learn the ability to determine whether to output a guiding question or the detection box coordinates in the current dialogue state.

[0167] By minimizing the weighted sum of cross-entropy loss and feedback category loss, the parameters of the language layer and intermediate layer are fine-tuned in the first set round.

[0168] In the second stage, after successful fine-tuning of the language layer, the visual layer is unfrozen, and the model is fine-tuned again. The object detection box recognition loss is constructed based on the deviation between the second prediction result and the corresponding real object detection box coordinates in the training samples. By minimizing the weighted sum of the object detection box recognition loss, cross-entropy loss, and feedback category loss, the parameters of the visual layer, language layer, and intermediate layer are fine-tuned and updated for the second set number of rounds. Finally, an object detection model that can understand dialogue intent, accurately output guidance prompts, and precisely locate the target is obtained.

[0169] Fine-tuning and training were performed according to the steps outlined above. After debugging, it was found that the model's learning rate was optimal. The learning effect is best when the value is near the desired value. Furthermore, the learning rate can be dynamically adjusted along the direction of gradient descent.

[0170] III. Training Data Augmentation

[0171] This invention provides enhanced prompts for the original dataset during model fine-tuning training. The dataset contains two parts: dialogue and bounding box recognition. The model needs to determine whether its next response should involve dialogue to supplement information or generate a bounding box after confirming the information. To strengthen this judgment, this invention incorporates special identifiers in the dataset processing. <ask> 、 <bbox>During the data preprocessing stage, the model's responses are divided into two categories. If the model output is in the form of dialogue content, then a prefix is ​​added before it. <ask>Labels: If the model output is in the form of bounding boxes, add a label before it. <bbox>Labels. Adding special identifiers tells the model to classify before generating, enabling the model to recognize that the two responses belong to different categories. It also makes the model's generation structure clearer. If subsequent experiments require adding new category content, such as action generation, simply add the label at the beginning. <act>This allows for the completion of a new classification. Furthermore, when calculating accuracy, it facilitates observation of which categories of responses the model is failing to generate.

[0172] IV. Multimodal Feature Alignment and Performance Verification

[0173] To ensure the model fully integrates with images, this invention incorporates the `image_pad` processing identifier into the dialogue portion of the data. During processing, the `processor` function matches the number of repetitions of this identifier with the number of image features. Inserting this identifier into the dialogue explicitly instructs the model to combine image analysis with the understanding of each line of dialogue.

[0174] To quantify the effectiveness of this invention, the Intersection over Union (IoU) ratio was used as the evaluation metric on a standard test set to compare the recognition accuracy of different models. As shown in the table below, Acc@0.1, Acc@0.5, and Acc@0.9 represent the average accuracy when the IoU thresholds are greater than or equal to 0.1, 0.5, and 0.9, respectively. Experimental results show that this invention significantly outperforms the baseline model Qwen2.5vl and the existing mainstream method PROGrasp at all IoU thresholds. Especially under higher accuracy positioning requirements (such as Acc@0.9), it maintains its leading advantage, verifying that the proposed method has higher recognition accuracy and positioning robustness.

[0175]

[0176] V. Real-world experiments and results analysis

[0177] In a real-world environment, depth cameras from the Intel RealSense series were used for on-site photography. Images were captured using the depth cameras and saved as both RGB and depth maps. The RGB images were used as references for the model, while the depth maps were used for 3D modeling of the objects.

[0178] For example, taking the example of grabbing wire strippers, the initial user dialogue "I want to cut something" indicates that the user needs certain tools for cutting. The model then determines, based on the user's semantic information, whether to continue outputting dialogue or to provide detection box results. The content of the user-model dialogue is as follows:

[0179] User: I want to cut something.

[0180] Model: <ask>Do you want the pliers in front of the tape?

[0181] User: Either is fine.

[0182] Model: <bbox>bbox_2d:[x1,y1,x2,y2]

[0183] VI. Image Processing

[0184] The camera used in this invention is the Intel RealSense D435 depth camera. It requires loading the official pyrealsense2 library. First, the camera is initialized. This invention uses... The image was captured at a specific resolution. Both the RGB color and depth channels were enabled. When the model began a new dialogue, the camera was used to capture footage of the experimental environment. After the dialogue, the model provided bounding box information for object recognition, which was then used to mark the object's location on the image.

[0185] Simultaneously, depth map information is obtained through the depth channel. A mask image is created based on the detection bounding box results. The principle of the mask image is to filter out non-target image parts using Boolean values ​​True and False. A two-dimensional array with the same resolution is defined, and the positions of pixels within the detection bounding box are set to True, while the rest are False. All images obtained in this part are saved in PNG format to preserve data accuracy and facilitate obtaining clear geometric features during subsequent point cloud reconstruction.

[0186] VII. Point Cloud Reconstruction and Filtering

[0187] 3D point cloud reconstruction is performed using relevant code from the Open3D library. The saved RGB and depth images are read to obtain the intrinsic parameter information of the depth camera. , , , . and It refers to the camera's focal length. and These are the principal point coordinates, i.e., the intersection of the optical axis and the image plane. The `createfrom_color_and_depth` function combines the RGB and depth maps, and the resulting variables, along with camera intrinsics, are fed into relevant functions to obtain the point cloud map.

[0188] The point cloud is processed using a mask filter. Combined with a saved mask image, which extracts RGB and depth information while preserving the bounding box portion, other masks are set, such as: 1) a valid depth mask: points with depth greater than 0 are set to True, others to False. 2) a desktop filtering mask: desktop point clouds are set to False, others to True (desktop point cloud filtering is explained in the next section). 3) an extreme depth filtering mask: points with depths greater than a certain threshold are set to False, others to True. The intersection of these masks yields the final object point cloud. This operation aims to prevent desktop point cloud interference during subsequent crawling strategy generation, thus avoiding the crawling strategy from focusing solely on the desktop.

[0189] Desktop point cloud selection: The desktop is obtained by applying the RANSAC algorithm to fit the maximum plane. The RANSAC algorithm is an algorithm that finds the plane that is matched by the majority of points through random sampling. The plane is obtained through this algorithm. And from this, the desktop normal vector is obtained. Once we have the planar expression for the desktop, we can calculate the distance from a point to the plane: Set a threshold; points with a distance less than the threshold are considered desktop point clouds.

[0190] 8. Edge Completion

[0191] Due to the limitations of single-view perspective, depth information about the back of an object cannot be obtained, resulting in a hollow back surface during point cloud reconstruction, which is highly detrimental to grasping strategies. This invention uses edge recognition combined with thickness supplementation to improve the grasping of missing areas. First, edges are identified using depth maps. Analysis of depth data reveals significant changes in depth values ​​at edges, such as a sharp drop from a large value to 0, showing a clear numerical boundary. Based on this characteristic, a detection method is defined. Edges are considered to meet the following conditions:

[0192] 1) Based on the camera's perspective, the back of the graphic to be processed is located away from the camera, i.e., at the top edge of the object. Therefore, if a pixel's y-axis position is one position higher than the camera, it must be 0.

[0193] 2) The area around a pixel is not all zero. There must be an object point cloud near the edge, so the neighborhood cannot be all zero, otherwise it may be noise.

[0194] Based on the above criteria, a filtering function is constructed. This yields the object's edge and provides an appropriate edge thickness.

[0195] IX. Crawling Strategy Generation

[0196] GraspNet is used to generate crawling strategies from the completed point cloud. It's a network that automatically calculates the geometric features of the point cloud, generates and sorts the crawling strategies, displaying the results in descending order of score. GraspNet is called to calculate these strategies on the completed point cloud, resulting in multiple candidate crawling strategies.

[0197] 10. Crawling Strategy Filtering

[0198] The generated grasping strategies are calculated solely by GraspNet. These strategies only consider the object's shape and do not take into account real-world physical constraints. For example, due to the presence of a desktop, the gripper cannot grasp from below the desktop, making such strategies unreasonable. Therefore, it is necessary to calculate the angle between the gripper direction and the desktop normal vector for each grasping strategy, and then select the appropriate strategy. The desktop normal vector has already been calculated and is... The gripping strategy returned by GraspNet primarily contains two values: Translation and Rotation. The value related to the gripper direction is Rotation, which has three columns representing the three axes. The first column, the x-axis, represents the approach direction of the gripper, i.e., the desired gripper orientation. The spatial angle is calculated using the approach direction vector and the desktop normal vector. It's important to ensure the desktop normal vector points upwards; if it points downwards, a flip operation is needed. The correct state is when the gripper orientation forms an obtuse angle with the desktop normal vector, meaning the gripper grips downwards. A permissible angle range is then set. This method filters out all grabbing strategies outside the specified range and applies a penalty. Here, the penalty coefficient is set to 0.1. The penalty coefficient is multiplied by the angle to obtain the deduction value. The final score of a successful grabbing strategy is the original score minus the deduction. Then, Graspnet's built-in score sorting function is used to optimally rank these strategies. The purpose of this method is to encourage the gripper to grab from directly above as much as possible, and to penalize tilted grabbing.

[0199] Because Graspnet automatically sorts the results to the optimal order, only the first result needs to be selected after filtering, which is then used for the robotic arm's final grasping strategy.

[0200] In summary, this invention provides a dialogue-guided object detection model training method, a grasping pose generation method, and an apparatus. By constructing a training sample set containing multi-turn dialogues and bounding box annotations, category labels are added to the user's guidance prompts and the final object detection box coordinates in the multi-turn dialogues. A phased fine-tuning strategy is adopted: first, the visual layer is frozen to optimize the language layer and intermediate layer; then, the visual layer is unfrozen for overall fine-tuning. On the one hand, the cross-entropy loss of generated sentences and guidance prompts enables the visual language model to learn how to provide prompts; on the other hand, the classification loss of whether the feedback belongs to guidance prompts or bounding box coordinates enables the visual language model to learn when further guidance prompts are needed or when to output a bounding box after confirming the grasping object. Finally, the prediction loss of the bounding boxes guides the visual language model to improve object detection accuracy, ultimately resulting in a visual language model that can understand the user's ambiguous intentions, proactively initiate multi-turn dialogues to clarify, and accurately locate the target object. This effectively solves the fundamental defects of existing interactive grasping systems that rely on explicit references and cannot handle semantically ambiguous commands.

[0201] Furthermore, a complete scheme for generating poses for pre-defined component grasping is provided. This scheme acquires target scene information through a visual sensing device, uses a trained target detection model to identify the grasping target and outputs the coordinates of the detection box; then, based on the detection box coordinates, it extracts the target region from the depth image, reconstructs the initial 3D point cloud using back projection and mask filtering techniques, and uses a planar detection algorithm to fit the supporting plane and normal vector; specifically, it identifies occlusion contours by analyzing abrupt changes in the depth image and performs point cloud expansion and completion on the occluded region to generate a complete object point cloud, effectively solving the problem of missing point clouds caused by object occlusion in a single viewpoint.

[0202] Furthermore, in the grasping pose generation stage, physical constraints are introduced to optimize and filter grasping strategies. By calculating the spatial angle between the approach direction of the grasping gripper and the normal vector of the supporting plane, unreasonable grasping poses are filtered out, and candidate strategies are penalized and reordered according to the size of the angle, ensuring that the output optimal grasping pose conforms to the constraints of the real physical scene, thereby significantly improving the grasping success rate and system usability.

[0203] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0204] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0205] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0206] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / bbox> < / ask> < / act> < / bbox> < / ask> < / bbox> < / ask> < / bbox> < / ask> < / bbox> < / bbox> < / bbox> < / bbox> < / ask> < / bbox> < / ask> < / bbox> < / ask>

Claims

1. A method for training a dialogue-guided object detection model, characterized in that, The method includes the following steps: Construct a training sample set, wherein each sample in the training sample set contains a multi-turn dialogue for recognizing a specified target in a sample scene image, wherein the multi-turn dialogue contains user instructions and guidance prompts in the form of alternating question and answer in natural language, and the coordinates of the target detection box as the final feedback result; Add corresponding feedback categories as tags to the guidance prompts and final feedback results in the multi-turn dialogue; The training sample set is used to fine-tune a multimodal visual language model containing a language layer, an intermediate layer, and a visual layer. The multimodal visual language model is a Qwenvl series model. During the fine-tuning process, the user command is used as input and the output is either a first prediction result for the guidance prompt or a second prediction result for the location of the specified target detection box. The visual layer is frozen to fine-tune the language layer and the intermediate layer. A cross-entropy loss is constructed based on the first prediction result and the corresponding guidance prompt. Determine the predicted feedback category of the first or second prediction result output; construct a feedback category loss based on the deviation between the predicted feedback category and the feedback category of the next level dialogue for the current user command in the sample; minimize the cross-entropy loss and the feedback category loss to fine-tune the parameters of the language layer and the intermediate layer for a first set number of rounds; unfreeze the visual layer; construct a target detection box recognition loss based on the deviation between the second prediction result and the target detection box coordinates; minimize the cross-entropy loss, the feedback category loss, and the target detection box recognition loss to fine-tune the parameters of all layers of the multimodal visual language model for a second set number of rounds to obtain a dialogue-guided target detection model. In each sample's multi-turn dialogue, the alternating question-and-answer format of the user instructions and the guidance prompts is configured to gradually guide the user towards a clear objective through at least three levels of dialogue rounds. These three levels include: a first-round dialogue for determining the target type, intermediate-round dialogues for determining the target attributes, and a final-round dialogue for ultimately confirming the target object.

2. The method for training a dialogue-guided target detection model according to claim 1, characterized in that, The process of unfreezing the visual layers adopts a back-to-forward progressive unfreezing strategy. First, the network layers closest to the output in the visual layers are unfrozen, and then they are unfrozen sequentially in the gradient backpropagation direction until all visual layers participate in training.

3. The method for training a dialogue-guided target detection model according to claim 1, characterized in that, The target detection box recognition loss uses the intersection-union loss function, calculated as follows: ； in, Represents the intersection and union loss value; This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. This represents the target detection box corresponding to the second prediction result; This represents the coordinates of the target detection box in the sample.

4. The method for training a dialogue-guided target detection model according to claim 3, characterized in that, The method further includes, during the process of fine-tuning the parameters of all layers of the multimodal visual language model in a second set round after unfreezing the visual layer: Introducing a loss based on bounding box deviation, the calculation formula is as follows: ； in, This represents the loss value indicating the degree of deviation of the detection box; Indicates batch size; This represents the set of coordinates of the target detection box. This represents the coordinates of the top-left corner of the target detection box. This represents the coordinates of the lower right corner of the target detection box; Indicates the first Coordinates in the second prediction result of each sample The predicted value; Indicates the first The coordinates of the target detection box in each sample The true value; Indicates smoothness The loss function is calculated as follows: ； in, For the The input variable of the function represents the difference between the predicted and true values ​​of the target detection box coordinates; This indicates configurable hyperparameters.

5. A method for generating object grasping pose using a robotic arm based on dialogue guidance, characterized in that, The method includes the following steps: Obtain the user's natural language expression of the capture command, and simultaneously acquire color and depth images of the target scene through visual sensing devices; Using the grabbing command and the color image as input, the target detection model in the dialogue-guided target detection model training method according to any one of claims 1 to 4 performs a multi-level response to the grabbing command to guide the clear grabbing target, and then identifies the grabbing target in the color image and outputs the corresponding detection box coordinates; Based on the detection box coordinates, the target region is extracted from the depth image and a mask is generated; based on the color image, the depth image, and the operating parameters of the visual sensing device, an initial 3D point cloud is reconstructed from the target region through back projection; the operating parameters include the camera's focal length and principal point coordinates; a supporting plane is fitted from the initial 3D point cloud using a planar detection algorithm, and the supporting plane normal vector is obtained; based on the mask, the depth image, and the supporting plane, the initial 3D point cloud is filtered to obtain an initial object point cloud with the background point cloud removed; By analyzing the depth abruptness features in the depth image information, the occlusion contours in the initial object point cloud are identified; based on the occlusion contours, new point cloud data is generated by outward expansion to complete the volume and generate a complete object point cloud. The complete object point cloud is input into the grasping strategy network to generate multiple candidate grasping poses of the preset grasping component; the spatial angle between the gripper approach direction and the upward-facing normal vector of the support plane for each candidate grasping pose is calculated; based on the spatial angle, the multiple candidate grasping poses are filtered according to preset rules to output the optimal grasping pose.

6. The method for generating object grasping pose of a robotic arm based on dialogue guidance according to claim 5, characterized in that, The grasping strategy network also outputs the original score of the candidate grasping pose; Based on the spatial angle, the multiple candidate grasping poses are filtered according to preset rules, including: Candidate grasping poses whose spatial angle exceeds a preset range are eliminated; for the selected candidate grasping poses, a preset penalty coefficient is introduced to calculate a penalty value based on the size of the spatial angle, and the penalty value is deducted from the corresponding original score to obtain an optimized score; wherein, the penalty value is the product of the preset penalty coefficient and the spatial angle; the grasping pose with the highest optimized score is selected as the output.

7. The method for generating object grasping pose of a robotic arm based on dialogue guidance according to claim 5, characterized in that, The initial 3D point cloud is filtered based on the mask, the depth image, and the supporting plane to obtain an initial object point cloud with the background point cloud removed, including: A first mask for localization is generated based on a portion of the detection box coordinates; a second mask for filtering valid depth points is generated for the depth image; a third mask for filtering desktop point clouds is generated based on the supporting plane; the initial 3D point cloud is filtered based on the first mask, the second mask, and the third mask to obtain the initial object point cloud.

8. The method for generating object grasping pose of a robotic arm based on dialogue guidance according to claim 5, characterized in that, By analyzing the depth abrupt change features in the depth image information, the occlusion contours in the initial object point cloud are identified, including: Traverse the valid pixels in the depth image whose depth values ​​are within a preset range, and determine the pixels that meet the following conditions as edge points on the occlusion contour: the depth value of the adjacent pixel directly above the pixel in the image coordinate system is zero, and there is at least one other pixel with a non-zero depth value among the neighboring pixels within a preset range around the pixel.

9. The method for generating object grasping pose of a robotic arm based on dialogue guidance according to claim 5, characterized in that, Calculate the spatial angle between the approach direction of the gripper and the upward-facing normal vector of the support plane for each candidate gripping pose, wherein the approach direction of the gripper is defined by the first column vector of the gripping pose rotation matrix output by the gripping strategy network, and before calculating the spatial angle, the direction of the normal vector of the support plane is determined; if the normal vector of the support plane is not upward-facing, a flipping operation is performed.

10. A robotic arm object grasping pose generation device based on dialogue guidance and visual reasoning, comprising a processor, a memory, and a computer program or instructions stored in the memory, characterized in that, The processor is configured to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the method as described in any one of claims 5 to 7.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 9.

Images