Language-guided robot grasping method and system based on uncertainty full-link penetration
By quantifying the semantic and depth uncertainty of visual language models, a gated probabilistic selector and an adaptive execution strategy are designed, which solves the problem that uncertainty does not penetrate to the execution layer in existing technologies and improves the success rate of robot grasping systems in uncertain scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2026-04-20
- Publication Date
- 2026-08-04
AI Technical Summary
Existing language-guided robot grasping systems have a high failure rate when facing uncertain scenarios such as transparent objects and reflective surfaces. Existing methods fail to effectively propagate the uncertainty of the perception layer to the selection and execution layers, and uncertainty scoring leads to a decrease in selection performance in most scenarios.
The semantic uncertainty of the visual language model is quantified by multiple random forward inferences. The three-dimensional covariance matrix is propagated by combining deep uncertainty. A gated probabilistic selector and an adaptive execution strategy are designed to realize the uncertainty from the perception layer to the execution layer and adaptively adjust the compliant control parameters.
It significantly improves the robustness of grasping in scenarios with difficult perception, such as transparent objects and reflective surfaces, and increases the accuracy of target selection and execution success rate. It also has good versatility and scalability.
Smart Images
Figure CN122066570B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to intelligent robot grasping technology, and more particularly to a language-guided robot grasping method and system based on the end-to-end uncertainty process. Background Technology
[0002] The application of visual language models in robot operation is becoming increasingly widespread. A typical language-guided grasping process is as follows: the user gives a natural language command, the visual language model outputs a two-dimensional pointing point in the image according to the command, the system uses a depth camera to backproject the two-dimensional point into three-dimensional space, generates the grasping pose, and performs the grasping.
[0003] The main problem with existing systems is that each stage treats the input as a deterministic value: the pointing point output by the visual language model is considered precise semantic localization, the depth value is considered precise geometric measurement, the strategy selection is based on deterministic distance metrics, and the execution strategy uses fixed parameters. This deterministic assumption leads to a significant increase in the grasping failure rate when facing transparent objects, reflective surfaces, occluded scenes, and other situations where there is considerable uncertainty in both depth and semantics.
[0004] Some existing works attempt to introduce uncertainty into the perception stage, such as probabilistically modeling depth maps or estimating confidence intervals for object detection results. However, these works have two key limitations: first, uncertainty remains only at the perception layer and does not propagate to the downstream selection and execution stages; second, even when attempting to use uncertainty to guide selection, it often leads to a decline in selection performance in most scenarios because the system fails to determine when the uncertainty score is reliable and when it should revert to the baseline strategy.
[0005] Therefore, there is a need for a language-guided robot grasping method that can carry uncertainty from the perception layer to the selection and execution layers and has the ability to judge credibility. Summary of the Invention
[0006] The purpose of this invention is to provide a language-guided robot grasping method and system based on the end-to-end uncertainty process.
[0007] The technical solution to achieve the purpose of this invention is: a language-guided robot grasping method based on the end-to-end uncertainty process, comprising the following steps:
[0008] S1. Uncertainty Quantification and Propagation in the Perception Layer: Multiple random forward inferences are performed on the visual language model to obtain multiple sets of two-dimensional semantic pointing point samples. A probabilistic mixture model is fitted to the two-dimensional semantic pointing point sample sets to obtain a two-dimensional semantic covariance matrix. Statistical modeling is performed on local regions of the depth image to obtain the depth uncertainty variance. The two-dimensional semantic covariance matrix and the depth uncertainty variance are propagated to three-dimensional space through the Jacobian matrix of the coordinate mapping model to synthesize a three-dimensional total covariance matrix.
[0009] S2. Gated probabilistic target selection: Calculate a joint probability score for each candidate target based on the three-dimensional total covariance matrix as a candidate proposal; extract uncertainty features and use a gating mechanism to determine whether to adopt the probability selection result; when the gating determines that the probability selection is credible, replace the baseline selection result with the probability selection result; otherwise, keep the baseline selection result unchanged.
[0010] S3. Uncertainty-driven adaptive execution: Perform eigenvalue decomposition on the three-dimensional total covariance matrix, adaptively set the compliant control parameters of the corresponding degrees of freedom according to the eigenvalues of each principal axis direction, realize compliant grasping in the direction of high uncertainty, and adjust the grasping approach strategy according to the covariance ellipsoid shape corresponding to the three-dimensional total covariance matrix.
[0011] Furthermore, in step S1, the multiple random forward inferences specifically include: enabling the random sampling mechanism of the visual language model during the inference phase, and setting the sampling temperature parameter. Execute on the same input image and language command Secondary independent forward reasoning Each reasoning operation yields a two-dimensional coordinate point. ,in This forms a two-dimensional semantic pointer sample set. ;
[0012] The process of fitting a probabilistic mixture model to the two-dimensional semantic pointer sample set includes: fitting the sample set using an iterative optimization algorithm. The component probability mixture model yields... The weight of each component mean Covariance Matrix ,in Regularization is applied to degenerate components that occur during the fitting process. When the number of samples of a component is lower than the preset minimum or the minimum eigenvalue of the covariance matrix is lower than the preset lower bound, regularization constraints are applied to the covariance matrix of that component to avoid generating singular covariance matrices.
[0013] Furthermore, in step S1, the process of propagating the two-dimensional semantic covariance matrix and the depth uncertainty variance to the three-dimensional space is as follows:
[0014] The mean of the components with the largest weights Taking a representative point, the corresponding depth value d is obtained from the depth image; a mapping function from two-dimensional pixel coordinates and depth to three-dimensional spatial coordinates is established based on the pinhole coordinate mapping model, and the mapping function is calculated in... Jacobian matrix at the location ,in Given a 3×3 matrix; construct the 2D-depth joint input covariance matrix. The matrix is a block-diagonal matrix, where the diagonal blocks are the two-dimensional semantic covariance matrix and the depth uncertainty variance, respectively. The three-dimensional geometric covariance matrix is calculated using the error propagation formula:
[0015]
[0016] The three-dimensional geometric covariance matrix is weighted and superimposed with the semantic covariances propagated from each component of the probabilistic mixture model to obtain the three-dimensional total covariance matrix. .
[0017] Furthermore, in step S1, the method for obtaining the depth uncertainty variance is as follows:
[0018] Representative point Centered on a depth image, a set of depth values for a local window region is extracted; invalid depth pixels in the set of depth values are filtered out, including pixels with depth values of zero, infinity, or exceeding a preset valid range; the local variance of the depth value set after filtering out invalid pixels is calculated as the depth uncertainty variance. When the number of effective pixels is less than the preset threshold, the statistical variance of the effective depth pixels of the entire image multiplied by the magnification factor is used as a conservative estimate.
[0019] Furthermore, in step S2, the joint probability score is calculated as follows:
[0020] For each candidate grab target in the scene Its three-dimensional position is Calculate the weighted joint negative log-likelihood of the candidate target relative to each component of the probabilistic mixture model after propagation to three-dimensional space:
[0021]
[0022] in For the first The weight of each component, For the first The coordinates of the average of each component after propagation to three-dimensional space For the first The three-dimensional total covariance matrix after propagation of each component The probability density function is a multivariate probability density function; the candidate objective with the smallest joint negative log-likelihood value is selected as the probability selection result.
[0023] Furthermore, in step S2, the gating mechanism specifically includes:
[0024] Uncertainty features are extracted from a set of two-dimensional semantic pointing point samples obtained from multiple inferences. These uncertainty features include at least one of the following: the two-dimensional spatial variance of the sample points, the entropy value of the weights of each component in the probabilistic mixture model, and the trace of the principal component covariance matrix. Material semantic features of the target object are extracted from the language instructions. These material semantic features are used to determine whether the target object belongs to a material category that is difficult to perceive in depth. The uncertainty features and material semantic features are input into a gating decision function. When the output of the gating decision function meets the preset switching conditions, the probabilistic selection result is used to replace the baseline selection result; otherwise, the baseline selection result is maintained.
[0025] Furthermore, the gating decision function is a rule-based interpretable gating function or a learning gating function based on a machine learning classifier:
[0026] When an interpretable gating function is used, the switching condition is: the uncertainty dispersion is lower than a preset dispersion threshold. Furthermore, the joint probability score is better than the baseline score by more than a preset difference threshold. When a learning gating function is used, the label selected with higher probability than the baseline in the historical crawled samples is used as the training target, and a binary classifier is trained as the gating decision function. The input features of the binary classifier include uncertainty features, material semantic features, and the difference between the probability score and the baseline score.
[0027] Furthermore, in step S3, the specific process of adaptively setting the compliance control parameters is as follows:
[0028] For the three-dimensional total covariance matrix Eigenvalue decomposition yields three eigenvalues. and the corresponding feature vectors , , ; in the principal axis coordinate system defined by the eigenvector, along the first The impedance stiffness is set for each principal axis direction as follows:
[0029]
[0030] in Based on the basic parameter values, The modulation coefficient and , For reference eigenvalues, To crop the upper bound, The trimming function transforms the stiffness matrix in the principal axis coordinate system to the robot base coordinate system through a rotation matrix composed of eigenvectors, thus obtaining the translation parameter matrix for multi-degree-of-freedom compliant control.
[0031] Furthermore, step S3 also includes approach direction adjustment and exploratory pre-contact strategies:
[0032] Approach direction adjustment: Taking the direction of the eigenvector e_3 corresponding to the minimum eigenvalue of the covariance ellipsoid as the preferred approach direction, the approach direction of the end effector is adjusted towards... The direction is weighted and deflected, with the deflection weights equal to the largest eigenvalue. and minimum eigenvalue The ratio is positively correlated;
[0033] Exploratory pre-contact: when the trace of the three-dimensional total covariance matrix Exceeding the preset exploration trigger threshold At the same time, before performing the formal grasping, an exploratory pre-contact action is inserted. The exploratory pre-contact action includes tactile detection within a preset range along the direction of the maximum eigenvector of the covariance ellipsoid to obtain contact feedback to correct the target position.
[0034] A system for implementing a language-guided robot grasping method based on end-to-end uncertainty is provided. The system includes:
[0035] The uncertainty quantification module is configured to perform multiple random samplings on the visual language model to obtain a two-dimensional semantic pointing point set, fit a probabilistic mixture model to obtain a two-dimensional semantic covariance matrix, and combine the deep uncertainty variance to obtain a three-dimensional total covariance matrix through Jacobian propagation.
[0036] The gated probability selection module is configured to calculate the joint probability score of each candidate target based on the three-dimensional total covariance matrix, and to determine whether to replace the baseline selection result with the probability selection result through a gating mechanism.
[0037] The adaptive execution module is configured to perform eigenvalue decomposition on the three-dimensional total covariance matrix and adaptively generate compliant control parameters and approach strategies for each degree of freedom.
[0038] The three-dimensional total covariance matrix serves as a unified representation of uncertainty, and is sequentially passed from the uncertainty quantification module to the gated probability selection module and the adaptive execution module, thus achieving full-link integration of uncertainty information.
[0039] Compared with the prior art, the significant advantages of this invention are:
[0040] (1) This invention provides a complete framework for propagating the semantic uncertainty of a visual language model from the perception layer through the selection layer to the execution layer, realizing the end-to-end penetration of uncertainty and solving the problem that traditional methods treat the output and depth value of the visual language model as deterministic inputs and ignore their uncertainty;
[0041] (2) By designing a gated probability selector, this invention extracts uncertainty features and material semantic features to achieve intelligent judgment of the credibility of uncertainty scores, thus solving the problem that the original uncertainty measurement causes the selection accuracy to degrade in most scenarios.
[0042] (3) This invention provides an uncertainty-driven adaptive execution strategy, which generates compliant control parameters adaptively through feature decomposition, and achieves compliant grasping in high uncertainty directions, which significantly improves the grasping robustness in difficult scenarios such as transparent objects and reflective surfaces.
[0043] (4) The method provided by the present invention has good versatility and scalability, and can be applied to a variety of difficult perception scenarios, including transparent objects and reflective objects. It can effectively improve the accuracy of target selection and the robustness of execution. Attached Figure Description
[0044] Figure 1 This is a flowchart of the language-guided robot grasping method and system based on the end-to-end uncertainty proposed in this embodiment of the invention.
[0045] Figure 2 This is a flowchart illustrating the uncertainty quantification and propagation process of the perception layer in an embodiment of the present invention.
[0046] Figure 3 This is a schematic diagram illustrating the working principle of the gated probability selector in an embodiment of the present invention.
[0047] Figure 4 This is a schematic diagram of the mapping between the covariance ellipsoid and stiffness in an embodiment of the present invention.
[0048] Figure 5 This is a schematic diagram illustrating the accuracy of various selection layer schemes in a test scenario according to one embodiment of the present invention.
[0049] Figure 6 This is a comparison chart of the robustness of various execution layer strategies in a test scenario according to one embodiment of the present invention.
[0050] Figure 7 A comparative illustration of selecting crawling targets on the dataset. Detailed Implementation
[0051] This invention proposes a language-guided robot grasping method and system based on end-to-end uncertainty propagation. By propagating the semantic uncertainty of the visual language model from the perception layer through the selection layer to the execution layer, it achieves end-to-end uncertainty propagation. Traditional methods treat the visual language model output and depth values as deterministic inputs, ignoring the inherent uncertainties in semantic direction and depth measurement, leading to high grasping failure rates in scenarios involving transparent objects and reflective surfaces. This invention quantifies the semantic uncertainty of the visual language model through multiple random samplings and quantifies the depth uncertainty through local statistical modeling. Both are jointly propagated to three-dimensional space to obtain a three-dimensional total covariance matrix, which is then used to design a gated probabilistic selector and an adaptive execution strategy.
[0052] The following is a detailed description of the specific steps of the language-guided robot grasping method based on the end-to-end uncertainty of this invention.
[0053] like Figure 1 As shown, the method includes the following steps:
[0054] Step 1: Multiple sampling of scene perception and visual language model:
[0055] The scene is captured using an image acquisition device to obtain an RGB image (I) and a depth map (D). The user inputs natural language commands. Input visual language model, set sampling temperature ,implement Each independent forward inference outputs a two-dimensional pointing coordinate at each inference step. To obtain the sample set In a preferred embodiment, N ≥ 5. In a specific embodiment, a Monte Carlo sampling strategy is used to implement the multiple random forward inferences.
[0056] Step 2, fitting the probabilistic mixture model:
[0057] An iterative optimization algorithm is used to fit a K-component probabilistic mixture model to the sample set S, and the weights of the K components are obtained. mean Covariance Matrix Degenerate components that appear during the fitting process are regularized to avoid generating singular covariance matrices. In a preferred embodiment, the probabilistic mixture model is a Gaussian mixture model, and the iterative optimization algorithm is the expectation-maximization (EM) algorithm.
[0058] Step 3, Deep Uncertainty Modeling:
[0059] A local window is taken centered on the mean of the principal components, and the depth values of the corresponding region in the depth map are extracted. After filtering out invalid depth pixels, the variance of the valid pixel depth values is calculated as the depth uncertainty variance. When the number of effective pixels is insufficient, a conservative estimate is obtained by multiplying the statistical variance of the entire image by the magnification factor.
[0060] Step 4, 2D to 3D covariance propagation:
[0061] Based on the pinhole coordinate mapping model, a mapping function from 2D pixel coordinates and depth to 3D spatial coordinates is established, and the Jacobian matrix is calculated. Construct a 2D-depth joint input covariance matrix. The three-dimensional geometric covariance matrix is calculated using the error propagation formula. The GMM components are propagated separately and then weighted and summed to obtain the total three-dimensional covariance matrix. .
[0062] Step 5, Gated Probabilistic Target Selection:
[0063] Calculate the joint negative log-likelihood score for each candidate target in the scene. Simultaneously, calculate the baseline selection result. Extract uncertainty features and material semantic features, and use a gating mechanism to determine whether to adopt the probabilistic selection result. If the gating determines that the current uncertainty score is reliable, the probabilistic selection result is used to replace the baseline selection; otherwise, the baseline selection result is retained.
[0064] Step 6, Uncertainty-driven adaptive execution:
[0065] Eigenvalue decomposition is performed on the three-dimensional total covariance matrix to obtain eigenvalues and eigenvectors. Impedance stiffness is adaptively generated for each direction based on the eigenvalues, with stiffness reduced to increase compliance in directions with high uncertainty. The approach direction is adjusted according to the principal axis of the covariance ellipsoid. An exploratory pre-contact action is triggered when uncertainty exceeds a threshold. Finally, impedance-controlled grasping is performed using the adjusted parameters.
[0066] like Figure 2 As shown, the flowchart of uncertainty quantification and propagation in the perception layer illustrates how the present invention extracts and fuses uncertainty information from multimodal inputs. The flowchart is divided into three parts: the input layer, the processing layer, and the fusion layer.
[0067] At the input layer, the system receives three types of input: RGB images. The RGB image provides visual information about the scene, the language commands convey the user's grasping intent, and the depth map provides three-dimensional geometric information about the scene.
[0068] At the processing layer, the system processes the input data independently. In the vision-language processing branch, the VLM multiple sampling module receives RGB images and speech commands, and sets the sampling temperature. ,implement Sub-independent forward reasoning, outputting a two-dimensional set of pointer points. The EM algorithm fitting module fits the sample set. conduct The component Gaussian mixture model is fitted, outputting two-dimensional semantic uncertainty. In the depth processing branch, the depth window extraction module extracts information from the depth map. Extract the region of interest, and the depth variance calculation module calculates the depth uncertainty variance.
[0069] At the fusion layer, the propagation and synthesis module receives two-dimensional semantic uncertainty and the pinhole camera model mapping function. And the variance of deep uncertainty, calculated by the Jacobian matrix The matrix is synthesized with the GMM components to output a three-dimensional total covariance matrix. This matrix fully describes the target's positioning uncertainty in three-dimensional space, providing a probabilistic confidence estimate for subsequent decision-making.
[0070] like Figure 3 As shown in the diagram, the working principle of the gated probability selector illustrates how the present invention performs secure target selection based on uncertainty information. The diagram consists of an input layer, a processing layer, a decision layer, and an output layer.
[0071] At the input layer, the system receives three types of basic information: the three-dimensional total covariance matrix, the candidate target locations, and language commands. The three-dimensional total covariance matrix reflects the uncertainty of the target location, the candidate target locations are the set of targets to be evaluated, and the language commands convey the user's high-level semantic needs.
[0072] At the processing layer, the baseline analysis module performs a preliminary evaluation of candidate targets to generate baseline results. The joint likelihood (NLL) scoring module combines the 3D total covariance matrix and the candidate target positions to calculate a score for each candidate target using a joint negative log-likelihood model; a lower score indicates that the target better meets the requirements. The feature extraction module further extracts uncertainty features and material semantic features for subsequent gating decisions.
[0073] At the decision-making level, a gating decision function (rule-based or learning-based) determines whether the current uncertainty score is credible. If the decision is "yes," the minimum objective is adopted as the final choice; if the decision is "no," the baseline result is retained. This design avoids the risk of simply replacing the baseline with uncertainty, achieving safe objective selection.
[0074] like Figure 4 As shown in the diagram, the mapping between the covariance ellipsoid and stiffness illustrates how the present invention adaptively adjusts the impedance to control stiffness based on the uncertainty distribution, revealing the negative correlation between uncertainty and stiffness in a geometrically intuitive way.
[0075] Figure 4The geometry of the covariance ellipsoid is shown, with its major, median, and minor axes corresponding to the directions of maximum, moderate, and minimum uncertainty, respectively. Three orthogonal eigenvectors point in these three directions, and their corresponding eigenvalues quantify the degree of uncertainty in each direction.
[0076] Each eigenvector is accompanied by a spring symbol to the right, which visually represents the stiffness in that direction. The direction with the greatest uncertainty corresponds to low stiffness (compliance), and sparse springs indicate low stiffness; the direction with moderate uncertainty corresponds to moderate stiffness; and the direction with the least uncertainty corresponds to high stiffness, and dense springs indicate high stiffness.
[0077] The formula box on the right provides the formula for calculating adaptive stiffness. This formula indicates that, with... As the reference stiffness, uncertainty The larger the stiffness The smaller the value (compliant), the greater the stiffness (rigid). This adaptive strategy makes the robot more compliant in directions with high uncertainty, and can compensate for positioning errors through passive adaptation, thereby improving the success rate of grasping.
[0078] like Figure 5 As shown in the figure, the accuracy comparison chart of the selection layer methods demonstrates the performance of different target selection methods on the overall sample and transparent subset, and verifies the effectiveness of the gating mechanism through quantitative data.
[0079] The horizontal axis lists four methods: baseline (single inference + nearest neighbor), direct selection of raw joint NLL, interpretable rule gating, and text-material hybrid gating. The vertical axis represents accuracy, ranging from 0.55 to 0.75. White hollow bars represent overall accuracy, and gray solid bars represent accuracy of transparent subsets.
[0080] from Figure 5 The following key findings can be observed: First, directly using the original joint NLL selection leads to accuracy degradation (overall 0.585 < baseline 0.600), validating the risk of simply substituting uncertainty for the baseline. Second, after introducing gating mechanisms, various gating schemes can restore accuracy above the baseline level. Interpretable rule gating performs best on the transparent subset (0.719), while text-material hybrid gating performs best overall (0.625).
[0081] These results demonstrate that the gating mechanism can effectively determine when to trust the uncertainty score, avoiding incorrect choices in high uncertainty scenarios, while preserving the performance improvement brought by uncertainty information.
[0082] like Figure 6 As shown in the figure, the robustness comparison diagram of each strategy in the execution layer illustrates the impact of different adaptive execution strategies on the robustness of fetching, as well as the gain effect in high uncertainty scenarios. Figure 6 It consists of a robustness histogram on the left and a gain comparison chart on the right.
[0083] The left-hand bar chart compares the robustness metrics of four execution strategies: baseline fixed execution (0.4166), approach adjustment only (0.4244), compliant adaptation only (0.5152), and full adaptive execution (0.5241). From... Figure 6 As can be seen, the robustness improvement of the approach adjustment strategy alone is limited, while the compliant adaptive strategy alone brings a significant improvement. The full adaptive execution strategy has the highest robustness, indicating that combining compliant adaptation and approach adjustment can further optimize performance.
[0084] The comparison chart on the right shows the gain of the complete adaptive execution strategy on different subsets. The gain for the transparent subset is +0.1029, and the gain for the high-uncertainty subset is +0.1615. The higher gain for the high-uncertainty subset indicates that the adaptive execution strategy proposed in this invention has more significant value in high-risk scenarios.
[0085] These results validate the effectiveness of the uncertainty-driven adaptive execution strategy: by reducing stiffness in the direction of high uncertainty, the robot can compensate for positioning errors through passive adaptation, thereby significantly improving the grasping success rate.
[0086] like Figure 7 As shown in the diagram, the comparison between uncertainty-driven selection and baseline selection visually demonstrates the performance differences between the two methods through a real-world scenario example. Figure 7 It consists of the dataset scene on the left and two comparison sub-images on the right.
[0087] The dataset scene on the left illustrates the experimental environment: an indoor scene with a blue spherical target object and white markers on the ground. The legend explains the meaning of the four markers: green dots represent real observations, red crosses represent baseline method observations, blue crosses represent observations using the method of this invention, and yellow stars represent visually servoed target points.
[0088] The two comparison subplots on the right show the results of baseline selection (incorrect) and uncertainty-driven selection (correct), respectively. In the baseline selection subplot, the red crosses (baseline method observations) deviate significantly from the green dots (true values), indicating that the baseline method has a large error when observing the target object. In the uncertainty-driven selection subplot, the blue crosses (observations of the proposed method) highly overlap with the green dots (true values), and the blue area around the object indicates the result of uncertainty-driven selection, showing that the proposed method can more accurately estimate the true position of the target object.
[0089] Figure 7This intuitively illustrates the advantages of the uncertainty-driven selection strategy: by making full use of uncertainty information, the method of this invention can achieve more accurate target positioning in complex scenarios, thereby providing a reliable foundation for subsequent crawling and execution.
[0090] Based on the same inventive concept, this invention also provides a language-guided robot grasping system based on the end-to-end uncertainty process, the system comprising:
[0091] The uncertainty quantification module is configured to perform multiple random samplings on the visual language model to obtain a two-dimensional semantic pointing point set, fit a probabilistic mixture model to obtain a two-dimensional semantic covariance matrix, and combine the deep uncertainty variance to obtain a three-dimensional total covariance matrix through Jacobian propagation.
[0092] The gated probability selection module is configured to calculate the joint probability score of each candidate target based on the three-dimensional total covariance matrix, and to determine whether to replace the baseline selection result with the probability selection result through a gating mechanism.
[0093] The adaptive execution module is configured to perform eigenvalue decomposition on the three-dimensional total covariance matrix and adaptively generate compliant control parameters and approach strategies for each degree of freedom.
[0094] The three-dimensional total covariance matrix serves as a unified representation of uncertainty, and is sequentially passed from the uncertainty quantification module to the gated probability selection module and the adaptive execution module, thus achieving full-link integration of uncertainty information.
[0095] The gating mechanism in the gated probability selection module includes a material perception submodule. The material perception submodule parses the material category information of the target object from the language command and uses the material category information as an auxiliary input feature for gating determination. For material categories that are difficult to perceive in depth, such as transparent materials and highly reflective materials, the gating switching conservatism is improved to avoid unreliable uncertainty scores from misleading the selection decision.
[0096] The following is a specific embodiment to further illustrate the feasibility and technical effect of the present invention.
[0097] Example
[0098] In this embodiment, a desktop grabbing scenario is used as the application scenario. The hardware platform includes: a six-degree-of-freedom robotic arm with a parallel gripper mounted on its wrist as an end effector; an RGB-D camera for acquiring color and depth images of the scene; and a main control computer equipped with a GPU for inference computation of the visual language model. In this embodiment, the visual language model adopts the Molmo series model, the probabilistic mixture model adopts a Gaussian mixture model, the iterative optimization algorithm adopts the expectation-maximization algorithm, the coordinate mapping model adopts a pinhole camera model, and the compliant control adopts impedance control. The depth image is directly acquired by the RGB-D camera.
[0099] In this embodiment, a test scenario containing 200 samples was used for verification, with the scenario containing transparent objects accounting for approximately 30%.
[0100] Table 1 shows the accuracy comparison results of each selection layer method on a 200-sample test set.
[0101] Table 1
[0102] method Overall accuracy Transparent subset accuracy Remark Baseline (single inference + nearest distance) 0.600 0.703 Reference Standard Original joint NLL direct selection 0.585 0.609 Degeneration Interpretable rule gating 0.610 0.719 Positive returns Text-material hybrid gating 0.625 0.703 optimal
[0103] Table 2 presents the comparison results of the robustness proxy metrics of each execution layer strategy on a 200-sample test set.
[0104] Table 2
[0105] Strategy overall Transparent subset gain High uncertainty subset gain Baseline fixed execution 0.4166 — — Approach adjustment only 0.4244 — — Compliance Adaptive Only 0.5152 — — Full adaptive execution 0.5241 +0.1029 +0.1615
[0106] As shown in Tables 1 and 2, the target selection accuracy is effectively improved by using the gated probabilistic selection method provided in this embodiment of the invention compared with the baseline scheme. The execution robustness is significantly improved by using the complete adaptive execution strategy, especially in high uncertainty subsets involving transparent objects and other difficult perception scenarios, which verifies the effectiveness of the technical solution of this invention.
Claims
1. A language-guided robot grasping method based on the end-to-end uncertainty process, characterized in that, Includes the following steps: S1. Uncertainty Quantification and Propagation in the Perception Layer: Multiple random forward inferences are performed on the visual language model to obtain multiple sets of two-dimensional semantic pointing point samples. A probabilistic mixture model is fitted to the two-dimensional semantic pointing point sample sets to obtain a two-dimensional semantic covariance matrix. Statistical modeling is performed on local regions of the depth image to obtain the depth uncertainty variance. The two-dimensional semantic covariance matrix and the depth uncertainty variance are propagated to three-dimensional space through the Jacobian matrix of the coordinate mapping model to synthesize a three-dimensional total covariance matrix. S2. Gated probabilistic target selection: Calculate a joint probability score for each candidate target based on the three-dimensional total covariance matrix as a candidate proposal; extract uncertainty features and use a gating mechanism to determine whether to adopt the probability selection result; when the gating determines that the probability selection is credible, replace the baseline selection result with the probability selection result; otherwise, keep the baseline selection result unchanged. S3. Uncertainty-driven adaptive execution: Perform eigenvalue decomposition on the three-dimensional total covariance matrix, adaptively set the compliant control parameters of the corresponding degrees of freedom according to the eigenvalues of each principal axis direction, realize compliant grasping in the direction of high uncertainty, and adjust the grasping approach strategy according to the covariance ellipsoid shape corresponding to the three-dimensional total covariance matrix.
2. The language-guided robot grasping method based on the end-to-end uncertainty process as described in claim 1, characterized in that, In step S1, the multiple random forward inferences specifically include: enabling the random sampling mechanism of the visual language model during the inference phase, and setting the sampling temperature parameter. Execute on the same input image and language command Secondary independent forward reasoning Each reasoning operation yields a two-dimensional coordinate point. ,in This forms a two-dimensional semantic pointer sample set. ; The process of fitting a probabilistic mixture model to the two-dimensional semantic pointer sample set includes: fitting the sample set using an iterative optimization algorithm. The component probability mixture model yields... The weight of each component mean Covariance Matrix ,in Regularization is applied to degenerate components that occur during the fitting process. When the number of samples of a component is lower than the preset minimum or the minimum eigenvalue of the covariance matrix is lower than the preset lower bound, regularization constraints are applied to the covariance matrix of that component to avoid generating singular covariance matrices.
3. The language-guided robot grasping method based on the end-to-end uncertainty process according to claim 2, characterized in that, In step S1, the two-dimensional semantic covariance matrix and the depth uncertainty variance are propagated to the three-dimensional space through the Jacobian matrix of the coordinate mapping model, specifically as follows: The mean of the components with the largest weights For a representative point, obtain the corresponding depth value d from the depth image; Based on a coordinate mapping model, a mapping function is established from two-dimensional pixel coordinates and depth to three-dimensional spatial coordinates. The mapping function is then calculated. Jacobian matrix at the location ,in Given a 3×3 matrix; construct the 2D-depth joint input covariance matrix. The matrix is a block-diagonal matrix, where the diagonal blocks are the two-dimensional semantic covariance matrix and the depth uncertainty variance, respectively. The three-dimensional geometric covariance matrix is calculated using the error propagation formula: ； The three-dimensional geometric covariance matrix is weighted and superimposed with the semantic covariances propagated from each component of the probabilistic mixture model to obtain the three-dimensional total covariance matrix. .
4. The language-guided robot grasping method based on the end-to-end uncertainty process according to claim 3, characterized in that, In step S1, the method for obtaining the depth uncertainty variance is as follows: Representative point Centered on a depth image, a set of depth values for a local window region is extracted; invalid depth pixels in the set of depth values are filtered out, including pixels with depth values of zero, infinity, or exceeding a preset valid range; the local variance of the depth value set after filtering out invalid pixels is calculated as the depth uncertainty variance. When the number of effective pixels is less than the preset threshold, the statistical variance of the effective depth pixels of the entire image multiplied by the magnification factor is used as a conservative estimate.
5. The language-guided robot grasping method based on the end-to-end uncertainty process according to claim 1, characterized in that, In step S2, the joint probability score is calculated as follows: For each candidate grab target in the scene Its three-dimensional position is Calculate the weighted joint negative log-likelihood of the candidate target relative to each component of the probabilistic mixture model after propagation to three-dimensional space: ； in For the first The weight of each component, For the first The coordinates of the average of each component after propagation to three-dimensional space For the first The three-dimensional total covariance matrix after propagation of each component The probability density function is a multivariate probability density function; the candidate objective with the smallest joint negative log-likelihood value is selected as the probability selection result.
6. The language-guided robot grasping method based on the end-to-end uncertainty process according to claim 5, characterized in that, In step S2, uncertainty features are extracted and a gating mechanism is used to determine whether to adopt the probability selection result. Specifically, this includes: Uncertainty features are extracted from a set of two-dimensional semantic pointer samples obtained from multiple inferences. These uncertainty features include the two-dimensional spatial variance of the sample points, the entropy values of the weights of each component in the probabilistic mixture model, and the trace of the principal component covariance matrix. Material semantic features of the target object are extracted from the language instructions. The uncertainty features and material semantic features are input into a gating decision function. When the output of the gating decision function meets the preset switching conditions, the probabilistic selection result is used to replace the baseline selection result; otherwise, the baseline selection result is maintained.
7. The language-guided robot grasping method based on the end-to-end uncertainty process according to claim 6, characterized in that, The gating decision function is either a rule-based interpretable gating function or a learning gating function based on a machine learning classifier. When an interpretable gating function is used, the switching condition is: the uncertainty dispersion is lower than a preset dispersion threshold. Furthermore, the joint probability score is better than the baseline score by more than a preset difference threshold. When a learning gating function is used, the label selected with higher probability than the baseline in the historical crawled samples is used as the training target, and a binary classifier is trained as the gating decision function. The input features of the binary classifier include uncertainty features, material semantic features, and the difference between the probability score and the baseline score.
8. The language-guided robot grasping method based on the end-to-end uncertainty process according to claim 1, characterized in that, In step S3, the compliance control parameters for the corresponding degrees of freedom are adaptively set based on the eigenvalues of each principal axis direction. The specific process is as follows: For the three-dimensional total covariance matrix Eigenvalue decomposition yields three eigenvalues. and the corresponding feature vectors , , , ; In the principal axis coordinate system defined by the eigenvector, along the first... The impedance stiffness is set for each principal axis direction as follows: ； in Based on the basic parameter values, The modulation coefficient and , For reference eigenvalues, To crop the upper bound, The trimming function transforms the stiffness matrix in the principal axis coordinate system to the robot base coordinate system through a rotation matrix composed of eigenvectors, thus obtaining the translation parameter matrix for multi-degree-of-freedom compliant control.
9. The language-guided robot grasping method based on the end-to-end uncertainty process according to claim 8, characterized in that, Step S3 also includes approach direction adjustment and exploratory pre-contact strategies: Approach direction adjustment: using the eigenvector corresponding to the smallest eigenvalue of the covariance ellipsoid. The direction is chosen as the preferred approach direction, directing the approach direction of the end effector towards... The direction is weighted and deflected, with the deflection weights equal to the largest eigenvalue. and minimum eigenvalue The ratio is positively correlated; Exploratory pre-contact: when the trace of the three-dimensional total covariance matrix Exceeding the preset exploration trigger threshold At the same time, before performing the formal grasping, an exploratory pre-contact action is inserted. The exploratory pre-contact action includes tactile detection within a preset range along the direction of the maximum eigenvector of the covariance ellipsoid to obtain contact feedback to correct the target position.
10. A language-guided robot grasping system based on end-to-end uncertainty, applied to the method described in any one of claims 1-9, characterized in that the system include: The uncertainty quantification module is configured to perform multiple random samplings on the visual language model to obtain a two-dimensional semantic pointing point set, fit a probabilistic mixture model to obtain a two-dimensional semantic covariance matrix, and combine the deep uncertainty variance to obtain a three-dimensional total covariance matrix through Jacobian propagation. The gated probability selection module is configured to calculate the joint probability score of each candidate target based on the three-dimensional total covariance matrix, and to determine whether to replace the baseline selection result with the probability selection result through a gating mechanism. The adaptive execution module is configured to perform eigenvalue decomposition on the three-dimensional total covariance matrix and adaptively generate compliant control parameters and approach strategies for each degree of freedom. The three-dimensional total covariance matrix serves as a unified representation of uncertainty, and is sequentially passed from the uncertainty quantification module to the gated probability selection module and the adaptive execution module, thus achieving full-link integration of uncertainty information.