Humanoid robot grasping-oriented sparse-view three-dimensional gaussian language field construction method and system

Abstract

The application discloses a sparse-view three-dimensional Gaussian language field construction method and system for humanoid robot grasping. The method comprises the following steps: acquiring an RGB image set under sparse-view and a natural language instruction of a user; reconstructing a three-dimensional point cloud through stereo matching, initializing a three-dimensional Gaussian primitive field after noise elimination and metric alignment; embedding two-dimensional semantic features in the field, jointly optimizing through a double-path semantic supervision module, and constructing a semantic-consistent three-dimensional Gaussian language field; wherein the double-path semantic supervision comprises an object perception path and a global context path, which respectively constrain local semantic consistency and overall semantic relationship, and the final semantic representation of each Gaussian primitive is obtained through weighted fusion; finally, candidate grasping postures are generated, semantic reordering is performed in combination with the language instruction, and the optimal grasping posture is selected through geometric-semantic joint scoring. The method can improve the execution accuracy of the robot grasping task under complex instructions under the condition of sparse image set input.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, robot technology and three-dimensional reconstruction, and particularly relates to a sparse-view three-dimensional Gaussian language field construction method and system for humanoid robot grasping. BACKGROUND

[0002] Existing three-dimensional scene understanding and language-guided grasping methods mainly rely on dense multi-view image input to complete scene reconstruction and target recognition. However, in actual robot applications, especially in home or industrial environments, only limited sparse-view images can be obtained, which makes it difficult to:

[0003] (1) Insufficient three-dimensional reconstruction quality: traditional 3D Gaussian Splatting (3D GS) or NeRF methods require dozens to hundreds of view inputs to ensure good reconstruction results, and under sparse conditions, geometric defects and texture blur are prone to occur.

[0004] (2) Poor semantic consistency: cross-view semantic mask segmentation results are often inconsistent, resulting in unstable semantic features of the reconstructed model.

[0005] (3) Unreliable grasping positioning: existing language-guided grasping methods are mostly based on two-dimensional images or dense point clouds, making it difficult to generate robust three-dimensional semantic guidance grasping poses under sparse-view conditions.

[0006] Therefore, there is an urgent need for a new method that can complete high-quality three-dimensional semantic reconstruction under sparse-view conditions and support language-guided grasping. SUMMARY

[0007] To solve the above technical problems, the application provides a sparse-view three-dimensional Gaussian language field construction method and system for humanoid robot grasping.

[0008] The technical solution adopted by the application to solve its technical problems is:

[0009] The sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping includes the following steps:

[0010] S100: Obtain an RGB input image set from sparse views and a user natural language instruction;

[0011] S200: Based on the input image set, generate a three-dimensional point graph through a stereo matching model, and perform global fusion and noise elimination;

[0012] S300: Align the processed three-dimensional point graph with the metric of the robot coordinate system to obtain a metric-aligned three-dimensional point cloud, and initialize a three-dimensional Gaussian primitive field based on the point cloud;

[0013] S400: Projects and embeds the semantic features of a 2D image into a 3D Gaussian primitive field, and performs joint optimization through a dual-path semantic supervision module to form a semantically consistent 3D Gaussian language field. The joint optimization includes: constraining local semantic consistency through an object-aware semantic supervision path; constraining overall semantic relations through a global context semantic supervision path; weightedly fusing the semantic features output from the two paths to obtain the final semantic representation of each Gaussian primitive; and jointly training and optimizing the geometric and semantic properties of the 3D Gaussian primitive field by combining the supervision loss functions of the two paths.

[0014] S500: On a three-dimensional Gaussian language field, candidate grasping postures are generated based on geometric stability. Then, the candidate grasping postures are reordered based on the semantic relevance of natural language instructions. Finally, the final grasping posture is selected through joint geometric and semantic scoring.

[0015] Preferably, S200 includes:

[0016] S210: 3D point map generation: for input sparse viewpoint image set Any pair of views Dense 3D point maps were predicted using the stereo matching model MASt3r. Relative pose and the confidence level of each pixel The 3D dot plot contains each pixel in the reference view. Three-dimensional coordinates in coordinate system i,j are the view indices, N is the total number of input images, (u,v) are the pixel positions, and H and W are the height and width of the image, respectively.

[0017] S220: Global Coordinate Transformation: Transforms the 3D point maps of each view or view pair to a unified global coordinate system based on their camera extrinsic parameters. The transformation formula is:

[0018] ;

[0019] in, This indicates that the point will be moved from the first... The rigid transformation matrix from the local coordinate system to the global coordinate system of the view. Let k be the coordinates of the k-th point in the local coordinate system, where k is the index of the point. This represents the coordinates of the k-th point after transformation to the global coordinate system;

[0020] S230: Global Pose Optimization: Constructing a View Connection Graph ,in Let be the vertex set, representing the set of all views. An edge set is defined if and only if the view pairs... edges are added between two nodes when the degree of overlap between them exceeds a threshold or the degree of feature matching exceeds a threshold ; a global pose optimization is performed on the view connection graph using a nonlinear least squares method

[0021] S240: voxelization down-sampling: divide the space into voxel units with edge length , and keep the point with the highest confidence in each voxel

[0022]

[0023] wherein,

[0024] S250: outlier removal: for each point , count the number of neighbors within a radius , if , remove the point as an outlier, wherein the radius , and the minimum neighbor number threshold

[0025] Preferably, S300 comprises:

[0026] S310: metric alignment: align the coordinates using a Sim(3) similarity transformation, the transformation matrix is defined as:

[0027]

[0028] wherein, is a scale factor, is a rotation matrix, is a translation vector, is the transpose of a zero vector;

[0029] Optimize the transformation parameters by minimizing the alignment loss function, specifically:

[0030]

[0031] wherein, is the alignment loss function, is the logarithmic mapping of the SO(3) rotation matrix to its Lie algebra, is the trade-off between the rotation and translation terms, is the translation vector of the i-th pose in the model coordinate, is the actual translation vector of the i-th pose in the robot calibration, is the actual rotation matrix of the i-th pose in the robot calibration, ​​​​​​​​​Ri is the rotation matrix of the i-th pose in the model coordinate system;

[0032] S320: Based on the aligned point cloud Initialize the set of Gaussian primitives For each reserved point Generate a Gaussian primitive:

[0033] ;

[0034] wherein, is the i-th Gaussian primitive, is the mean of the Gaussian primitive, is the covariance matrix of the Gaussian primitive, is the color attribute of the Gaussian primitive, is the transparency attribute of the Gaussian primitive, color is obtained by projecting to the input image and using weighted average to sample the pixel color, transparency is initialized according to the local point cloud density;

[0035] wherein, the covariance matrix is estimated by the local neighborhood covariance of the points, and the estimation formula is:

[0036] ;

[0037] wherein, is the current point coordinate, is the local neighborhood point set of the current point , is the point coordinate in the neighborhood point set, is a numerical stability constant, and the value is , is the unit matrix.

[0038] Preferably, the local semantic consistency is constrained by the object perception semantic supervision path in S400, including:

[0039] S410: Semantic mask generation: for each input image Use the segmentation model SAM to generate a set of semantic masks Each mask corresponds to a candidate object region;

[0040] S420: Local feature extraction: in each mask region, extract a local semantic feature vector through the semantic encoding model CLIP ;

[0041] ​S430: 3D projection mapping: project the pixels within the mask to the 3D Gaussian primitive field through camera parameters, determine the Gaussian primitive subset covering the mask ;

[0042] S440: semantic consistency constraint: calculate the semantic feature mean value of the Gaussian primitive subset :

[0043] ;

[0044] wherein, is the semantic feature of the kth Gaussian primitive, is the kth Gaussian primitive in the subset, is the Gaussian primitive subset covering the jth mask of the ith image, is the number of Gaussian primitives in the subset;

[0045] and the semantic feature mean value is constrained to be consistent with the local semantic feature extracted from the image mask through a loss function, defined as:

[0046] ;

[0047] wherein, is the object-aware semantic loss function, is the semantic feature extracted from the jth mask of the ith image, is the semantic feature mean value of the corresponding Gaussian primitive subset.

[0048] Preferably, the overall semantic relationship is constrained in S400 through the global context semantic supervision path; the semantic features output by the dual paths are weighted and fused to obtain the final semantic representation of each Gaussian primitive; the geometric and semantic attributes of the 3D Gaussian primitive field are jointly trained and optimized through the supervision loss function of the dual paths, including:

[0049] S450: global semantic feature extraction: for each input image extract global semantic features using a visual-linguistic model ;

[0050] S460: global semantic consistency constraint: take the semantic features of the entire image as soft labels, calculate the mean value of all Gaussian primitive semantic features :

[0051] ;

[0052] wherein, denotes the total number of Gaussian primitives, denotes the set of all Gaussian primitives; ​

[0053] And the average value is constrained to maintain consistency with the global semantic features of the image by a loss function, defined as:

[0054] ;

[0055] in, The global semantic consistency loss function is... The global semantic features extracted from the i-th image are... This represents the average of the semantic features of all Gaussian primitives.

[0056] S470: Dual-path feature fusion: The features obtained from the object-aware path and the features obtained from the global context path are weighted and fused to obtain the final semantic representation of each Gaussian primitive.

[0057] ;

[0058] in, Let k be the final semantic representation of the k-th Gaussian primitive. The weighting coefficients for object-aware paths. The weight coefficients for the global context path. , Let be the semantic feature of the k-th Gaussian primitive under the object-aware path. This represents the semantic features of the k-th Gaussian primitive within the global context path.

[0059] S480: Overall Optimization: Overall Loss Function with Dual-Path Semantic Supervision for:

[0060] ;

[0061] in, These are the weighting coefficients used to control the overall supervision strength.

[0062] Preferably, in S500, candidate grasping postures are generated based on geometric stability in a three-dimensional Gaussian language field, specifically including:

[0063] S510: Construct a workspace centered on the target object within the reconstructed 3D Gaussian field. ,in, The coordinates of a point in space. Center of the object The maximum Gaussian scale of the object. Let be the collision radius of the mechanical gripper;

[0064] S520: Uniformly sample multiple seed points within the workspace, and for each seed point, generate a set of Gaussian primitives in its neighborhood. Extracting surface normal information ;

[0065] S530: Calculate the local surface covariance matrix by principal component analysis of the normals :

[0066] ;

[0067] where is the average of all normals in the neighborhood, is the j-th normal vector in the neighborhood, is the total number of Gaussians in the neighborhood;

[0068] S540: Based on the local stable coordinate system obtained by eigen-decomposition of the covariance matrix, use the GPG algorithm to generate a candidate grasp set that satisfies the mechanical stability , and calculate the geometric score of each candidate grasp :

[0069] ;

[0070] where are the left and right contact point normals, is the grasp direction, is used to measure the geometric consistency of the contact normal and the grasp direction.

[0071] Preferably, the candidate grasp is reordered in S500 based on the semantic relevance of the natural language instruction, and finally the final grasp pose is selected through the joint scoring of geometry and semantics, which specifically includes:

[0072] S550: For each candidate grasp , aggregate the semantic features of the Gaussians in its contact area to get the semantic vector of each candidate grasp :

[0073] ;

[0074] where, is the number of Gaussians in the contact area of the candidate grasp ;

[0075] S560: Get the query feature of the user's natural language instruction through the CLIP text encoder , calculate the semantic relevance score :

[0076] ;

[0077] ​S570: obtaining a final score by weightedly fusing the geometry score and the semantic score :

[0078] ;

[0079] wherein is a weight coefficient;

[0080] S580: ranking all candidate grasps according to the final score and selecting top candidate grasps as the final grasp plan:

[0081] ;

[0082] wherein, is the final grasp plan set, is the number of grasps selected.

[0083] The sparse-view three-dimensional Gaussian language field construction system for humanoid robot grasping includes:

[0084] A data acquisition module is configured to acquire an RGB input image set from a sparse view and a user natural language instruction;

[0085] A three-dimensional geometry initialization module is configured to generate a three-dimensional point graph based on the input image set through a stereo matching model, and perform global fusion and noise elimination; the processed three-dimensional point graph is aligned with the metric of the robot coordinate system to obtain a metrically aligned three-dimensional point cloud, and a three-dimensional Gaussian primitive field is initialized based on the point cloud;

[0086] A language field construction and optimization module is configured to project and embed two-dimensional image semantic features into the three-dimensional Gaussian primitive field, and perform joint optimization through a dual-path semantic supervision module to form a three-dimensional Gaussian language field with consistent semantics; wherein the joint optimization includes: constraining local semantic consistency through an object perception semantic supervision path; constraining overall semantic relationship through a global context semantic supervision path; weighting and fusing semantic features output by the dual paths to obtain a final semantic representation of each Gaussian primitive; and performing joint training and optimization of the geometry and semantic properties of the three-dimensional Gaussian primitive field by combining the supervision loss functions of the dual paths;

[0087] A language-guided grasping generation module is configured to generate candidate grasp poses based on geometric stability on the three-dimensional Gaussian language field, re-rank the candidate grasps based on semantic relevance of the natural language instruction, and finally select a final grasp pose through geometric and semantic joint scoring.

[0088] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping when executing the computer program.

[0089] A computer readable storage medium stores a computer program, and the computer program implements the steps of the sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping when executed by a processor.

[0090] The sparse-view three-dimensional Gaussian language field construction method and system for humanoid robot grasping first effectively fuse the local object features and global context information of the two-dimensional image through the dual-path semantic supervision module, significantly enhancing the integrity and consistency of the three-dimensional scene semantic representation under sparse-view; secondly, with the joint optimization mechanism of geometry and semantics, the generated robot grasping pose meets the physical stability and semantic requirements of natural language instructions at the same time, improving the understanding and execution accuracy of complex tasks; finally, the entire system only relies on sparse RGB image input, reducing the hardware deployment cost and enhancing the applicability and flexibility in real scenes. BRIEF DESCRIPTION OF DRAWINGS

[0091] Figure 1 The flowchart of the sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping in an embodiment of the present application;

[0092] Figure 2 The principle diagram of the sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping in another embodiment of the present application;

[0093] Figure 3 The principle diagram of the dual-path semantic feature enhancement module in an embodiment of the present application. DETAILED DESCRIPTION

[0094] In order to make the person skilled in the art better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0095] In one embodiment, as shown in Figure 1 and Figure 2 The present application proposes a sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping, and realizes language-guided robot grasping based on the method, comprising the following steps:

[0096] S100: obtaining an RGB input image set from a sparse-view and a user natural language instruction;

[0097] S200: based on the input image set, generating a three-dimensional point graph through a stereo matching model, and performing global fusion and noise elimination;

[0098] S300: The processed three-dimensional point map is aligned with the metric of the robot coordinate system, a metrically aligned three-dimensional point cloud is obtained, and a three-dimensional Gaussian primitive field is initialized based on the point cloud;

[0099] S400: Project and embed the two-dimensional image semantic features into the three-dimensional Gaussian primitive field, and perform joint optimization through a double-path semantic supervision module to form a semantic-consistent three-dimensional Gaussian language field; wherein the joint optimization includes: constraining local semantic consistency through an object perception semantic supervision path; constraining overall semantic relationship through a global context semantic supervision path; weighting and fusing the semantic features output by the double paths to obtain the final semantic representation of each Gaussian primitive; and performing joint training and optimization of the geometric and semantic attributes of the three-dimensional Gaussian primitive field through a supervision loss function combining the double paths;

[0100] S500: On the three-dimensional Gaussian language field, candidate grasping poses are generated based on geometric stability, the candidate grasps are then reordered based on the semantic relevance of natural language instructions, and finally the final grasping pose is selected through geometric and semantic joint scoring.

[0101] The present application effectively solves the problem of difficulty in unifying geometric and semantic information in the robot grasping task under sparse view by constructing a semantic-enhanced three-dimensional Gaussian language field. Its beneficial effects are as follows: first, the double-path semantic supervision module is used to effectively embed the local object features and global context information of the two-dimensional image into the three-dimensional scene, significantly improving the integrity and consistency of the three-dimensional semantic representation under sparse view; second, through geometric and semantic joint optimization, the generated grasping pose meets the requirements of physical stability and semantic of the language instruction, improving the understanding and execution accuracy of the robot for complex instructions; finally, the entire process only requires sparse RGB image input, reducing the deployment cost of the system and enhancing the adaptability to real scenes.

[0102] Further, S200 and S300 are sparse view three-dimensional geometry initialization, which is used to convert the sparse view RGB input image sequence into a three-dimensional point cloud consistent with the metric of the robot workspace and having good geometric integrity, and initialize a three-dimensional Gaussian primitive field based on the point cloud. It generally includes the following sub-steps: pixel-level point mapping generation of image pairs → global fusion of point maps → point cloud redundancy / noise elimination → alignment with the robot coordinate system (Sim(3)) → Gaussian primitive initialization based on the point cloud, which is expanded one by one as follows.

[0103] In one embodiment, S200 includes:

[0104] S210: Three-dimensional point map generation: for any pair of views in the input sparse view image set ​Predicting dense 3D point cloud using stereo matching model MASt3r , relative poses (or per-view camera poses relative to local reference) and confidence of each pixel for subsequent redundancy elimination and weighted fusion; wherein the 3D point cloud contains 3D coordinates of each pixel in the reference view coordinate system ; i, j are view indices, N is the total number of input images, (u, v) is pixel position, H and W are height and width of the image respectively;

[0105] S220: global coordinate transformation: transforming 3D point cloud of each view or view pair to a unified global coordinate system according to its camera extrinsic parameters, transformation formula is:

[0106] ;

[0107] wherein, Rkis the rigid transformation matrix from point in the local coordinate system of the kth view to the global coordinate system, pkis the coordinate of the kth point in the local coordinate system, k is the index of the point, pkis the coordinate of the kth point after transformation to the global coordinate system;

[0108] S230: global pose optimization: constructing view connection graph , wherein V is the vertex set, representing the set of all views, E is the edge set, edge exists if and only if the overlap between the view pair exceeds the threshold or the feature matching degree exceeds the threshold; performing global pose optimization on the view connection graph using nonlinear least squares method;

[0109] S240: voxelization down-sampling: dividing the space into voxel units with edge length , retaining the point with the highest confidence in each voxel :

[0110] ;

[0111] wherein, ;

[0112] S250: outlier removal: for each point , calculating the number of neighbors within a radius , if , remove the point ​This point is then removed as an outlier, where the radius is set. Minimum number of neighbors threshold .

[0113] Furthermore, reconstructed point clouds often lack absolute scale or orientation and need to be aligned with the robot's actual working coordinate system to ensure that the grasping posture can be directly executed by the robot.

[0114] In one embodiment, S300 includes:

[0115] S310: Metric Alignment: Coordinate alignment is performed using Sim(3) similarity transformation (including scaling, rotation, and translation). The transformation matrix is ​​defined as:

[0116] ;

[0117] in, As a scale factor, Let be a rotation matrix. It is a translation vector. It is the transpose of the zero vector;

[0118] The transformation parameters are optimized by minimizing the alignment loss function, specifically:

[0119] ;

[0120] in, To align the loss function, For the SO(3) rotation matrix, the logarithmic mapping to its Lie algebra, To balance the rotation and translation terms, Let i be the translation vector of the i-th pose in the model coordinate system. Let be the actual translation vector of the i-th pose under robot calibration. Let be the actual rotation matrix of the i-th pose under robot calibration. Let be the rotation matrix of the i-th pose in the model coordinate system;

[0121] S320: Based on aligned point cloud Initialize the Gaussian primitive set For each reserved point Generate a Gaussian primitive:

[0122] ;

[0123] in, For the i-th Gaussian primitive, The mean of the Gaussian primitives. Let be the covariance matrix of the Gaussian primitive. For the color attribute of Gaussian primitives, For the transparency attribute of the Gaussian primitive, color By projecting to the input image and taking a weighted average of the sampled pixel colors, transparency is obtained. Initialize by local point cloud density.

[0124] where the covariance matrix is estimated from the local neighborhood of points, with the estimation formula:

[0125] .

[0126] where is the coordinate of the current point, is the local neighborhood point set of the current point , is the coordinate of the point in the neighborhood point set, is a numerical stability constant, taking the value , is the identity matrix.

[0127] Further, the goal of S400 is to project and embed two-dimensional image semantic features into a three-dimensional Gaussian primitive field, forming a semantically consistent three-dimensional representation, thereby maintaining both geometric and semantic consistency under sparse viewing angle conditions. This process is achieved through a dual-path semantic supervision module: object-aware supervision (local) + global context supervision (global), as shown in Figure 3 .

[0128] The input is an initial set of three-dimensional Gaussian primitives , where each primitive contains a position , a covariance , a color , and a transparency ; the input also includes a set of sparse viewing angle images and their corresponding camera poses , as well as user natural language query terms , encoded as semantic feature vectors .

[0129] In one embodiment, the object-aware semantic supervision path in S400 constrains local semantic consistency, including:

[0130] S410: Semantic mask generation: for each input image , a set of semantic masks is generated using the segmentation model SAM, each mask corresponding to a candidate object region;

[0131] S420: Local feature extraction: within each mask region, local semantic feature vectors are extracted by the semantic encoding model CLIP ;

[0132] S430: Three-dimensional projection mapping: projecting the pixels within the mask to the three-dimensional Gaussian primitive field through camera parameters to determine the Gaussian primitive subset covering the mask ;

[0133] S440: Semantic consistency constraint: calculating the semantic feature mean value of the Gaussian primitive subset :

[0134] ;

[0135] wherein, is the semantic feature of the kth Gaussian primitive, is the kth Gaussian primitive in the subset, is the Gaussian primitive subset covering the jth mask of the ith image, is the number of Gaussian primitives in the subset;

[0136] and the semantic feature mean value is constrained to be consistent with the local semantic feature extracted from the image mask through a loss function, defined as:

[0137] ;

[0138] wherein, is the object-aware semantic loss function, is the semantic feature extracted from the jth mask of the ith image, is the semantic feature mean value of the corresponding Gaussian primitive subset.

[0139] In one embodiment, the global context semantic supervision path in S400 constrains the overall semantic relationship; the semantic features output by the dual paths are weighted and fused to obtain the final semantic representation of each Gaussian primitive; the geometric and semantic attributes of the three-dimensional Gaussian primitive field are jointly trained and optimized through the supervision loss function of the dual paths, including:

[0140] S450: Global semantic feature extraction: using a visual-linguistic model to extract global semantic features for each input image ;

[0141] S460: Global semantic consistency constraint: taking the semantic features of the entire image as soft labels to calculate the mean value of all Gaussian primitive semantic features :

[0142] ;

[0143] wherein, indicates the total number of Gaussian primitives,​​ denotes the set of all Gaussian primitives;

[0144] and the mean value is constrained to be consistent with the global semantic features of the image by a loss function, defined as:

[0145] ;

[0146] where, is the global semantic consistency loss function, is the global semantic feature extracted from the ith image, is the mean value of all Gaussian primitive semantic features; further, the semantic difference across objects can be further maintained by contrastive learning, for example, constructing positive and negative sample pairs.

[0147] S470: Dual-path feature fusion: the features obtained by the object perception path and the features obtained by the global context path are weighted and fused to obtain the final semantic representation of each Gaussian primitive:

[0148] ;

[0149] where, is the final semantic representation of the kth Gaussian primitive, is the weight coefficient of the object perception path, is the weight coefficient of the global context path, , which can be dynamically adjusted according to task requirements, and the recommended value is , is the semantic feature of the kth Gaussian primitive under the object perception path, is the semantic feature of the kth Gaussian primitive under the global context path;

[0150] S480: Overall optimization: the overall loss function of the dual-path semantic supervision is:

[0151] ;

[0152] where, is the weight coefficient for controlling the global supervision strength.

[0153] Further, after training / optimization, each Gaussian primitive has both geometric properties (position, covariance, normal vector, etc.) and semantic features . The Gaussian language field can support subsequent language-guided grasping generation. Grasping generation and selection are performed on the semantic consistent three-dimensional Gaussian language field, mainly including candidate grasping generation, semantic relevance reordering, and geometric-semantic joint scoring.

[0154] In one embodiment, the candidate grasping poses are generated based on geometric stability on the three-dimensional Gaussian language field in S500, specifically including:

[0155] S510: Building a workspace centered on the target object in the reconstructed three-dimensional Gaussian field , wherein, is the coordinate of a point in space, is the center of the object, is the maximum Gaussian scale of the object, is the collision radius of the mechanical claw;

[0156] S520: Uniformly sampling multiple seed points in the workspace , for each seed point extracting surface normal information ;

[0157] S530: Aggregating the normals by principal component analysis and calculating the local surface covariance matrix :

[0158] ;

[0159] wherein is the average of all normals in the neighborhood, is the jth normal vector in the neighborhood, is the total number of Gaussian primitives in the neighborhood;

[0160] S540: Based on the local stable coordinate system obtained by eigen-decomposing the covariance matrix, generating a candidate grasping set satisfying mechanical stability using the GPG algorithm , and calculating the geometric score of each candidate grasp :

[0161] ;

[0162] wherein are the left and right contact point normals, is the grasping direction, is used to measure the geometric consistency of the contact normal and the grasping direction.

[0163] In one embodiment, the candidate grasps are reordered based on the semantic relevance of the natural language instructions in S500, and finally the final grasping pose is selected through geometric and semantic joint scoring, specifically including:

[0164] S550: For each candidate grasp , aggregating the semantic features of the Gaussian primitives in its contact area ​, obtain a semantic vector of each candidate grasp :

[0165] ;

[0166] wherein, is the number of Gaussian primitives within the contact region of the candidate grasp ;

[0167] S560: obtain the user natural language instruction through the CLIP text encoder to obtain query features , calculate the semantic relevance score :

[0168] ;

[0169] S570: obtain the final score by weighted fusion of the geometric score and the semantic score :

[0170] ;

[0171] wherein, is a weight coefficient;

[0172] S580: sort all candidate grasps according to the final score , and select the top candidate grasps as the final grasp scheme:

[0173] ;

[0174] wherein, is the final selected grasp scheme set, is the selected number of grasps.

[0175] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0176] (1) Adapt to sparse view conditions: traditional three-dimensional semantic reconstruction methods often rely on dense multi-view input, and it is difficult to maintain robustness under limited observation. The present application realizes high-precision three-dimensional geometric initialization under sparse view conditions through point graph fusion and Sim(3) alignment method based on the stereo matching model MASt3r. Experimental results show that even under the condition of only 3 view inputs, the three-dimensional point cloud constructed can still maintain good geometric integrity with the Gaussian field.

[0177] (2) Semantic and geometric unified modeling: The invention proposes a dual-channel semantic supervision mechanism, combining object perception features and context perception features, to achieve integrated modeling of semantics and geometry in differentiable rendering optimization. This method can maintain semantic consistency in sparse view scenes, significantly outperforming single-path semantic supervision methods.

[0178] (3) Language-guided precise grasping: The invention can generate executable grasping poses that match natural language instructions under zero-shot conditions through joint screening of geometric stability (GPG algorithm score) and semantic relevance (CLIP alignment score). Experimental verification shows that this method has significantly higher grasping success rate on zero-shot objects than existing methods.

[0179] (4) Strong universality and scalability: The sparse view three-dimensional Gaussian language field construction method proposed by the invention is not only suitable for language-guided grasping, but also can be extended to open vocabulary three-dimensional understanding, robot operation task planning, and other fields, with wide application prospects.

[0180] In one embodiment, a sparse view three-dimensional Gaussian language field construction system for humanoid robot grasping is also provided, comprising:

[0181] A data acquisition module for acquiring an RGB input image set from a sparse view and a user natural language instruction;

[0182] A three-dimensional geometry initialization module for generating a three-dimensional point graph based on the input image set through a stereo matching model, and performing global fusion and noise elimination; the processed three-dimensional point graph is aligned with the metric of the robot coordinate system to obtain a metric-aligned three-dimensional point cloud, and a three-dimensional Gaussian primitive field is initialized based on the point cloud;

[0183] A language field construction and optimization module for projecting and embedding two-dimensional image semantic features into a three-dimensional Gaussian primitive field, and performing joint optimization through a dual-path semantic supervision module to form a three-dimensional Gaussian language field with consistent semantics; wherein the joint optimization includes: constraining local semantic consistency through an object perception semantic supervision path; constraining overall semantic relationship through a global context semantic supervision path; weighting and fusing the semantic features output by the dual-path to obtain the final semantic representation of each Gaussian primitive; and jointly training and optimizing the geometric and semantic properties of the three-dimensional Gaussian primitive field through the supervision loss function of the dual-path;

[0184] A language-guided grasping generation module for generating candidate grasping poses based on geometric stability on the three-dimensional Gaussian language field, then reordering the candidate grasps based on the semantic relevance of the natural language instruction, and finally selecting the final grasping pose through geometric and semantic joint scoring.

[0185] The specific limitations of the sparse-view three-dimensional Gaussian language field construction system for humanoid robot grasping can refer to the limitations of the sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping described above, and will not be repeated here. Each module in the sparse-view three-dimensional Gaussian language field construction system for humanoid robot grasping described above can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each of the above modules.

[0186] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping when executing the computer program.

[0187] In one embodiment, a computer readable storage medium is also provided, which stores a computer program, and the computer program implements the steps of the sparse-view three-dimensional Gaussian language field construction method for humanoid robot grasping when executed by a processor.

[0188] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0189] The principle and implementation of the present application are described by using specific examples. The above description of the examples is only used to help understand the core idea of the present application. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principle of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for constructing a sparse-view 3D Gaussian language field for humanoid robot grasping, characterized in that, Includes the following steps: S100: Acquire RGB input image set from sparse viewpoints and user natural language commands; S200: Based on the input image set, a 3D point map is generated through a stereo matching model, and global fusion and noise reduction are performed; S300: The processed 3D point map is aligned with the robot coordinate system to obtain a metric-aligned 3D point cloud, and a 3D Gaussian primitive field is initialized based on the point cloud. S400: Projects and embeds the semantic features of a 2D image into a 3D Gaussian primitive field, and performs joint optimization through a dual-path semantic supervision module to form a semantically consistent 3D Gaussian language field. The joint optimization includes: constraining local semantic consistency through an object-aware semantic supervision path; constraining overall semantic relations through a global context semantic supervision path; weightedly fusing the semantic features output from the two paths to obtain the final semantic representation of each Gaussian primitive; and jointly training and optimizing the geometric and semantic properties of the 3D Gaussian primitive field by combining the supervision loss functions of the two paths. S500: On a three-dimensional Gaussian language field, candidate grasping postures are generated based on geometric stability. Then, the candidate grasping postures are reordered based on the semantic relevance of natural language instructions. Finally, the final grasping posture is selected through joint geometric and semantic scoring, and the humanoid robot is controlled to complete the grasping task according to the final grasping posture. In S400, local semantic consistency is constrained through object-aware semantic supervision paths, including: S410: Semantic mask generation: For each input image Generate a set of semantic masks using the segmentation model SAM. Each mask Corresponding to a candidate object region; S420: Local Feature Extraction: Within each masked region, local semantic feature vectors are extracted using the semantic coding model CLIP. ; S430: 3D Projection Mapping: Projecting the pixels within the mask onto a 3D Gaussian primitive field using camera parameters to determine the subset of Gaussian primitives covering the mask. ; S440: Semantic Consistency Constraint: Computing Gaussian Primitive Subsets semantic feature mean : ； in, Let k be the semantic features of the k-th Gaussian primitive. For the k-th Gaussian primitive in the subset, To cover the Gaussian primitive subset of the i-th image's j-th mask, The number of Gaussian primitives in this subset; Furthermore, the mean of the semantic features is constrained to maintain consistency with the local semantic features extracted from the image mask by a loss function, defined as: ； in, For object-aware semantic loss function, The semantic features extracted from the j-th mask of the i-th image, This represents the mean of the semantic features of the corresponding subset of Gaussian primitives; In S400, the overall semantic relations are constrained through a global contextual semantic supervision path; the semantic features output from the two paths are weighted and fused to obtain the final semantic representation of each Gaussian primitive; and the geometric and semantic properties of the 3D Gaussian primitive field are jointly trained and optimized by combining the supervision loss functions of the two paths, including: S450: Global semantic feature extraction: For each input image Extracting global semantic features using a visual-language model ; S460: Global Semantic Consistency Constraint: Using the semantic features of the entire image as soft labels, calculate the average of all Gaussian primitive semantic features. : ； in, This represents the total number of Gaussian primitives. This represents the set of all Gaussian primitives; And the average value is constrained to maintain consistency with the global semantic features of the image by a loss function, defined as: ； in, The global semantic consistency loss function is... The global semantic features extracted from the i-th image are... This represents the average of the semantic features of all Gaussian primitives. S470: Dual-path feature fusion: The features obtained from the object-aware path and the features obtained from the global context path are weighted and fused to obtain the final semantic representation of each Gaussian primitive. ； in, Let k be the final semantic representation of the k-th Gaussian primitive. The weighting coefficients for object-aware paths. The weight coefficients for the global context path. , Let be the semantic feature of the k-th Gaussian primitive under the object-aware path. This represents the semantic features of the k-th Gaussian primitive within the global context path. S480: Overall Optimization: Overall Loss Function with Dual-Path Semantic Supervision for: ； in, These are weighting coefficients used to control the overall supervision strength; In the S500, candidate grasping poses are generated based on geometric stability within a 3D Gaussian language field, specifically including: S510: Construct a workspace centered on the target object within the reconstructed 3D Gaussian field. ,in, The coordinates of a point in space. Center of the object The maximum Gaussian scale of the object. Let be the collision radius of the mechanical gripper; S520: Uniformly sample multiple seed points within the workspace, and for each seed point, generate a set of Gaussian primitives in its neighborhood. Extracting surface normal information ; S530: Calculate the local surface covariance matrix by combining the normals through principal component analysis. : ； in It is the average of all normals in the neighborhood. Let j be the j-th normal vector in the neighborhood. This represents the total number of Gaussian primitives in the neighborhood. S540: Based on the locally stable coordinate system obtained from the eigenvalue decomposition covariance matrix, the GPG algorithm is used to generate a candidate grasping set that satisfies mechanical stability. And calculate each candidate grab Geometric score : ； in These are the normals to the left and right contact points, respectively. To grasp the direction, Used to measure the geometric consistency between the contact normal and the gripping direction; In S500, candidate crawling actions are reordered based on the semantic relevance of natural language instructions. Finally, the final crawling posture is selected through a joint geometric and semantic scoring, which includes: S550: Fetch each candidate It aggregates the semantic features of Gaussian primitives within its contact area. This yields the semantic vector for each candidate crawler. : ； in, For candidate crawling The number of Gaussian primitives within the contact area; S560: Enables user natural language commands Query features are obtained using the CLIP text encoder. Calculate semantic relevance score : ； S570: The final score is obtained by weighted fusion of geometric and semantic scores. : ； in These are the weighting coefficients; S580: Based on final rating Sort all candidate crawls and select the top ones. One candidate was selected as the final crawling solution: ； in, This is the final set of crawling schemes selected. The selected number of fetches.

2. The method according to claim 1, characterized in that, S200 includes: S210: 3D point map generation: for input sparse viewpoint image set Any pair of views Dense 3D point maps were predicted using the stereo matching model MASt3r. Relative pose and the confidence level of each pixel The 3D dot plot contains each pixel in the reference view. Three-dimensional coordinates in coordinate system i,j are the view indices, N is the total number of input images, (u,v) are the pixel positions, and H and W are the height and width of the image, respectively. S220: Global Coordinate Transformation: Transforms the 3D point maps of each view or view pair to a unified global coordinate system based on their camera extrinsic parameters. The transformation formula is: ； in, This indicates that the point will be moved from the first... The rigid transformation matrix from the local coordinate system to the global coordinate system of the view. Let k be the coordinates of the k-th point in the local coordinate system, where k is the index of the point. This represents the coordinates of the k-th point after transformation to the global coordinate system; S230: Global Pose Optimization: Constructing a View Connection Graph ,in Let be the vertex set, representing the set of all views. An edge set is defined if and only if the view pairs... When the overlap between edges exceeds a threshold or the feature matching degree exceeds a threshold, the edges... The nonlinear least squares method is used in the viewpoint connectivity graph. Perform global pose optimization; S240: Voxelization downsampling: dividing the space into voxel units Its side length is The point with the highest confidence level is retained within each voxel. : ； in, ; S250: Outlier Removal: For each point Calculate its radius Number of neighbors within the range ,like This point is then removed as an outlier, where the radius is set. Minimum number of neighbors threshold .

3. The method according to claim 2, characterized in that, The S300 includes: S310: Metric Alignment: Coordinate alignment is performed using the Sim(3) similarity transformation, and the transformation matrix is ​​defined as: ； in, As a scale factor, Let be a rotation matrix. It is a translation vector. It is the transpose of the zero vector; The transformation parameters are optimized by minimizing the alignment loss function, specifically: ； in, To align the loss function, For the SO(3) rotation matrix, the logarithmic mapping to its Lie algebra, To balance the rotation and translation terms, Let i be the translation vector of the i-th pose in the model coordinate system. Let be the actual translation vector of the i-th pose under robot calibration. Let be the actual rotation matrix of the i-th pose under robot calibration. Let be the rotation matrix of the i-th pose in the model coordinate system; S320: Based on aligned point cloud Initialize the Gaussian primitive set For each reserved point Generate a Gaussian primitive: ； in, For the i-th Gaussian primitive, The mean of the Gaussian primitives. Let be the covariance matrix of the Gaussian primitive. For the color attribute of Gaussian primitives, The transparency property of the Gaussian primitive, color By The transparency is obtained by projecting the image onto the input image and using a weighted average sampling of pixel colors. Initialize based on local point cloud density; Wherein, the covariance matrix The estimation is obtained from the local neighborhood covariance of the point, and the estimation formula is: ； in, The coordinates of the current point. For the current point The local neighborhood point set, The coordinates of the points in the neighborhood point set. Let be the numerical stability constant, and let its value be . , It is an identity matrix.

4. A sparse-view 3D Gaussian language field construction system for humanoid robot grasping, performing the method as described in any one of claims 1 to 3, characterized in that, include: The data acquisition module is used to acquire RGB input image sets from sparse perspectives and user natural language commands; The 3D geometry initialization module is used to generate a 3D point map based on the input image set through a stereo matching model, and to perform global fusion and noise reduction. The processed 3D point map is aligned with the robot coordinate system to obtain a metrically aligned 3D point cloud, and a 3D Gaussian primitive field is initialized based on this point cloud. The language field construction and optimization module projects and embeds the semantic features of two-dimensional images into a three-dimensional Gaussian primitive field. Joint optimization is performed through a dual-path semantic supervision module to form a semantically consistent three-dimensional Gaussian language field. This joint optimization includes: constraining local semantic consistency through an object-aware semantic supervision path; constraining overall semantic relations through a global contextual semantic supervision path; weighted fusion of the semantic features output from the two paths to obtain the final semantic representation of each Gaussian primitive; and joint training and optimization of the geometric and semantic properties of the three-dimensional Gaussian primitive field by combining the supervised loss functions of the two paths. The language-guided grasping generation module is used to generate candidate grasping poses based on geometric stability in a three-dimensional Gaussian language field. Then, the candidate grasping poses are reordered based on the semantic relevance of natural language instructions. Finally, the final grasping pose is selected through joint geometric and semantic scoring.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images