A mechanical arm grasping method based on reference segmentation and space perception

Abstract

The application discloses a mechanical arm grabbing method based on reference segmentation and space perception, comprising the following steps: S1, acquiring point cloud data and natural language instructions in a mechanical arm grabbing scene, and pre-processing to extract corresponding modal features; S2, inputting the modal features of the point cloud data and the modal features of the natural language instructions into a reference segmentation and space perception model to obtain a grabbing posture result set; S3, selecting an optimal grabbing posture in the grabbing posture result set as a control instruction, and driving the mechanical arm to grab a target object through the control instruction. The method can effectively improve the understanding ability of the mechanical arm to the semantic target and the grabbing success rate in a complex environment while keeping low computing resource consumption, and realizes accurate, efficient and robust grabbing posture prediction performance.

Description

Technical Field

[0001] This invention belongs to the field of robot vision and intelligent grasping technology, specifically relating to a robotic arm grasping method based on referential segmentation and spatial perception. Background Technology

[0002] With the widespread application of robots in warehousing and logistics, services, and industrial automation, the ability of robotic arms to perform flexible and precise grasping tasks in unstructured environments has become a research hotspot. To achieve efficient recognition and robust grasping of target objects in complex scenes, researchers have gradually combined visual understanding with deep learning technology, developing various grasping posture detection methods.

[0003] Traditional grasping pose detection methods often rely on uniformly sampling potential grasping points from the entire point cloud and then evaluating their feasibility one by one. This strategy is computationally intensive and often suffers from performance limitations due to the lack of prior guidance on the distribution of grasping regions in the environment. To improve grasping efficiency and accuracy, some studies have proposed the concept of "graspness," which uses geometric priors to quickly identify high-potential grasping regions and introduces neural networks for modeling, significantly improving the performance of 6-DOF grasping pose prediction.

[0004] On the other hand, with the development of multimodal learning, the demand for using natural language expressions to assist robots in completing complex tasks is constantly growing. Against this backdrop, 3D Referring Expression Segmentation (3D-RES) has gradually emerged as an new approach, guiding models to locate specific targets by parsing the spatial relationships described in referring expressions (natural language instructions). However, existing 3D-RES methods still perform poorly under complex semantic and spatial ambiguity conditions, exhibiting low recognition accuracy and limiting their direct application in grasping tasks. Summary of the Invention

[0005] The purpose of this invention is to address the above-mentioned problems by proposing a robotic arm grasping method based on referential segmentation and spatial awareness. While maintaining low computational resource consumption, this method can effectively improve the robotic arm's ability to understand semantic targets and the grasping success rate in complex environments, achieving accurate, efficient and robust grasping posture prediction performance.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The present invention proposes a robotic arm grasping method based on referential segmentation and spatial perception, comprising the following steps:

[0008] S1. Acquire point cloud data and natural language commands in the robotic arm grasping scene, and preprocess them to extract the corresponding modal features;

[0009] S2. Input the modal features of the point cloud data and the modal features of the natural language commands into the index segmentation and spatial awareness model to obtain the grasping posture result set. The denotation segmentation and spatial awareness model performs the following operations:

[0010] S21. Generating spatial location estimation of natural language instructions based on modal features of point cloud data and modal features of natural language instructions. ;

[0011] S22. Estimation of spatial location based on modal features of point cloud data and natural language commands. Using spatially perceptual aggregation mechanisms, semantic masks are generated for the subject of natural language instructions. ;

[0012] S23. Based on semantic masking The candidate point set is filtered using the candidate point scoring function. ;

[0013] S24. Separately capture the candidate point set. A grasping posture parameter space is constructed for each candidate point, and the optimal grasping posture for each candidate point is selected from the grasping posture parameter space. The optimal grasping postures of all candidate points form a grasping posture result set. ;

[0014] S3, Select any grasping posture result set The optimal grasping posture is selected as the control command, and the robotic arm is driven to move the gripper to grasp the target object.

[0015] Preferably, the preprocessing of point cloud data is as follows:

[0016] 1) Voxel downsampling: Voxelize the point cloud data according to a preset voxel threshold to obtain downsampled point cloud data. N represents the number of points in the downsampled point cloud data;

[0017] 2) Normal Vector and Geometric Feature Extraction: Based on a preset number of neighborhood points, the normal vectors of each point in the downsampled point cloud data are estimated using the least squares method by fitting a plane, and geometric features are formed. Geometric features include the spatial coordinates and normal vector of the corresponding point;

[0018] 3) Normalization: Normalize the downsampled point cloud data to a unit cube;

[0019] 4) Statistical filtering: Calculate the average distance of the neighborhood points for each point after normalization, and remove the corresponding points that exceed the average distance by a preset multiple of the standard deviation as outliers.

[0020] 5) Superpoint partitioning: Clustering algorithms are used to partition the statistically filtered point cloud data into superpoint sets. ,in, The feature dimension of the superpoint is represented. Represent the number of superpoints and extract the set of superpoints. Coordinates of the Chinese Super League Set of superpoints and super set Coordinates of the Chinese Super League As modal features of point cloud data;

[0021] The preprocessing of natural language instructions is as follows:

[0022] 1) Word segmentation and dependency relations: Use natural language processing tools to construct a dependency syntax tree and extract words, including subjects, adjectives and prepositional phrases;

[0023] 2) Word embedding encoding: Converting all words into embedding vectors using the BERT model. As a modal feature of natural language instructions, among which, Indicates the number of words, This represents the vector dimension of a word.

[0024] Preferably, the clustering algorithm is the K-means algorithm.

[0025] Preferably, the spatial location estimation of the natural language instruction is generated based on the modal features of point cloud data and the modal features of the natural language instruction. The details are as follows:

[0026] S211, embedding vector and super set Initial cross-modal fusion is performed separately to obtain corresponding language embedding features. and point cloud embedding features The formula is as follows:

[0027] ;

[0028] in, Represents the embedding vector The learnable mapping matrix, Represents the set of superpoints The learnable mapping matrix, where DDI(•) represents the dependency-driven interaction mechanism. The feature dimension of the superpoint is represented. Indicates the projection dimension;

[0029] S212. Calculate the text embedding and superpoint set for each word. The similarity of the Chinese Super League points, then the first Text embedding of 1 word With superpoint set The j-th superpoint similarity The formula is as follows:

[0030] ;

[0031] in,

[0032] ;

[0033] in, Text embedding representing all words, S represents the superpoint feature. , Representing language embedding features Learnable matrix, Represents the set of superpoints Learnable matrix, That is to say, the first The probability that a word is located in the j-th superpoint. , sim(•) represents the similarity function, defined as follows: , For transpose;

[0034] S213. Arrange the initial positions of all words in the point cloud space into vectors to serve as spatial location estimates for natural language instructions. Then the first The initial position of each word in the point cloud space The formula is as follows:

[0035] ;

[0036] in, Let be the spatial coordinates of the j-th superpoint.

[0037] Preferably, the spatial location estimation is based on the modal features of the point cloud data and the natural language commands. Using spatially perceptual aggregation mechanisms, semantic masks are generated for the subject of natural language instructions. The details are as follows:

[0038] S221. Estimate the spatial location of each natural language instruction. and super set Coordinates of the Chinese Super League Correspondingly construct spatial location encoding vector and superpoint location encoding vector The formula is as follows:

[0039] ;

[0040] Where APE(•) represents the absolute position encoding function;

[0041] S222, Encode vector based on spatial location and superpoint location encoding vector Construct query vector and key vector The formula is as follows:

[0042] ;

[0043] in, A learnable query weight matrix. The key weight matrix is ​​a learnable matrix. For splicing operations;

[0044] S223, Query vector and key vector Input a multi-head attention module to obtain a spatially aware embedding representation of the subject of a natural language instruction. :

[0045] ;

[0046] in, The weight matrix is ​​a learnable value. This represents the softmax function;

[0047] S224. Computing the spatially-aware embedding representation of the subject of a natural language instruction With superpoint set By analyzing the response values ​​of all superpoints and generating the target semantic response map, the spatially perceived embedding representation of the subject of the natural language instruction is obtained. With superpoint set The j-th superpoint response value The formula is as follows:

[0048] ;

[0049] in, For the Sigmoid function, That is, it represents the subject and the j-th superpoint. Similarity;

[0050] S225. Binarize the target semantic response map and generate a semantic mask. The formula is as follows:

[0051] ;

[0052] Where θ is a preset binarization threshold. Indicates the corresponding response value. This indicates whether a point in the cluster to which each superpoint belongs belongs to the target object. 1 represents yes, and 0 represents no.

[0053] Preferably, based on semantic mask The constructed crawl candidate point scoring function filters out the crawl candidate point set. The details are as follows:

[0054] S231. Extract all semantic masks from the downsampled point cloud data. The cluster of superpoints with a value of 1 forms the target candidate point set. , This indicates the number of candidate points in the target candidate point set;

[0055] S232, For the u-th candidate point , The local geometric features of the corresponding candidate points are extracted by constructing a neighborhood with a preset radius threshold r. The local geometric features include normal vector, curvature, local point density, and normal consistency.

[0056] S233. Construct a scoring function for candidate points, mapping the local geometric features of each candidate point to a score value. The formula is as follows:

[0057] ;

[0058] in, Represents the u-th candidate point The k-th dimension normalized local geometric feature, The weight coefficients represent the normalized local geometric features in the k-th dimension. , Dimensions representing local geometric features;

[0059] S234. Based on the preset capture scoring threshold Filter the candidate points, that is, retain all score values. Greater than the capture score threshold The candidate points form an initial screening set, and it is determined whether the number of candidate points in the initial screening set exceeds the preset number of candidate points. If so, farthest point sampling is performed to select a first preset number of candidate points from the initial screening set to form a capture candidate point set. Otherwise, the initial screening set is directly used as the candidate set for crawling. ,in, , M is the set of candidate points to be captured. The number of candidate points in the data, and the capture of the candidate point set. Each candidate point contains spatial coordinates and a score.

[0060] Preferably, the candidate point set is captured separately. A grasping posture parameter space is constructed for each candidate point, and the optimal grasping posture for each candidate point is selected from the grasping posture parameter space. The optimal grasping postures of all candidate points form a grasping posture result set. The details are as follows:

[0061] S241, For the nth candidate point The grasping posture parameter space is constructed as follows:

[0062] With the nth candidate point A local coordinate system is constructed to determine the center of the grasp. An approach direction is sampled on a hemispherical surface of radius R that is away from the target object in the local coordinate system. There are L grasping postures under each approach direction, which forms the nth candidate point. Grasping posture parameter space L is the second preset quantity. For the nth candidate point The qth grasping posture;

[0063] S242. Filter the grasping postures in the grasping posture parameter space of each candidate point and retain the qualified grasping postures as candidate grasping postures.

[0064] S243. Calculate and sort the scores of all grabbing postures in the candidate grabbing postures, and retain the grabbing posture with the highest score among the candidate grabbing postures of each candidate point as the optimal grabbing posture of the corresponding candidate point.

[0065] S244. Obtain the grasping posture result set , Indicates the nth candidate point The optimal grasping posture, grasping posture result set Each optimal grasping pose has six degrees of freedom parameters. , means as follows:

[0066] ;

[0067] in, To capture the center position, To capture the X-axis coordinate of the center, To capture the Y-axis coordinate of the center, To capture the Z-axis coordinate of the center; The approach direction, i.e., the initial approach unit vector of the robotic arm's gripper, determines the orientation of the robotic arm's gripper relative to the target object. The attitude rotation angle is the rotation angle of the gripper of the robotic arm relative to the approach direction; d represents the opening width, which is the angle of the gripper of the robotic arm; d represents the approach depth, which is the depth to which the gripper of the robotic arm advances along the approach direction.

[0068] Preferably, the grasping postures in the grasping posture parameter space of each candidate point are filtered separately, including:

[0069] 1) Geometric evaluation: Select grasping postures that simultaneously satisfy the conditions that the included normal angle is less than a preset degree and the curvature is less than a preset value;

[0070] 2) Collision detection: Remove grabbing postures with collision risks from the grabbing postures selected by geometric evaluation.

[0071] Preferably, the scores of all grasping postures among the candidate grasping postures are calculated based on a stability scoring function. The stability scoring function comprises a first MLP model, a second MLP model, a normalization function, and an activation function connected in sequence. Then, the score of the nth candidate point... The qth grasping posture rating The formula is as follows:

[0072] ;

[0073] in, This represents the stability scoring function.

[0074] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0075] To address the problems of unclear target understanding, inaccurate grasping area localization, and low grasping posture generation efficiency in current robotic arm grasping tasks, this invention proposes a robotic arm grasping method based on referential segmentation and spatial perception. This method aims to accurately locate target objects and generate high-quality grasping postures from both semantic understanding and spatial structure perspectives. Specifically, guided by natural language instructions, and combined with 3D point cloud data perception, a semantic and spatial linkage mechanism is constructed between the point cloud and the language expression. The descriptive target is extracted from the natural language instructions, and the precise semantic mask and grasping area of ​​the target object are determined by spatial positional relationship reasoning through a spatial perception aggregation mechanism. High-potential grasping points are then screened to achieve high-precision segmentation of the target object region. Subsequently, the grasping quality is estimated through local geometric features within the grasping area, and finally, the six-degree-of-freedom parameters of the optimal grasping posture are output. While maintaining low computational resource consumption, this method effectively improves the robotic arm's ability to understand semantic targets and the grasping success rate in complex environments, achieving accurate, efficient, and robust grasping posture prediction performance. It is particularly suitable for semantically driven human-robot collaboration, service robots, and multi-object operation scenarios. Attached Figure Description

[0076] Figure 1 This is a flowchart of the robotic arm grasping method based on reference segmentation and spatial perception according to the present invention;

[0077] Figure 2 This is a flowchart of steps S1, S21, and S22 of the present invention;

[0078] Figure 3 The flowcharts for steps S23 and S24 of the present invention are shown below. Detailed Implementation

[0079] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0080] It should be noted that when a component is referred to as being "connected" to another component, it can be directly connected to the other component or there may be an intervening component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application.

[0081] like Figures 1-3 As shown, a robotic arm grasping method based on referential segmentation and spatial perception includes the following steps:

[0082] S1. Acquire point cloud data and natural language commands from the robotic arm grasping scene, and preprocess them to extract the corresponding modal features.

[0083] In one embodiment, the preprocessing of point cloud data is as follows:

[0084] 1) Voxel downsampling: Voxelize the point cloud data according to a preset voxel threshold to obtain downsampled point cloud data. N represents the number of points in the downsampled point cloud data;

[0085] 2) Normal Vector and Geometric Feature Extraction: Based on a preset number of neighborhood points, the normal vectors of each point in the downsampled point cloud data are estimated using the least squares method by fitting a plane, and geometric features are formed. Geometric features include the spatial coordinates and normal vector of the corresponding point;

[0086] 3) Normalization: Normalize the downsampled point cloud data to a unit cube;

[0087] 4) Statistical filtering: Calculate the average distance of the neighborhood points for each point after normalization, and remove the corresponding points that exceed the average distance by a preset multiple of the standard deviation as outliers.

[0088] 5) Superpoint partitioning: Clustering algorithms are used to partition the statistically filtered point cloud data into superpoint sets. ,in, The feature dimension of the superpoint is represented. Represent the number of superpoints and extract the set of superpoints. Coordinates of the Chinese Super League Set of superpoints and super point set Coordinates of the Chinese Super League As modal features of point cloud data;

[0089] The preprocessing of natural language instructions is as follows:

[0090] 1) Word segmentation and dependency relations: Use natural language processing tools to construct a dependency syntax tree and extract words, including subjects, adjectives and prepositional phrases;

[0091] 2) Word embedding encoding: Converting all words into embedding vectors using the BERT model. As a modal feature of natural language instructions, among which, Indicates the number of words. This represents the vector dimension of a word.

[0092] In one embodiment, the clustering algorithm is the K-means algorithm.

[0093] Specifically, such as Figure 2 As shown, point cloud data can be acquired through an RGB-D camera (color image depth camera) or a LiDAR sensor. It is generally three-dimensional point cloud data with a sampling resolution of 640×480. It is used to record the spatial coordinates (x, y, z) of each point in the robotic arm grasping scene. x represents the X-axis coordinate of the robotic arm coordinate system, y represents the Y-axis coordinate of the robotic arm coordinate system, and z represents the Z-axis coordinate of the robotic arm coordinate system. The robotic arm coordinate system OXYZ can be arbitrarily set according to actual needs, and O represents the origin of the robotic arm coordinate system.

[0094] Point cloud data preprocessing (Point Encoder) is as follows:

[0095] 1) Voxel downsampling: With a preset voxel threshold size of 5mm, the point cloud data is voxelized to obtain downsampled point cloud data. N represents the number of points in the downsampled point cloud data, and each point has three-dimensional spatial coordinates.

[0096] 2) Normal Vector and Geometric Feature Extraction: A preset neighborhood number of points K=16 is selected. A plane is fitted using the least squares method to estimate the normal vector of each point in the downsampled point cloud data. This normal vector is then appended to the original 3D spatial coordinates to form geometric features. If both the normal vector and the spatial coordinates are three-dimensional, then the dimension of the geometric feature is 6.

[0097] 3) Normalization processing: The point cloud data after uniform scaling and downsampling is made so that its center is located at the origin (0,0,0) of the robot arm coordinate system and its range falls within [−1,1], that is, it is normalized to the unit cube.

[0098] 4) Statistical filtering: Calculate the average distance of the neighborhood points for each point after normalization, and remove outliers that exceed twice the standard deviation of the average distance. That is, use a standard deviation factor of 2.0 and set the neighborhood radius to 0.05m.

[0099] 5) Superpoint partitioning: Clustering algorithms are used to partition the statistically filtered point cloud data into a set of structurally stable superpoints. and with the superpoint set Coordinates of the Chinese Super League Together, they serve as modal features of point cloud data. Superpoints are the center points of each cluster.

[0100] The user provides a natural language instruction (Text), a freely expressible language input, i.e., a referential expression, such as "the blue ball next to the light blue cup." The preprocessing of the natural language instruction (Text Encoder) is as follows:

[0101] 1) Word segmentation and dependency parsing: Use the natural language processing tool StanfordNLP to construct a dependency syntax tree and extract subjects, adjectives, prepositional phrases, etc.

[0102] 2) Word embedding encoding: Convert all words into embedding vectors of length 768 using the BERT model. , as modal features of natural language instructions.

[0103] The system acquires 3D point cloud data of the target scene (robotic arm grasping scene) and natural language commands input by the user. Preprocessing is then performed on both the point cloud data and the natural language commands. Specifically, the point cloud data undergoes voxel downsampling, normal vector and geometric feature extraction, normalization, statistical filtering, and superpoint partitioning. The natural language commands undergo word segmentation, dependency analysis, and word embedding encoding. Modal features of both the point cloud data and the natural language commands are extracted to provide a foundation for subsequent target object localization and grasping posture estimation.

[0104] S2. Input the modal features of the point cloud data and the modal features of the natural language commands into the index segmentation and spatial awareness model to obtain the grasping posture result set. The denotation segmentation and spatial awareness model performs the following operations:

[0105] S21. Generating spatial location estimation of natural language instructions based on modal features of point cloud data and modal features of natural language instructions. .

[0106] In one embodiment, spatial location estimation of natural language instructions is generated based on modal features of point cloud data and modal features of natural language instructions. The details are as follows:

[0107] S211, embedding vector and super set Initial cross-modal fusion is performed separately to obtain corresponding language embedding features. and point cloud embedding features The formula is as follows:

[0108] ;

[0109] in, Represents the embedding vector The learnable mapping matrix, Represents the set of superpoints The learnable mapping matrix, where DDI(•) represents the dependency-driven interaction mechanism. The feature dimension of the superpoint is represented. Indicates the projection dimension;

[0110] S212. Calculate the text embedding and superpoint set for each word. The similarity of the Chinese Super League points, then the first Text embedding of 1 word With superpoint set The j-th superpoint similarity The formula is as follows:

[0111] ;

[0112] in,

[0113] ;

[0114] in, Text embedding representing all words, S represents the superpoint feature. , Representing language embedding features Learnable matrix, Represents the set of superpoints Learnable matrix, That is to say, the first The probability that a word is located in the j-th superpoint. , sim(•) represents the similarity function, defined as follows: , For transpose;

[0115] S213. Arrange the initial positions of all words in the point cloud space into vectors to serve as spatial location estimates for natural language instructions. Then the first The initial position of each word in the point cloud space The formula is as follows:

[0116] ;

[0117] in, Let be the spatial coordinates of the j-th superpoint.

[0118] Specifically, such as Figure 2 As shown, the embedding vector and super set Each object is projected into a unified-dimensional embedding space for initial cross-modal fusion (multimodal encoder). DDI(•) represents Dependency-Driven Interaction, used to enhance language context. Specifically, it employs a Dependency-Driven Interaction module (DDI) to guide the structural relationships between different words in a natural language sentence based on the dependency grammar structure (Dependency Tree), thereby enhancing the language embedding representation and achieving semantic alignment with point cloud features. For example, in a natural language instruction like "grab the red cup on the left," dependency relationships identify "cup" as the subject, "red" as an adjective, and "left" as a prepositional phrase. Therefore, objects with a red color and a left-leaning position are prioritized in the embedding space, with "cup" as the final semantic response target. Dependency-Driven Interaction is a well-known existing technology and will not be elaborated further. The text embedding and superpoint set for each word are then calculated. The similarity of points in the Chinese Super League (CSL) is used to establish preliminary semantic-spatial alignment relationships. Then, the similarity is used to calculate the initial position estimates of each word in the point cloud space, and these estimates are integrated to obtain the spatial position estimates of the natural language instructions. .

[0119] S22. Estimation of spatial location based on modal features of point cloud data and natural language commands. Using spatially perceptual aggregation mechanisms, semantic masks are generated for the subject of natural language instructions. .

[0120] In one embodiment, the spatial location of point cloud data and natural language instructions is estimated. Using spatially perceptual aggregation mechanisms, semantic masks are generated for the subject of natural language instructions. The details are as follows:

[0121] S221. Estimate the spatial location of each natural language instruction. and super set Coordinates of the Chinese Super League Correspondingly construct spatial location encoding vector and superpoint location encoding vector The formula is as follows:

[0122] ;

[0123] Where APE(•) represents the absolute position encoding function;

[0124] S222, Encode vectors based on spatial location and superpoint location encoding vector Construct query vector and key vector The formula is as follows:

[0125] ;

[0126] in, A learnable query weight matrix. The key weight matrix is ​​a learnable matrix. For splicing operations;

[0127] S223, Query vector and key vector Input a multi-head attention module to obtain a spatially aware embedding representation of the subject of a natural language instruction. :

[0128] ;

[0129] in, The weight matrix is ​​a learnable value matrix. This represents the softmax function;

[0130] S224. Computing the spatially-aware embedding representation of the subject of a natural language instruction With superpoint set By analyzing the response values ​​of all superpoints and generating the target semantic response map, the spatially perceived embedding representation of the subject of the natural language instruction is obtained. With superpoint set The j-th superpoint response value The formula is as follows:

[0131] ;

[0132] in, For the Sigmoid function, That is, it represents the subject and the j-th superpoint. Similarity;

[0133] S225. Binarize the target semantic response map and generate a semantic mask. The formula is as follows:

[0134] ;

[0135] Where θ is a preset binarization threshold. Indicates the corresponding response value. This indicates whether a point in the cluster to which each superpoint belongs belongs to the target object. 1 represents yes, and 0 represents no.

[0136] Specifically, such as Figure 2 As shown, by introducing a spatially-aware aggregation mechanism, the semantic alignment between the subject and the target spatial region is further improved. This integrates language embedding with the hyperpoint spatial structure to achieve a spatially-aware embedding representation of the subject of a natural language instruction. Structural enhancement is performed to give the subject a stronger sense of spatial context and generate an accurate semantic mask, that is, to obtain the semantic mask of the target object based on the referential expression (natural language instruction) to complete referential segmentation. Figure 2 The spatial-aware attention module corresponds to steps S221-S223, and the mask encoder module corresponds to steps S224-S225. APE(•) can be an absolute position encoding function mapped by an MLP model (Multilayer Perceptron). In this embodiment, a preset binarization threshold θ=0.5 is set to binarize the target semantic response map to generate a semantic mask. By constructing a cross-modal fusion between natural language instructions and point cloud data, spatial response modeling of target objects described by natural language instructions is achieved, a semantic mask (Instance Mask) is initially generated, and coarse-grained target localization is completed.

[0137] S23. Based on semantic masking The candidate point set is filtered using the candidate point scoring function. .

[0138] In one embodiment, based on a semantic mask The constructed crawl candidate point scoring function filters out the crawl candidate point set. The details are as follows:

[0139] S231. Extract all semantic masks from the downsampled point cloud data. The cluster of superpoints with a value of 1 forms the target candidate point set. , This indicates the number of candidate points in the target candidate point set;

[0140] S232, For the u-th candidate point , The local geometric features of the corresponding candidate points are extracted by constructing a neighborhood with a preset radius threshold r. The local geometric features include normal vector, curvature, local point density, and normal consistency.

[0141] S233. Construct a scoring function for candidate points, mapping the local geometric features of each candidate point to a score value. The formula is as follows:

[0142] ;

[0143] in, Represents the u-th candidate point The k-th dimension normalized local geometric feature, The weight coefficients represent the normalized local geometric features in the k-th dimension. , Dimensions representing local geometric features;

[0144] S234. Based on the preset capture scoring threshold Filter the candidate points, that is, retain all score values. Greater than the capture score threshold The candidate points form an initial screening set, and it is determined whether the number of candidate points in the initial screening set exceeds the preset number of candidate points. If so, the farthest point sampling is performed to select a first preset number of candidate points from the initial screening set to form a capture candidate point set. Otherwise, the initial screening set is directly used as the candidate set for crawling. ,in, , M is the set of candidate points to be captured. The number of candidate points in the data, and the capture of the candidate point set. Each candidate point contains spatial coordinates and a score.

[0145] Specifically, such as Figure 3 As shown, firstly, based on the semantic mask... Extract all semantic masks from the downsampled point cloud data. The points in the cluster containing the superpoints with a value of 1 (extracting the point cloud of the target object) form a set of candidate target points. (i.e., Instance Point) serves as a candidate region for crawling point analysis. Then, a neighborhood is constructed using a preset radius threshold r to extract the local geometric features of the corresponding candidate points. Based on the local geometric features of each candidate point, a rule-driven crawling candidate point scoring function is constructed, forming a crawling candidate point set. Local geometric features include normal vectors, curvature, local point density, and normal uniformity, which can be adjusted according to actual needs. These features can be directly generated by Open3D software based on preset parameters. In this embodiment, a neighborhood is constructed using a preset radius threshold r = 0.05 m, a technique well-known to those skilled in the art, and will not be elaborated further here. The candidate point scoring function can be expressed as a weighted sum, used to quickly evaluate the grasping potential of candidate points, without relying on a neural network model. Based on the preset grasping scoring threshold... =0.6 is used to filter candidate points, that is, to retain all score values. Greater than the capture score threshold The candidate points form an initial screening set. If the number of candidate points in the initial screening set exceeds the preset number of candidate points (e.g., 512), Farthest Point Sampling (FPS) is performed to select a first preset number (e.g., 128) of candidate points from the initial screening set to form a spatially uniform capture candidate point set. Otherwise, the initial screening set is directly used as the candidate set for crawling. ,in, Capture candidate point set Each candidate point contains spatial coordinates and a score, which makes it easier to select the optimal grasping posture with a reasonable structure and the highest score based on the score, and use it as the basis for posture generation.

[0146] S24. Separately capture the candidate point set. A grasping posture parameter space is constructed for each candidate point, and the optimal grasping posture for each candidate point is selected from the grasping posture parameter space. The optimal grasping postures of all candidate points form a grasping posture result set. .

[0147] In one embodiment, the candidate point set is captured respectively. A grasping posture parameter space is constructed for each candidate point, and the optimal grasping posture for each candidate point is selected from the grasping posture parameter space. The optimal grasping postures of all candidate points form a grasping posture result set. The details are as follows:

[0148] S241, For the nth candidate point The grasping posture parameter space is constructed as follows:

[0149] With the nth candidate point A local coordinate system is constructed to determine the center of the grasp. An approach direction is sampled on a hemispherical surface of radius R that is away from the target object in the local coordinate system. There are L grasping postures under each approach direction, which forms the nth candidate point. Grasping posture parameter space L is the second preset quantity. For the nth candidate point The qth grasping posture;

[0150] S242. Filter the grasping postures in the grasping posture parameter space of each candidate point and retain the qualified grasping postures as candidate grasping postures.

[0151] S243. Calculate and sort the scores of all grabbing postures in the candidate grabbing postures, and retain the grabbing posture with the highest score among the candidate grabbing postures of each candidate point as the optimal grabbing posture of the corresponding candidate point.

[0152] S244. Obtain the grasping posture result set , Indicates the nth candidate point The optimal grasping posture, grasping posture result set Each optimal grasping posture has six degrees of freedom parameters. , means as follows:

[0153] ;

[0154] in, To capture the center position, To capture the X-axis coordinate of the center, To capture the Y-axis coordinate of the center, To capture the Z-axis coordinate of the center; The approach direction, i.e., the initial approach unit vector of the robotic arm's gripper, determines the orientation of the robotic arm's gripper relative to the target object. The attitude rotation angle is the rotation angle of the gripper of the robotic arm relative to the approach direction; d represents the opening width, which is the angle of the gripper of the robotic arm; d represents the approach depth, which is the depth to which the gripper of the robotic arm advances along the approach direction.

[0155] In one embodiment, the grasping postures in the grasping posture parameter space of each candidate point are filtered, including:

[0156] 1) Geometric evaluation: Select grasping postures that simultaneously satisfy the conditions of a normal angle less than a preset degree and a curvature less than a preset value;

[0157] 2) Collision detection: Remove grabbing postures with collision risks from the grabbing postures selected by geometric evaluation.

[0158] In one embodiment, the score of all grasping poses among the candidate grasping poses is calculated based on a stability scoring function, which includes a first MLP model, a second MLP model, a normalization function, and an activation function connected in sequence. Then, the score of the nth candidate point... The qth grasping posture rating The formula is as follows:

[0159] ;

[0160] in, This represents the stability scoring function.

[0161] Specifically, such as Figure 3 As shown, for the crawling candidate point set Multiple possible grasping poses are constructed for each candidate point, and the optimal grasping pose for each candidate point is selected. The output grasping pose is the six-degree-of-freedom parameters of the optimal grasping pose. This information is used by the robotic arm for subsequent grasping. In this embodiment, for example, there are L=48 grasping postures sampled in the approach direction. Each candidate point can generate several combinations of grasping postures, forming the grasping posture parameter space for each candidate point. , ,in It's easy to understand that, depending on actual needs, several approach directions can be used for each candidate point to obtain more grasping postures. The approach direction refers to the unit vector along which the robotic arm's end effector (such as a gripper) "approaches" the target object during the grasping operation. Each grasping posture is 7-dimensional, corresponding to six degrees of freedom.

[0162] The grasping postures in the grasping posture parameter space of each candidate point are screened, including geometric evaluation and collision detection. The geometric evaluation uses Open3D software to check the surface structure of the gripper contact area, prioritizing areas with consistent normals and flat gripping surfaces, such as normal angles less than 30 degrees and surface flatness. curvature The grasping posture is determined. Collision detection can be performed using FCL software to simulate whether the gripper path collides with surrounding objects or the scene, eliminating all postures with collision risk. These are all technologies well-known to those skilled in the art and will not be elaborated upon here. Qualified grasping postures will be retained for subsequent scoring and ranking. For each qualified grasping posture, a stability scoring function is used to score it; a higher score indicates a higher probability of successful grasping. Finally, the grasping posture with the highest score for each candidate point is retained, and the grasping posture result set is output. (i.e., Pose). The six degrees of freedom parameters of each highest-scoring grasping pose are compiled, including the position of the grasping center. That is, the spatial coordinates in the robot arm coordinate system.

[0163] S3, Select any grasping posture result set The optimal grasping posture is selected as the control command, and the robotic arm is driven to move the gripper to grasp the target object.

[0164] Specifically, any set of grasping posture results can be selected. An optimal grasping posture is encapsulated as a control command and sent to the motion control module of the robotic arm through a standard interface to drive the end effector (gripper) to perform the corresponding grasping task.

[0165] In summary, this method first extracts modal features from point cloud data and natural language instructions. Based on referential segmentation and a spatial awareness model, it obtains the potential locations of words in the point cloud data, marking the spatial regions most relevant to the target language semantics. By introducing a spatial awareness aggregation mechanism, it constructs spatial location encoding vectors and superpoint location encoding vectors, which are then fused with the surrounding point cloud structure. A multi-head attention module enhances the structural awareness of the subject of the natural language instructions, further optimizing the target semantic response map and generating a final target mask—the semantic mask—with a more coherent spatial structure and more accurate semantic matching. In the refined target region, a neighborhood is constructed for each candidate point, local geometric features are extracted, and a grabbing candidate point scoring function is constructed to score the grabbing feasibility of the grabbing posture. A neighborhood average score fusion strategy improves stability. High-scoring candidate points are selected to ensure the rationality of the spatial distribution of grabbing actions. For each candidate point in the grabbing candidate point set, a proximity direction is sampled to form the grabbing posture parameter space, thus obtaining the corresponding six-degree-of-freedom grabbing posture set. The grasping postures in the grasping posture parameter space of each candidate point are filtered through geometric evaluation and collision detection. Qualified grasping postures are retained as candidate grasping postures, and their grasping quality is evaluated using a stability scoring function. Finally, the grasping posture with the highest score is output for each candidate point. The grasping posture is represented as a six-degree-of-freedom parameter format including position, orientation, and execution parameters, and sent to the robotic arm through a standard interface to realize the automatic grasping task of target objects driven by natural language commands. It supports the output of multiple grasping postures to adapt to multi-target scheduling and dynamic scene changes.

[0166] Before practical use, the reference segmentation and spatial awareness model of this invention undergoes model training. First, a dataset is obtained consisting of preprocessed point cloud data and natural language instructions (i.e., modal features of the point cloud data and modal features of the natural language instructions). The natural language instructions are encoded in 768 dimensions by a pre-trained model (such as the BERT model). The input point cloud data is normalized to [-1,1], with the number of points controlled to within 50,000. Then, the dataset is used as input to the reference segmentation and spatial awareness model. Specifically, during training, weakly supervised training is performed on the semantic mask generation module based on reference expression (steps S21 and S22). The modal features of the point cloud data and the modal features of the natural language instructions are aligned and uniformly projected into a 128-dimensional space. A dependency-driven interaction mechanism and a spatial awareness aggregation mechanism are introduced to enhance the semantic spatial consistency between the subject and the target object. The loss function uses a weighted combination of binary cross-entropy loss and Dice loss. For the rule-guided grasping scoring module (step S23) and grasping pose generation module (step S24), firstly, local geometric features are extracted from points within each semantic mask region and input into the grasping candidate point scoring function to obtain a grasping score, with L1 loss used for supervision. Subsequently, a grasping pose parameter space is generated for each candidate point, and geometric evaluation and collision detection are performed using Open3D and FCL software respectively. The grasping poses are then ranked and optimized based on the scores to obtain a grasping pose result set. The Adam optimizer is used during training, with an initial learning rate of 0.001, a batch size of 16, and 150 training epochs. Specific training procedures can be set according to actual needs; these techniques are well-known to those skilled in the art and will not be elaborated upon here. In practical use, the preprocessed point cloud data to be executed and the natural language command are input, and the corresponding grasping posture result set can be output by the trained label segmentation and spatial awareness model. Then, a control command can be selected to perform the corresponding grasping task. The label segmentation and spatial awareness model has strong versatility and grasping success rate, and is suitable for various complex language expressions and multi-object point cloud scenarios.

[0167] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] The above embodiments are merely illustrative of more specific and detailed examples described in this application, and should not be construed as limiting the scope of the application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A robotic arm grasping method based on referential segmentation and spatial perception, characterized in that: Includes the following steps: S1. Obtain point cloud data and natural language commands in the robotic arm grasping scene, and perform preprocessing to extract the corresponding modal features, namely the superpoint set and the coordinates of the superpoints in the superpoint set as the modal features of the point cloud data, which are then converted into embedding vectors by the BERT model as the modal features of the natural language commands. S2. Input the modal features of the point cloud data and the modal features of the natural language commands into the index segmentation and spatial awareness model to obtain the grasping posture result set. The referencing segmentation and spatial awareness model performs the following operations: S21. Generating spatial location estimation of natural language instructions based on modal features of point cloud data and modal features of natural language instructions. The details are as follows: S211, embedding vector and super point set Initial cross-modal fusion is performed separately to obtain corresponding language embedding features. and point cloud embedding features The formula is as follows: ； in, Represents the embedding vector The learnable mapping matrix, Represents the set of superpoints The learnable mapping matrix, DDI(•), represents the dependency-driven interaction mechanism, which achieves semantic alignment between language embedding features and point cloud embedding features based on the dependency syntax structure of natural language instructions. The feature dimension of the superpoint is represented. Indicates the projection dimension. The vector dimension representing a word; S212. Calculate the text embedding and superpoint set for each word. The similarity of the Chinese Super League points, then the first Text embedding of 1 word With superpoint set The j-th superpoint similarity The formula is as follows: ； in, ； in, Text embedding representing all words, S represents the superpoint feature. , Representing language embedding features Learnable matrix, Represents the set of superpoints Learnable matrix, That is to say, the first The probability that a word is located in the j-th superpoint. , sim(•) represents the similarity function, defined as follows: , For transpose, Indicates the number of words, Indicates the number of points exceeding the limit; S213. Arrange the initial positions of all words in the point cloud space into vectors to serve as spatial location estimates for natural language instructions. Then the first The initial position of each word in the point cloud space The formula is as follows: ； in, Let J be the spatial coordinates of the j-th superpoint; S22. Estimation of spatial location based on modal features of point cloud data and natural language commands. Using spatially perceptual aggregation mechanisms, semantic masks are generated for the subject of natural language instructions. The details are as follows: S221. Estimate the spatial location of each natural language instruction. and super set Coordinates of the Chinese Super League Correspondingly construct spatial location encoding vector and superpoint location encoding vector The formula is as follows: ； Where APE(•) represents the absolute position coding function, that is, the absolute position coding function mapped by the multilayer perceptron; S222, Encode vectors based on spatial location and superpoint location encoding vector Construct query vector and key vector The formula is as follows: ； in, A learnable query weight matrix. The key weight matrix is ​​a learnable matrix. For splicing operations; S223, Query vector and key vector Input a multi-head attention module to obtain a spatially aware embedding representation of the subject of a natural language instruction. : ； in, The weight matrix is ​​a learnable value matrix. This represents the softmax function; S224. Computing the spatially-aware embedding representation of the subject of a natural language instruction With superpoint set By analyzing the response values ​​of all superpoints and generating the target semantic response map, the spatially perceived embedding representation of the subject of the natural language instruction is obtained. With superpoint set The j-th superpoint response value The formula is as follows: ； in, For the Sigmoid function, That is, it represents the subject and the j-th superpoint. Similarity; S225. Binarize the target semantic response map and generate a semantic mask. The formula is as follows: ； Where θ is a preset binarization threshold. This indicates the corresponding response value. This indicates whether a point in the cluster to which each superpoint belongs belongs to the target object. 1 represents yes, and 0 represents no; S23. Based on semantic masking The candidate point set is filtered using the candidate point scoring function. ; S24. Separately capture the candidate point set. A grasping posture parameter space is constructed for each candidate point, and the optimal grasping posture for each candidate point is selected from the grasping posture parameter space. The optimal grasping postures of all candidate points form a grasping posture result set. ; S3, Select any grasping posture result set The optimal grasping posture is selected as the control command, and the robotic arm is driven to move the gripper to grasp the target object.

2. The robotic arm grasping method based on referential segmentation and spatial perception as described in claim 1, characterized in that: The preprocessing of the point cloud data is as follows: 1) Voxel downsampling: Voxelize the point cloud data according to a preset voxel threshold to obtain downsampled point cloud data. N represents the number of points in the downsampled point cloud data; 2) Normal Vector and Geometric Feature Extraction: Based on a preset number of neighborhood points, the normal vectors of each point in the downsampled point cloud data are estimated using the least squares method by fitting a plane, and geometric features are formed. The geometric features include the spatial coordinates and normal vector of the corresponding point; 3) Normalization: Normalize the downsampled point cloud data to a unit cube; 4) Statistical filtering: Calculate the average distance of the neighborhood points for each point after normalization, and remove the corresponding points that exceed the average distance by a preset multiple of the standard deviation as outliers. 5) Superpoint partitioning: Clustering algorithms are used to partition the statistically filtered point cloud data into superpoint sets. ,in, Represent the feature dimension of the superpoints and extract the set of superpoints. Coordinates of the Chinese Super League Set of superpoints and super set Coordinates of the Chinese Super League As modal features of point cloud data; The preprocessing of the natural language instructions is as follows: 1) Word segmentation and dependency relations: A dependency syntax tree is constructed using natural language processing tools, and words are extracted, including subjects, adjectives, and prepositional phrases; 2) Word embedding encoding: Converting all words into embedding vectors using the BERT model. , as modal features of natural language instructions.

3. The robotic arm grasping method based on referential segmentation and spatial perception as described in claim 2, characterized in that: The clustering algorithm is the K-means algorithm.

4. The robotic arm grasping method based on referential segmentation and spatial perception as described in claim 2, characterized in that: The semantic mask The constructed crawl candidate point scoring function filters out the crawl candidate point set. The details are as follows: S231. Extract all semantic masks from the downsampled point cloud data. The cluster of superpoints with a value of 1 forms the target candidate point set. , This indicates the number of candidate points in the target candidate point set; S232, For the u-th candidate point , The local geometric features of the corresponding candidate points are extracted by constructing a neighborhood with a preset radius threshold r. The local geometric features include normal vector, curvature, local point density, and normal consistency. S233. Construct a scoring function for candidate points, mapping the local geometric features of each candidate point to a score value. The formula is as follows: ； in, Represents the u-th candidate point The k-th dimension normalized local geometric feature, The weight coefficients represent the normalized local geometric features in the k-th dimension. , Dimensions representing local geometric features; S234. Based on the preset capture scoring threshold Filter the candidate points, that is, retain all score values. Greater than the capture score threshold The candidate points form an initial screening set, and it is determined whether the number of candidate points in the initial screening set exceeds the preset number of candidate points. If so, the farthest point sampling is performed to select a first preset number of candidate points from the initial screening set to form a capture candidate point set. Otherwise, the initial screening set is directly used as the candidate set for crawling. ,in, , M is the set of candidate points to be captured. The number of candidate points in the data, and the capture of the candidate point set. Each candidate point contains spatial coordinates and a score.

5. The robotic arm grasping method based on referential segmentation and spatial perception as described in claim 1, characterized in that: The respective crawling of candidate point sets A grasping posture parameter space is constructed for each candidate point, and the optimal grasping posture for each candidate point is selected from the grasping posture parameter space. The optimal grasping postures of all candidate points form a grasping posture result set. The details are as follows: S241, For the nth candidate point The grasping posture parameter space is constructed as follows: With the nth candidate point A local coordinate system is constructed to determine the center of the grasp. An approach direction is sampled on a hemispherical surface of radius R that is away from the target object in the local coordinate system. There are L grasping postures under each approach direction, which forms the nth candidate point. Grasping posture parameter space L is the second preset quantity. For the nth candidate point The qth grasping posture; S242. Filter the grasping postures in the grasping posture parameter space of each candidate point and retain the qualified grasping postures as candidate grasping postures. S243. Calculate and sort the scores of all grabbing postures in the candidate grabbing postures, and retain the grabbing posture with the highest score among the candidate grabbing postures of each candidate point as the optimal grabbing posture of the corresponding candidate point. S244. Obtain the grasping posture result set , Indicates the nth candidate point The optimal grasping posture, the grasping posture result set Each optimal grasping posture has parameters , means as follows: ； in, To capture the center position, To capture the X-axis coordinate of the center, To capture the Y-axis coordinate of the center, To capture the Z-axis coordinate of the center; The approach direction, i.e., the initial approach unit vector of the robotic arm's gripper, determines the orientation of the robotic arm's gripper relative to the target object. The attitude rotation angle is the rotation angle of the gripper of the robotic arm relative to the approach direction; d represents the opening width, which is the angle of the gripper of the robotic arm; d represents the approach depth, which is the depth to which the gripper of the robotic arm advances along the approach direction.

6. The robotic arm grasping method based on referential segmentation and spatial perception as described in claim 5, characterized in that: The process of filtering the grasping postures in the grasping posture parameter space for each candidate point includes: 1) Geometric evaluation: Select grasping postures that simultaneously satisfy the conditions that the included normal angle is less than a preset degree and the curvature is less than a preset value; 2) Collision detection: Remove grabbing postures with collision risks from the grabbing postures selected by geometric evaluation.

7. The robotic arm grasping method based on referential segmentation and spatial perception as described in claim 5, characterized in that: The scores of all grasping postures in the candidate grasping postures are calculated based on a stability scoring function, which includes a first MLP model, a second MLP model, a normalization function, and an activation function connected in sequence. The score for the nth candidate point... The qth grasping posture rating The formula is as follows: ； in, This represents the stability scoring function.

Images