Open vocabulary capturing method and system based on Gaussian force field

Abstract

The invention discloses an open vocabulary grabbing method and system based on a Gaussian force field, and relates to the technical field of artificial intelligence, a grabbing instruction is input into a trained posture grabbing model in a robot, and a robot grabbing result is obtained; the training process of the posture grabbing model comprises the following steps: constructing a target object point set, a holder model point set and an attraction point set; obtaining a transformed holder model point set and a transformed attraction point set through the transformation matrix; calculating the exponential function sum # imgabs0 # of the distance between all point pairs in the target object point set, the transformed holder model point set and the transformed attraction point set; minimizing the sum # imgabs1 # of exponential functions, obtaining balance between the repulsive force and the attractive force, and optimizing trainable parameters in the attitude grabbing model by adopting an iterative gradient descent method; according to the open vocabulary grabbing method and system, the three-dimensional Gaussian model embedded in the language is combined with the Gaussian force field, and open vocabulary grabbing of the robot in the complex three-dimensional environment is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an open vocabulary crawling method and system based on Gaussian force field. Background Art

[0002] 1) Robot grasping posture detection; The goal of robot grasping posture detection is to determine the optimal posture of a robot hand (such as a manipulator or gripper) in order to firmly grasp the target object. Early methods usually simplified the grasping task into a two-dimensional problem, defining the grasping posture as a directed rectangle of fixed height. Since these methods ignore the complexity in three-dimensional space, they can only achieve three-degree-of-freedom operations and cannot meet the requirements of accuracy and flexibility in practical applications.

[0003] To overcome this limitation, researchers have begun to explore six-degree-of-freedom grasping techniques. These methods combine depth information or point cloud data to enable robots to perceive the 3D geometry and spatial position information of objects, thereby generating more accurate grasping postures. In the case of accurate depth data, these methods have achieved remarkable results. However, in the real world, it is not always easy to obtain high-quality depth information. For example, for transparent or reflective objects, the performance of depth sensors can be severely affected.

[0004] At the same time, with the rapid development of deep learning and computer vision technology, neural network-based grasping posture detection methods have gradually emerged. These methods use a large amount of data to train convolutional neural networks, enabling them to extract rich features from images. However, these methods usually require a large amount of labeled data for training, and obtaining this data is often costly. In addition, these methods have weak generalization capabilities when dealing with objects with irregular or complex shapes, making it difficult to meet the needs of practical applications.

[0005] 2) 3D feature fields for manipulation; In order to improve the robot's ability to understand the three-dimensional environment, some researchers have proposed the idea of ​​constructing a three-dimensional feature field. The three-dimensional feature field can represent the geometric and semantic information in the scene in a unified way, providing the robot with a more complete environmental perception ability. At present, the methods of constructing a three-dimensional feature field are mainly divided into two categories: implicit representation and explicit representation.

[0006] Implicit representation methods, such as Neural Radiance Field (NeRF), learn a continuous scene representation through a neural network, which can achieve high-quality scene reconstruction and new perspective rendering. However, NeRF requires a lot of computing resources and has a slow rendering speed, making it difficult to meet real-time requirements. In addition, it is difficult to directly extract explicit geometric information from implicit representations, which limits its application in robotic manipulation.

[0007] Explicit representation methods use explicit three-dimensional geometric structures to represent scenes. For example, the 3D Gaussian Splatting (3DGS) technique uses Gaussian points in space to explicitly represent scenes. This method can achieve high-quality scene rendering at a low computational cost, while facilitating the direct extraction of geometric information. The 3DGS-based method further combines the visual-language model to embed language features into Gaussian points, and constructs a language embedded Gaussian model (Language Embedded Gaussian). This model can support open vocabulary scene segmentation and semantic query, providing new possibilities for robots to understand and operate the environment.

[0008] However, existing methods still have some shortcomings. First, they usually rely on a large amount of labeled data, and obtaining and labeling this data requires a lot of manpower and material resources. Second, when dealing with dynamic scenes or scene changes, these methods are inefficient and it is difficult to update scene information in a timely manner. In addition, for objects with complex or irregular shapes, the generalization ability of existing methods still needs to be improved. In summary, there is an urgent need for an efficient and general method to enable robots to understand complex three-dimensional environments and operate based on natural language instructions.

[0009] Disadvantage 1 of existing technologies: Slow rendering speed and difficulty in extracting clear geometric information. Although the neural radiance field (NeRF)-based methods have made progress in scene modeling, they are difficult to apply to robot operations in real time due to their slow rendering speed. At the same time, these methods have difficulty in extracting clear geometric features from the scene, which limits their application in grasping tasks.

[0010] Disadvantage 2 of existing technology: reliance on a large amount of annotated data. Existing grasping methods usually require a large amount of annotated grasping posture data. The acquisition and annotation process of this data is time-consuming and costly, and it is difficult to meet the needs of practical applications.

[0011] Disadvantage 3 of existing technology: Poor handling of irregular or complex shaped objects. When faced with irregular or complex shaped objects, the generalization ability of existing methods is weak and it is difficult to accurately predict stable grasping postures.

[0012] The above shortcomings limit the robot's ability to perform open vocabulary grasping tasks based on natural language instructions in complex three-dimensional environments. An efficient and universal method is urgently needed to solve these problems. Summary of the invention

[0013] Based on the technical problems existing in the background technology, the present invention proposes an open vocabulary capture method and system based on Gaussian force field, which combines the language embedded three-dimensional Gaussian model with the Gaussian force field to realize the open vocabulary capture of the robot in a complex three-dimensional environment.

[0014] The present invention proposes an open vocabulary grasping method and system based on Gaussian force field, which inputs grasping instructions into a posture grasping model that has been trained in a robot to obtain a grasping result of the robot; The training process of the posture grasping model is as follows: Constructing a target object point set, a gripper model point set and an attraction point set, and defining that there is a repulsive force between the target object point set and the gripper model point set, and there is an attractive force between the target object point set and the attraction point set; Constructing a transformation matrix for each point in the target object point set, mapping the gripper model point set and the attraction point set to the local coordinate system of each point based on the transformation matrix, and obtaining a transformed gripper model point set and a transformed attraction point set; Calculate the sum of the exponential functions of the distances between all pairs of points in the target object point set, the transformed gripper model point set, and the transformed attractor point set ,Will As a repulsive force function, As an attraction function; Minimize the sum of exponential functions , a balance is achieved between repulsion and attraction, and the trainable parameters in the posture grasping model are optimized using an iterative gradient descent method.

[0015] Furthermore, the process of constructing the target object point set is as follows: Obtain multi-view RGB video sequences and natural language instructions; Track and segment objects in RGB video sequences based on a visual segmentation model to obtain temporally consistent object masks; Extract semantic features from the segmented object mask based on the CLIP model; Through dimensionality reduction processing, the extracted semantic features are embedded into a three-dimensional Gaussian model to construct a language embedding Gaussian model; Extract semantic features from natural language instructions based on the CLIP model to obtain language instruction features; The semantic similarity between the language instruction features and the semantic features of each Gaussian point in the scene is calculated, and the Gaussian points corresponding to the semantic similarity greater than a fixed threshold are segmented to obtain the target object point set.

[0016] Furthermore, a gripper model point set is obtained by sampling the three-dimensional model of the gripper; and an attraction point set is obtained by uniformly sampling the space region between the gripper jaws.

[0017] Furthermore, the transformation matrix of each point in the target object point set is constructed , specifically: ; in, For point The spatial coordinates of For point The rotation matrix of the normal vector is and are the learnable parameters of the posture grasping model, corresponding to the points along The translation in the direction of the normal vector and around the point The rotation angle of the normal vector of point is the first points.

[0018] Furthermore, the transformed gripper model point set , is the gripper model point set, is the transformation matrix; Transformed attraction point set , The set of attraction points.

[0019] Furthermore, the absolute values ​​of the repulsive force function and the attractive force function are the same, and the calculation formula of the repulsive force function is as follows:

[0020] in, is the transformed gripper model point set and the target object point set The repulsive force between and are the learnable parameters of the posture grasping model, express The Points, The target object point set The Points, is the Euclidean distance between two points, is the temperature parameter, is the number of points in the gripper model point set, is the number of points in the target object point set.

[0021] Furthermore, during the training of the posture grasping model and the online execution after the training is completed, the minimum force function value corresponding to each point in the target object point set is recorded; Select from all points in the target object point set such that the total force function The smallest point and its corresponding parameters and ; Based on parameters and Calculate the optimal transformation matrix , and applied to the gripper model point set and the attraction point set, the optimal gripper posture can be obtained.

[0022] An open vocabulary grasping system based on Gaussian force field, which inputs grasping instructions into the trained posture grasping model in the robot to obtain the robot grasping result; The training process of the posture grasping model includes a point set construction unit, a point set transformation unit, a calculation unit, and a gradient optimization unit; The point set construction unit is used to construct the target object point set, the gripper model point set and the attraction point set, and defines that there is a repulsive force between the target object point set and the gripper model point set, and there is an attractive force between the target object point set and the attraction point set; The point set transformation unit is used to construct a transformation matrix for each point in the target object point set, and based on the transformation matrix, the gripper model point set and the attraction point set are mapped to the local coordinate system of each point to obtain a transformed gripper model point set and a transformed attraction point set; The calculation unit is used to calculate the sum of the exponential functions of the distances between all pairs of points in the target object point set, the transformed gripper model point set, and the transformed attraction point set. ,Will As a repulsive force function, As an attraction function; The gradient optimization unit is used to minimize the sum of exponential functions , a balance is achieved between repulsion and attraction, and the trainable parameters in the posture grasping model are optimized using an iterative gradient descent method.

[0023] Furthermore, a gripper model point set is obtained by sampling the three-dimensional model of the gripper; and an attraction point set is obtained by uniformly sampling the space region between the gripper jaws.

[0024] Furthermore, the construction process of the target object point set in the point set construction module is as follows: Obtain multi-view RGB video sequences and natural language instructions; Track and segment objects in RGB video sequences based on a visual segmentation model to obtain temporally consistent object masks; Extract semantic features from the segmented object mask based on the CLIP model; Through dimensionality reduction processing, the extracted semantic features are embedded into a three-dimensional Gaussian model to construct a language embedding Gaussian model; Extract semantic features from natural language instructions based on the CLIP model to obtain language instruction features; The semantic similarity between the language instruction features and the semantic features of each Gaussian point in the scene is calculated, and the Gaussian points corresponding to the semantic similarity greater than a fixed threshold are segmented to obtain the target object point set.

[0025] The advantages of the open vocabulary crawling method and system based on Gaussian force field provided by the present invention are: by constructing a Gaussian model of language embedding, the system has the scene understanding ability of open vocabulary, can segment and identify any object in a complex scene, support natural language-based queries and operations, and realize semantic understanding of complex three-dimensional environments. At the same time, the proposed Gaussian force field method does not require a large amount of labeled data, and can estimate the six-degree-of-freedom posture of the gripper by self-supervision, reducing the cost and difficulty of data acquisition. By balancing the repulsive force and attractive force in the Gaussian force field, the posture of the gripper is quickly optimized by the gradient descent method, and efficient and stable crawling of the target object is achieved, which is suitable for real-time applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the training process of the posture grasping model of the present invention; Figure 2 A schematic diagram of a Gaussian force field, showing the force relationship between the target object point set, the gripper model point set, and the attracting point set; Figure 3 Schematic diagram of the application of open vocabulary crawling method in a simulated environment. DETAILED DESCRIPTION

[0027] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0028] like Figures 1 to 3 As shown, the present invention proposes an open vocabulary grasping method based on Gaussian force field, which inputs the grasping instruction into the posture grasping model that has been trained in the robot to obtain the robot grasping result; The training process of the posture grasping model is as follows: Constructing a target object point set, a gripper model point set and an attraction point set, and defining that there is a repulsive force between the target object point set and the gripper model point set, and there is an attractive force between the target object point set and the attraction point set; Constructing a transformation matrix for each point in the target object point set, mapping the gripper model point set and the attraction point set to the local coordinate system of each point based on the transformation matrix, and obtaining a transformed gripper model point set and a transformed attraction point set; Calculate the sum of the exponential functions of the distances between all pairs of points in the target object point set, the transformed gripper model point set, and the transformed attractor point set ,Will As a repulsive force function, As an attraction function; Minimize the sum of exponential functions , a balance is achieved between repulsion and attraction, and the trainable parameters in the posture grasping model are optimized using an iterative gradient descent method.

[0029] This embodiment uses 3D Gaussian Splatting (3DGS) and a visual-language basic model to build a language-embedded Gaussian model to achieve 3D scene segmentation under open language queries. In addition, a Gaussian Force Field, a self-supervised gripper posture estimation method, is proposed to avoid the reliance on a large amount of labeled grasping posture data.

[0030] The core idea of ​​this embodiment is to combine the language-embedded three-dimensional Gaussian model with the Gaussian force field to realize the open vocabulary grasping of the robot in a complex three-dimensional environment. First, using pre-trained models such as SAM2 and CLIP, semantic and feature information is extracted from the video sequence of the scene, and a language-embedded Gaussian model is constructed to provide an open vocabulary understanding of the scene. Then, for the specified natural language instructions, the target object is segmented and the corresponding Gaussian force field is constructed. The Gaussian force field consists of a point cloud of the target object, a point cloud of the gripper model, and attraction points. By designing a suitable force model, the repulsive force between the positive point sets prevents the gripper from colliding with the object, and the attraction between the positive and negative point sets promotes the gripper to a stable grasping posture. Finally, by performing iterative gradient optimization on the force field, the optimal six-degree-of-freedom posture of the gripper is predicted and obtained, and stable grasping of the target object is achieved.

[0031] The implementation of the open vocabulary crawling method of this embodiment is mainly achieved by relying on the following modules: (1) Scene model construction module: Use three-dimensional Gaussian scattering (3DGS) technology to process the input multi-view RGB video sequence and construct a three-dimensional Gaussian model of the scene; (2) Gaussian model building module for language embedding: Using the SAM2 and CLIP models, objects in the scene are tracked and segmented, the language features of the objects are extracted, and embedded into the Gaussian model to achieve open vocabulary scene representation; SAM2 is a new generation of image and video segmentation model released by Meta; the CLIP model is a multimodal pre-trained neural network proposed by OpenAI in 2021; For the input multi-view RGB video sequence, the SAM2 model is used to track and segment the objects in the video sequence to obtain a temporally consistent object mask. The CLIP model is used to extract semantic features from the segmented object mask, and the high-dimensional language features are embedded into a three-dimensional Gaussian model through dimensionality reduction to construct a language embedded Gaussian model.

[0032] In one of the embodiments, the process of constructing the target object point set is as follows: obtain a multi-view RGB video sequence and a natural language instruction; track and segment the objects in the RGB video sequence based on a visual segmentation model to obtain a time-consistent object mask; extract semantic features from the segmented object mask based on a CLIP model; embed the extracted semantic features into a three-dimensional Gaussian model through dimensionality reduction processing to construct a language embedded Gaussian model; extract semantic features from the natural language instruction based on the CLIP model to obtain a language instruction feature; calculate the semantic similarity between the language instruction feature and the semantic feature of each Gaussian point in the scene, segment the Gaussian points corresponding to the semantic similarity greater than a fixed threshold, and obtain the target object point set.

[0033] By building a Gaussian model of language embedding, the system is equipped with open vocabulary scene understanding capabilities, can segment and identify any object in complex scenes, support natural language-based queries and operations, and achieve semantic understanding of complex three-dimensional environments.

[0034] (3) Gaussian force field construction module: Based on the specified natural language instructions, the Gaussian point cloud of the target object is segmented and combined with the gripper model to construct the Gaussian force field.

[0035] The construction of Gaussian force field is the core of this embodiment. By simulating the interaction between the gripper and the target object in the Gaussian force field, the self-supervised optimization of the gripper posture is achieved. Specifically, the Gaussian force field consists of a target object point set, a gripper model point set, and an attraction point set. By setting the force relationship between the positive and negative point sets, the gripper is guided to achieve the optimal grasping posture.

[0036] (3-1) Definition of Gaussian force field; The Gaussian force field consists of the following three types of point sets: Target object point set : The Gaussian point set of the target object obtained from (2) is composed of ,in , is the number of points in the target object point set, The target object Gaussian points.

[0037] Gripper Model Point Set : It is obtained by sampling the three-dimensional model of the gripper and is expressed as ,in , is the number of points of the gripper model.

[0038] Attraction point set : The spatial region between the gripper jaws is uniformly sampled and expressed as ,in , is the number of points in the attraction point set.

[0039] In the Gaussian force field: target object point set and the gripper model point set It is defined as a positive point set, represented by the symbol "+". There is a repulsive force between the positive point sets, preventing the gripper from colliding with the target object. It is defined as a negative point set, represented by the symbol “-”. Target object point set With the set of attraction points There is an attractive force between them, guiding the gripper towards the optimal grasping position of the target object.

[0040] (3-2) Gripper posture parameterization; For the target object point set Each point in , set learnable parameters for the gripper’s posture and , and They are defined as The translation amount of the normal vector direction (usually taken as the normal vector of the target object surface) and the rotation amount around the point Based on the above parameters, define the gripper at point Rigid body transformation at First, set the point The spatial coordinates of , the corresponding rotation matrix of the normal vector is .in and They are important groups for describing rigid body motion, representing the Special Euclidean Group and the Special Orthogonal Group in three-dimensional space. is the group of all 3×3 rotation matrices, is the group of all 4×4 rigid body transformation matrices. Then, the transformation matrix It is expressed as: ; (1) Among them, the first matrix Indicates that the gripper is placed at point The second matrix Represents rotation and translation about the normal vector.

[0041] (3-3) Definition of force function; Through the transformation matrix , the gripper model point set and the set of attraction points Map to point The local coordinate system of the gripper model is obtained by transforming the point set and the transformed set of attraction points Next, define the repulsive force and attractive force between the gripper and the target object: and the attraction function The repulsive force function is is the transformed gripper model point set and the target object point set The repulsive and attractive forces between is the transformed attraction point set and the target object point set The absolute values ​​of the attraction, repulsion and attraction functions are the same, which can be summarized as the sum of the exponential functions of the distances between all point pairs. The calculation formula of the repulsion function is as follows: ; (2) in, is the transformed gripper model point set The Points, yes The Points, represents the Euclidean distance between two points, is the temperature parameter, controlling the attenuation of the force, and are the learnable parameters of the posture grasping model, is the number of points in the gripper model point set, is the number of points in the target object point set. The attraction function is similar to the repulsion function, but with a negative sign in the expression.

[0042] (4) Gripper posture optimization module: By optimizing the Gaussian force field and using the gradient descent method, the optimal six-degree-of-freedom posture of the gripper is predicted and obtained.

[0043] The goal of this example is to find the optimal parameters and , so that the total force function By minimizing the above formula (2), a balance can be achieved between the repulsive force and the attractive force, avoiding the collision between the gripper and the target object while guiding the gripper to the optimal grasping position and posture. The iterative gradient descent method is used to adjust the parameters and Optimize and update the parameters according to the gradient descent method. During the optimization process, record each point The corresponding minimum force function value. Finally, from all points Select the total force function The smallest point and its corresponding parameters and . The optimal transformation matrix Applied to the gripper model point set and the attraction point set, the optimal gripper posture can be obtained. This posture ensures that the repulsive force between the gripper and the target object is minimized and the attraction force is maximized, achieving stable grasping of the target object.

[0044] This embodiment proposes to balance the repulsive force and attractive force in a Gaussian force field and use a gradient descent method to quickly optimize the posture of the gripper, thereby achieving efficient and stable grasping of the target object, which is suitable for real-time applications.

[0045] In summary, this embodiment, by constructing a language-embedded Gaussian model, enables the system to have the ability to understand scenes with open vocabulary, can segment and identify any object in a complex scene, supports natural language-based queries and operations, and realizes semantic understanding of complex three-dimensional environments. At the same time, the proposed Gaussian force field method does not require a large amount of labeled data, and can estimate the six-degree-of-freedom posture of the gripper by self-supervision, reducing the cost and difficulty of data acquisition. By balancing the repulsive force and attractive force in the Gaussian force field, the posture of the gripper is quickly optimized using the gradient descent method, and efficient and stable grasping of the target object is achieved, which is suitable for real-time applications. In addition, the method of this embodiment can still perform grasping tasks on unseen objects in the case of zero samples, especially when dealing with objects with complex or irregular shapes, showing a strong generalization ability, which significantly improves the robot's ability to operate in complex environments.

[0046] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An open vocabulary crawling method based on Gaussian force field, characterized in that: Input the grasping instruction into the posture grasping model that has been trained in the robot to obtain the robot grasping result; The training process of the posture grasping model is as follows: Constructing a target object point set, a gripper model point set and an attraction point set, and defining that there is a repulsive force between the target object point set and the gripper model point set, and there is an attractive force between the target object point set and the attraction point set; Constructing a transformation matrix for each point in the target object point set, mapping the gripper model point set and the attraction point set to the local coordinate system of each point based on the transformation matrix, and obtaining a transformed gripper model point set and a transformed attraction point set; Calculate the sum of the exponential functions of the distances between all pairs of points in the target object point set, the transformed gripper model point set, and the transformed attractor point set ,Will As a repulsive force function, As an attraction function; Minimize the sum of exponential functions ,The iterative gradient descent method is used to optimize the trainable parameters in the pose grasping model.

2. The open vocabulary crawling method based on Gaussian force field according to claim 1 is characterized in that: The process of constructing the target object point set is as follows: Obtain multi-view RGB video sequences and natural language instructions; Track and segment objects in RGB video sequences based on a visual segmentation model to obtain temporally consistent object masks; Extract semantic features from the segmented object mask based on the CLIP model; Through dimensionality reduction processing, the extracted semantic features are embedded into a three-dimensional Gaussian model to construct a language embedding Gaussian model; Extract semantic features from natural language instructions based on the CLIP model to obtain language instruction features; The semantic similarity between the language instruction features and the semantic features of each Gaussian point in the scene is calculated, and the Gaussian points corresponding to the semantic similarity greater than a fixed threshold are segmented to obtain the target object point set.

3. The open vocabulary crawling method based on Gaussian force field according to claim 1, characterized in that: The gripper model point set is obtained by sampling the three-dimensional model of the gripper; and the attraction point set is obtained by uniformly sampling the space area between the gripper jaws.

4. The open vocabulary crawling method based on Gaussian force field according to claim 1, characterized in that: Construct the transformation matrix for each point in the target object point set , specifically: ； in, For point The spatial coordinates of For point The rotation matrix of the normal vector is and are the learnable parameters of the posture grasping model, corresponding to the points along The translation in the direction of the normal vector and around the point The rotation angle of the normal vector of point is the first points.

5. The open vocabulary crawling method based on Gaussian force field according to claim 1, characterized in that: The transformed gripper model point set , is the gripper model point set, is the transformation matrix; Transformed attraction point set , The set of attraction points.

6. The open vocabulary crawling method based on Gaussian force field according to claim 1, characterized in that: The absolute values ​​of the repulsive force function and the attractive force function are the same. The calculation formula of the repulsive force function is as follows: in, is the transformed gripper model point set and the target object point set The repulsive force between and are the learnable parameters of the posture grasping model, express The Points, The target object point set The Points, is the Euclidean distance between two points, is the temperature parameter, is the number of points in the gripper model point set, is the number of points in the target object point set.

7. The open vocabulary crawling method based on Gaussian force field according to claim 6, characterized in that: During the training of the posture grasping model and the online execution after the training is completed, the minimum force function value corresponding to each point in the target object point set is recorded; Select from all points in the target object point set such that the total force function The smallest point and its corresponding parameters and ; Based on parameters and Calculate the optimal transformation matrix , and applied to the gripper model point set and the attraction point set, the optimal gripper posture can be obtained.

8. An open vocabulary crawling system based on Gaussian force field, characterized in that: Input the grasping instruction into the posture grasping model that has been trained in the robot to obtain the robot grasping result; The training process of the posture grasping model includes a point set construction unit, a point set transformation unit, a calculation unit, and a gradient optimization unit; The point set construction unit is used to construct the target object point set, the gripper model point set and the attraction point set, and defines that there is a repulsive force between the target object point set and the gripper model point set, and there is an attractive force between the target object point set and the attraction point set; The point set transformation unit is used to construct a transformation matrix for each point in the target object point set, and based on the transformation matrix, the gripper model point set and the attraction point set are mapped to the local coordinate system of each point to obtain a transformed gripper model point set and a transformed attraction point set; The calculation unit is used to calculate the sum of the exponential functions of the distances between all pairs of points in the target object point set, the transformed gripper model point set, and the transformed attraction point set. ,Will As a repulsive force function, As an attraction function; The gradient optimization unit is used to minimize the sum of exponential functions , a balance is achieved between repulsion and attraction, and the trainable parameters in the posture grasping model are optimized using an iterative gradient descent method.

9. The open vocabulary crawling system based on Gaussian force field according to claim 8, characterized in that: The gripper model point set is obtained by sampling the three-dimensional model of the gripper; and the attraction point set is obtained by uniformly sampling the space area between the gripper jaws.

10. The open vocabulary crawling system based on Gaussian force field according to claim 8, characterized in that: The construction process of the target object point set in the point set construction module is as follows: Obtain multi-view RGB video sequences and natural language instructions; Track and segment objects in RGB video sequences based on a visual segmentation model to obtain temporally consistent object masks; Extract semantic features from the segmented object mask based on the CLIP model; Through dimensionality reduction processing, the extracted semantic features are embedded into a three-dimensional Gaussian model to construct a language embedding Gaussian model; Extract semantic features from natural language instructions based on the CLIP model to obtain language instruction features; The semantic similarity between the language instruction features and the semantic features of each Gaussian point in the scene is calculated, and the Gaussian points corresponding to the semantic similarity greater than a fixed threshold are segmented to obtain the target object point set.

Images