Multi-modal large model three-dimensional perception method based on Riemannian manifold priori guidance

Abstract

The invention discloses a multi-modal large model three-dimensional perception method based on Riemannian manifold prior guidance, and relates to a computer vision technology. The method comprises the following steps: firstly, constructing a multi-modal large model and point cloud sensing network coordinated cross attention reinforcement learning joint training framework, so that the large model obtains a three-dimensional scene space topology understanding capability; secondly, a conformal property on a Riemannian manifold is utilized to construct a three-dimensional contour and topological association, and multi-scale invariance of homotopy mapping learning contours is introduced, so that the challenges of shape diversity and great size difference of objects are solved; in addition, a lightweight three-dimensional attention gate filtering mechanism is introduced into a point cloud encoder, and more effective global and local point cloud geometric semantic association is established. And finally, when the motion pose quality is evaluated, fusing prior knowledge of the three-dimensional physical relationship to form mixed physical measurement so as to overcome label noise caused by single evaluation scale. And universal understanding and high-precision perception of the robot on complex three-dimensional environments and objects can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a multi-modal large model three-dimensional perception method based on Riemannian manifold prior guidance. BACKGROUND

[0002] It is a long-term and key goal to realize robots capable of performing human command-driven actions in general scenarios. Among them, the ability of robots to accurately grasp target objects in cluttered and crowded environments constitutes the basis of this goal, and is crucial for applications such as automated home services, manufacturing part selection, and waste sorting. Early researchers usually obtain spatial poses by analyzing object shapes or extracting artificially designed primary features. However, such artificially designed features are difficult to describe complex shapes of objects, and can only express objects that conform to strong structured rules, resulting in a lack of practicality in diversified scenarios. To deal with more complex and variable objects, data-driven deep learning methods are widely used to extract grasping attributes from point clouds and have made some progress in specific scenarios.

[0003] However, existing technologies still face significant challenges in processing object perception in complex three-dimensional scenes. On the one hand, there is a common problem of unbalanced object sizes, which leads to greater uncertainty in the outlines of smaller objects. Existing solutions either have difficulty generalizing to real-world noisy environments or are limited to Euclidean features, failing to fully explore high-dimensional deep geometric and topological features of objects, thus unable to establish a relationship between object outlines and their topological structures, limiting their ability to understand new object shapes. This makes existing methods perform poorly when faced with challenges such as user demand diversification in the real world, significant deviation of object outlines from training data, and large differences in object sizes and diverse outline curvature characteristics. On the other hand, although some research attempts to integrate visual language models to enhance the model's overall understanding of the scene, such methods often rely on a two-stage training process, which is prone to introducing errors; or lack direct compatibility with point cloud data, making them only suitable for planar scenes with image input, resulting in their ability to understand only planar positional relationships. When objects overlap in the vertical direction, these methods fail, greatly limiting their widespread deployment in the real world. In addition, methods that only use text-point cloud data lack filtering of target-related regions, and their search space is the entire point cloud rather than the relevant region as a component, which not only reduces the accuracy of establishing connections, but also increases the overall computational complexity. SUMMARY

[0004] The present application aims at the problems that existing large models are difficult to learn the geometric semantics of target objects in three-dimensional space by using single modal data in complex three-dimensional scenes, and the problems of unbalanced object size, insufficient contour topology exploration and limited multi-modal information fusion in complex scenes in the existing robot grasping method.

[0005] To achieve the above-mentioned application purposes, the present application provides the following technical solutions.

[0006] The present application trains the multi-modal large model and the point cloud network jointly through the cross-attention reinforcement learning framework, and introduces the conformality of negative curvature manifold to solve the problems of unbalanced object size and contour topology understanding. At the same time, combined with the gate filtering mechanism of lightweight three-dimensional attention weight and the evaluation metric of three-dimensional physical relationship priori knowledge, the general understanding and high-precision perception of robots to complex three-dimensional environment and objects are realized.

[0007] The present application specifically includes the following steps:

[0008] 1) Prepare a multi-modal three-dimensional grasping perception dataset;

[0009] 2) Construct a multi-modal large model three-dimensional perception overall framework;

[0010] 3) Design a Riemannian manifold multi-size perception domain regression module based on homotopy mapping;

[0011] 4) Construct a point cloud encoder based on three-dimensional attention mechanism;

[0012] 5) Design a hybrid physical metric method guided by physical priori knowledge;

[0013] 6) Use a cross-attention reinforcement learning algorithm to jointly train the model;

[0014] 7) Use the trained model to generate three-dimensional scene perception and grasping action.

[0015] In step 1), the specific method for preparing the multi-modal three-dimensional grasping perception data set is: using a publicly available large-scale six-degree-of-freedom grasping data set GraspNet-1Billion to train and evaluate the model. The data set contains a total of 97280 RGB-D images and more than 1 billion six-degree-of-freedom grasping poses, and the RGB-D images are taken from different perspectives of 190 cluttered scenes, each containing 512 RGB-D images. The data set provides accurate 3D mesh models for 88 objects and densely annotates the six-degree-of-freedom poses and grasping poses of the objects in each scene. These objects include shampoo, plastic basin, paper packaging box, etc., among which 32 objects are selected from the YCB data set, 13 objects are selected from the Dex-Net 2.0 method, and 43 objects are self-collected. Each object has an accurate computer-aided design (CAD) model, which can be automatically annotated through force analysis. In each scene, about 10 objects are randomly selected from the entire object set and placed randomly. The data set is divided into a training set (scenes 1-100), a "seen" object validation set (scenes 101-130), a "familiar" object validation set (scenes 131-160), and a "unseen" object test set (scenes 161-190). Each scene consists of 256 pictures taken by two cameras attached to the robot arm, covering 256 different viewpoints on a quarter sphere. To overcome the interference caused by incomplete point clouds and improve the detection ability of new objects, the invention reconstructs the depth images of multiple perspectives in a single scene into truncated signed distance functions for training and evaluation.

[0016] In step 2), the specific method for constructing a multi-modal large model three-dimensional perception overall framework is:

[0017] The multi-modal large model receives language instructions and original images containing object clutter;

[0018] The multi-modal large model encodes the language instructions into language features and extracts the bounding box of the target object from the original image;

[0019] The cropped target object is input into the image encoder of the multi-modal large model to generate object-centered bounding box visual features, and the language features and the bounding box visual features are fused to form a joint training value vector;

[0020] A point cloud perception network guided by Riemannian manifold prior is introduced, which takes scene point cloud as input, predicts a set of six-degree-of-freedom motion poses, and encodes them into spatial features as query vectors for the next stage of joint training;

[0021] The motion pose features generated by the point cloud perception network are used as queries, and the target object-centered bounding box visual language features encoded by the multi-modal large model are used as values;

[0022] To further combine the spatial features with the visual language features, the bounding box visual position features are used as the key vectors, the three-dimensional positions of the centers of the bounding boxes are projected to a nonlinear space and encoded by a multilayer perception machine, and then the position embeddings are fused with the visual features of the bounding boxes as the key vectors of the cross-attention layer;

[0023] The cross-attention features of the fusion of the visual, language and action poses obtained after the cross-attention layer processing are input into a cross-attention reinforcement learning network, the network is trained by model-free deep reinforcement learning, and a series of actions are predicted to remove the blocking objects and complete the grasping of the specified target.

[0024] In step 3), the specific method of the Riemannian manifold multi-size perception domain regression module based on homotopy mapping is as follows:

[0025] A plurality of cylinders with different radii and related point cutting are set as the perception domain, and the radii are uniformly distributed within the maximum width range of the manipulator;

[0026] The homotopy mapping is introduced to map the target from the geometric features of the perception domain to the homotopy space and then to the Riemannian manifold, thereby realizing the smooth mapping of the Euclidean space and the Riemannian manifold, and the homotopy function is constructed as:

[0027]

[0028] In the formula, represents the homotopy mapping function, which describes the continuous transformation process from the mapping to the mapping ; represents the point input into the mapping function, which usually represents the geometric features of the perception domain; is a continuously changing parameter, when its norm changes from 0 to 1, the mapping continuously changes from to ; , is a continuous mapping between two topological spaces.

[0029] The cylinder feature vectors with different radii and different heights processed by the homotopy mapping are projected to the Riemannian manifold space, and the Lorentz model is used as the Riemannian manifold;

[0030] The similarity loss is calculated on the multi-scale Riemannian manifold of the three-dimensional contour of the same object, and the calculation method is as follows:

[0031]

[0032] In the formula, denotes a multi-scale profile similarity loss computed on a Riemannian manifold; denotes the number of different size perception domains; denotes a Riemannian distance between two points computed on a Lorentz model with curvature c; denotes the vector of the i-th and j-th different size cylindrical perception domain feature mapped to the Riemannian manifold space.

[0033] The module also introduces a corresponding seed feature as a guide for the fusion of different scale local features, and filters the information in the seed feature using a gate filtering mechanism based on point cloud three-dimensional weight attention. The specific filtering rule is: by calculating the importance weight of each neuron relative to other neurons in the channel, an attention weight matrix is constructed, and then a gating signal is generated through a sigmoid activation function to adaptively weight and filter the information in the seed feature, retaining important geometric features and suppressing noise and redundant information.

[0034] In step 4), the specific method for constructing the point cloud encoder based on the three-dimensional attention mechanism is:

[0035] A three-dimensional attention module for point clouds is proposed, which assigns a unique weight to each neuron, and the importance of the t-th neuron is calculated by an energy function

[0036]

[0037] wherein, denotes the importance measure of the t-th neuron relative to other neurons; denote the weight and bias for linear transformation of the t-th neuron, respectively; denotes the feature vector of the t-th target neuron; denotes the feature vector of the other neurons in the same channel except the target neuron; denotes the total number of neurons in the channel; denotes the weight regularization coefficient.

[0038] And by minimizing the energy function , setting the partial derivatives of it with respect to and to zero, the optimal closed-form solution of the weight and the bias is derived using analytical method, and the neuron importance is calculated to form the weight matrix , and the processed feature vector is obtained by multiplying the feature vector X:

[0039] ​​​

[0040] where, denotes the output feature vector after 3D attention weight processing; denotes the activation function; denotes the weight matrix composed of the importance of all neurons; denotes the input feature vector.

[0041] In step 5), the specific method of designing the physical prior knowledge guided hybrid physical measurement method is:

[0042] Introduce force closure measurement loss:

[0043]

[0044] where, denotes the force closure measurement loss based on the reflection rule; denote the position coordinates of the two contact points in three-dimensional space, respectively; denotes the vector from point to point ; , denote the unit outer normal vectors at the contact points and , respectively; denotes the L2 norm of the vector.

[0045] Introduce flatness measurement loss:

[0046]

[0047] where, denotes the flatness measurement loss of the curvature near the contact point; denotes the number of neighboring points of each contact point; denotes the unit normal vector of the i-th query contact point; denotes the unit normal vector of the j-th neighboring point of the i-th contact point; denotes the dot product operation of the vector.

[0048] Introduce contact area measurement loss:

[0049]

[0050] where, denotes the measurement loss of the contact area; denotes the number of point cloud sampling points in the contact area, which is used to approximate the contact area.

[0051] Introduce the collision constraint indicator collision constraint indicator:​

[0052]

[0053] wherein, represents a constraint index for avoiding collision between the robot hand and the object surface; represents the minimum value of the distances between the two contact points and the object surface; represents the lower limit of the constraint distance; represents the upper limit of the constraint distance; represents the hyperbolic tangent activation function.

[0054] The action score predicted by the action detection network is weighted and regularized to calculate the overall physical constraint loss of the i-th point:

[0055]

[0056] wherein, represents the overall physical constraint loss of the i-th point; represents the grasp action prediction score of the i-th point; respectively represent the weighting coefficients of the flatness, contact area, and collision constraint loss; represents the total number of points.

[0057] Training loss of the point cloud network is the sum of the base loss , the overall physical constraint loss R, and the multi-scale contour similarity constraint loss on the Riemannian manifold

[0058]

[0059] wherein, represents the total training loss of the point cloud network; represents the grasp detection loss of the baseline method; represents the overall physical constraint loss; represents the multi-scale contour similarity constraint loss on the Riemannian manifold weighting coefficient.

[0060] The specific method of jointly training the model using the cross-attention reinforcement learning algorithm in step 6) is as follows:

[0061] The point cloud data processed by the point cloud encoder based on the three-dimensional attention mechanism is used to generate action candidates, and the appropriate perception domain parameters are selected by the Riemannian manifold multi-dimensional perception domain regression module based on homotopy mapping;

[0062] The six-degree-of-freedom pose coordinates of the grasping action are output by the multi-layer perception machine;

[0063] ​​In the cross-attention layer, the action pose coordinates are taken as the query vector, the target object visual language features encoded by the multi-modal large model are taken as the value vector, and the bounding box visual position features are taken as the key vector, so as to realize the fusion of multi-modal information and obtain cross-attention features;

[0064] The cross-attention features are input into the policy network and the critic network, and the discrete SAC algorithm is used for reinforcement learning training, the policy network is defined as , and the critic network outputs the Q function ;

[0065] In the policy evaluation step, the soft state value function is defined as:

[0066]

[0067] In the formula, represents the soft state value function loss when the parameters of the policy are ; represents the policy probability distribution of taking action in state ; represents the Q value function of taking action in state ; represents the temperature parameter, which is used to control the balance between policy exploration and utilization.

[0068] The soft Q function is trained by minimizing the soft Bellman residual, and the calculation formula is:

[0069]

[0070] In the formula, represents the training loss of the Q function; represents the parameters of the Q function network; represents the value output by the Q function with parameters when taking action in state ; represents the target Q value calculated based on the Bellman equation; represents the environment reward obtained at time step t; and respectively represent the state and action of the next time step.

[0071]

[0072] In the formula, represents the target Q value calculated based on the Bellman equation; denotes the environment reward obtained at time step t; denotes the state distribution of taking action at state denotes the soft state value function at next state and action .

[0073]

[0074] where denotes the soft state value function, denotes the value output by the Q function with parameters at next state and action . denotes the temperature parameter for controlling the balance between policy exploration and exploitation; denotes the natural logarithm function; denotes the state distribution of taking action at next state .

[0075] The policy is updated by the soft Q function computed in the policy evaluation step. By multiplying the expected KL divergence by the temperature parameter and ignoring the division function, the policy parameters are learned to obtain the final loss function :

[0076]

[0077] where denotes the loss function of the temperature parameter ; denotes the state distribution of taking action at state ; denotes the target entropy for controlling the randomness degree of the policy.

[0078] In step 7), the specific method of using the trained model for three-dimensional scene perception and grasp action generation is: using the jointly trained multi-modal large model combined with the point cloud perception network to perceive the three-dimensional scene and generate grasp actions. In the inference stage, the model first receives the down-sampled point cloud data, and based on the guidance of multi-modal information fusion and physical prior knowledge, synthesizes and outputs a set of initial grasp pose parameters. Then, in order to obtain the optimal grasp candidate, non-maximum suppression is performed on these grasp pose parameters, and collision checking is performed on the filtered candidate poses, while evaluating whether they meet the force closure condition under the current friction coefficient to ensure physical feasibility. Finally, through this series of screening and evaluation, the six-degree-of-freedom action pose that can guide the robot to accurately grasp the target object is inferred, and the grasp action sequence that can be executed in the actual or simulated environment is generated.

[0079] Compared with the prior art, the present application has the following advantages:

[0080] By introducing the cross-attention reinforcement learning framework to jointly train the multi-modal large model and the point cloud perception network, the present application effectively endows the large model with the ability to understand complex three-dimensional environments and objects, overcoming the limitations of existing large models in three-dimensional space perception, enabling groundable three-dimensional action generation, and greatly expanding the deployment range of robots in the real world. In particular, the present application uses the conformal property of negative curvature manifolds and variable curvature space to solve the problem of declining perception accuracy caused by object size imbalance in existing methods, enabling the model's perception field to automatically adapt to object size and enabling the extraction of the topological structure of object contours, thereby significantly improving the generalization perception ability for new shape objects. At the same time, by introducing a lightweight three-dimensional attention weight gate filtering mechanism in the point cloud Transformer encoder, the present application can establish more effective global and local point cloud depth geometric semantic associations, thereby more accurately understanding the global three-dimensional physical relationships between objects in the scene. Furthermore, by incorporating three-dimensional physical relationship prior knowledge from human experience into pose quality evaluation, a hybrid metric with physical meaning is formed, effectively solving the label noise problem caused by single evaluation scale, significantly increasing the difference between different three-dimensional poses, and enabling the model to learn high-precision geometric information such as occlusion and collision in three-dimensional space, thereby ensuring the high precision and robustness of the robot's grasp action. In summary, the present application realizes the general and high-precision understanding and perception of complex three-dimensional scenes and objects, and can provide more reliable and intelligent robot grasping solutions for applications such as automated home service, manufacturing part selection, and waste classification. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1The complex environment variable curvature geometric prior guidance multimodal large model embodiment of the present application shows how the language instructions, images, point clouds and other multi-modal inputs are encoded, fused and finally generate the complete process of the grasping action.

[0082] Figure 2 The variable curvature manifold multi-size perception domain regression module based on homotopy mapping designed by the present application shows how the perception domains of different sizes are projected to the Riemannian manifold space through homotopy mapping.

[0083] Figure 3 The point cloud 3D attention gate filtering designed by the present application shows the cooperative relationship of the energy function calculation, gate filtering and other core components.

[0084] Figure 4 The point cloud Transformer block structure diagram designed by the present application.

[0085] Figure 5 The point cloud Transformer encoder architecture diagram designed by the present application.

[0086] Figure 6 The hybrid physical metric method guided by the physical prior knowledge proposed by the present application shows how multiple physical metrics such as force closure, flatness, contact area and collision constraint jointly act. DETAILED DESCRIPTION

[0087] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the method of the present application is described in detail below in combination with the drawings and examples. The present embodiment is implemented on the premise of the technical solution of the present application, and the implementation mode and specific operation process are given, but the protection scope of the present application is not limited to the following examples.

[0088] The implementation mode of the embodiment of the present application includes the following steps:

[0089] 1. Prepare a multi-modal three-dimensional grasping perception dataset.

[0090] The disclosed large-scale six-degree-of-freedom grasping dataset GraspNet-1Billion is used for training and evaluation of the model. The dataset contains a total of 97280 RGB-D images and more than 1 billion six-degree-of-freedom grasping poses. The RGB-D images are taken from different perspectives of 190 cluttered scenes, and each scene contains 512 RGB-D images. The dataset provides accurate 3D mesh models for 88 objects and densely labels the six-degree-of-freedom poses and grasping poses of objects in each scene. These objects include shampoo, plastic basin, paper packaging box, etc. Among them, 32 objects are selected from the YCB dataset, 13 antagonistic objects are selected from the Dex-Net 2.0 method, and 43 objects are self-collected. Each object has an accurate computer-aided design (CAD) model, and the grasping pose labeling of the dataset is achieved by calling the calculation interface of the existing physical engine library Bullet Physics and combining the self-programmed automatic labeling logic. The core steps include: first, sampling grasping points on the surface of the CAD model of a single object and generating candidate grasping poses; then, using the physical engine to perform force closure analysis on each candidate pose to calculate its grasping stability score; finally, projecting the stable grasping poses of each object to the scene coordinate system and filtering out invalid grasps that interfere with other objects in the scene through collision detection. In each scene, about 10 objects are randomly selected from the entire object set and placed randomly. The dataset is divided into a training set (scenes 1-100), a “seen” object validation set (scenes 101-130), a “familiar” object validation set (scenes 131-160), and a “unseen” object test set (scenes 161-190). Each scene consists of 256 pictures taken by two cameras attached to the robot arm, covering 256 different viewpoints on a quarter sphere. To overcome the interference caused by incomplete point clouds and improve the detection ability of new objects, the invention reconstructs the depth images of multiple perspectives in a single scene into truncated signed distance functions for training and evaluation. The truncation distance of TSDF is set to 0.04m, and the voxel resolution is set to 5mm×5mm×5mm.

[0091] 2. Construct a multi-modal large model three-dimensional perception overall framework.

[0092] As shown in Figure 1 The multi-modal large model three-dimensional perception method proposed by the invention aims to solve the problem that large models cannot directly understand the three-dimensional spatial information contained in point cloud data, especially the positional relationship between overlapping or occluded objects in a cluttered environment. The framework jointly models vision, language, and action, and the overall process is as follows:

[0093] First, the multi-modal large model receives language instructions (such as: “give me an apple”) and original images containing object clutter. And scene point cloud data.

[0094] For the image and language processing path, the large model first understands the human intention behind the language instruction and encodes it into language features. The multi-modal large model adopts the DeepseekVLV2ForCausalLM architecture, and the language encoder adopts the Deepseek-VL2-tiny built-in language model, with an output language feature dimension of 512 dimensions. For the target object specified in the human intention, the OpenCV connected component analysis algorithm is used to extract the bounding box from the object mask image, and the target object is obtained by cropping the original image with the corresponding bounding box.

[0095] Subsequently, these cropped target objects are input into the image encoder of the multi-modal large model (output feature dimension of 512), to generate object-centered bounding box visual features, which serve as the key vector for subsequent joint training.

[0096] At the same time, the language instruction is input into the text encoder of the multi-modal large model, and the obtained language features are fused with the visual features through element-wise multiplication to form the value vector for joint training.

[0097] Next, for the point cloud data processing path, the point cloud is fed into a fully attention point cloud encoder that provides three-dimensional spatial perception, obtaining features with three-dimensional spatial perception capabilities. Then, through a Riemannian manifold multi-scale perception domain regression module based on homotopy mapping, the conformality of the manifold space is used to constrain the multi-scale contour, and finally a hybrid physical metric method guided by physical prior knowledge is used for screening to ensure that the generated motion poses meet a series of physical constraints. In summary, this path takes scene point cloud as input, predicts a set of six-degree-of-freedom motion poses, and encodes them into spatial features as query vectors for the next stage of joint training.

[0098] Then, the motion pose features generated by the point cloud perception network are taken as the query, and the visual language features centered on the target object encoded by the multi-modal large model are taken as the value.

[0099] To further combine the spatial features with the visual language features, the bounding box visual position features are used as the key vector. Specifically, the three-dimensional position of the bounding box center is projected into a nonlinear space and encoded through a multi-layer perception machine, and then this position embedding is fused with the visual features of the bounding box as the key vector of the cross-attention layer. After processing through the cross-attention layer, the cross-attention features that fuse visual, language, and motion pose are obtained.

[0100] Finally, the cross-attention feature after joint training is input into the cross-attention reinforcement learning network. The network is composed of a policy network and a critic network, and is trained in a simulated environment built by the PyBullet robot simulation engine through model-free deep reinforcement learning based on the discrete SAC algorithm.

[0101] During the training process, the cross-attention feature at the t-th step is taken as the state , and the action pose selection at the t-th step is taken as the decision . The policy is defined as , and the critic outputs the Q function . Both the policy and the critic linear layers take the cross-attention feature as input and output the logarithm of the action pose and the value. In this way, the cross-attention reinforcement learning network can predict a series of actions to remove obstacles and ultimately complete the grasping of the specified target, thereby achieving the six-degree-of-freedom grasping task.

[0102] 3. Design a Riemannian manifold multi-size perception domain regression module based on homotopy mapping.

[0103] As shown in Figure 2 , the Riemannian manifold multi-size perception domain regression module based on homotopy mapping designed by the application aims to solve the shortcomings of multi-modal large models in processing the geometric features contained in the object contour, especially for objects with large size differences and unbalanced point cloud embedding. Considering that the contour and size distribution of objects are very different, but the geometric topology behind them is limited, the application proposes to learn high-dimensional deep geometric topological information to realize the understanding of general three-dimensional objects, and the topological structure of the object can be learned from the continuous deformation process of the multi-scale scaling of the object contour.

[0104] To achieve this goal, the present application proposes a multi-scale perception domain grouping module with non-Euclidean manifold embedding, which can benefit from the learning of object contour explicitly and adapt to a wide range of data characteristics. Specifically, first, a plurality of cylinders with different radii and cutting related points are set as perception domains, and the radii are uniformly distributed within the maximum width range of the robot hand. This is done to solve the problem of detail ambiguity or missed grasp points caused by fixed range perception domains, and to obtain multi-scale perception domain shapes. In this embodiment, a UR5 robot hand is used, with a maximum width of 0.1m, and the cylinder radii are set to 4 levels: 0.02m, 0.03m, 0.04m, and 0.05m. The cylinder height is set to 4 levels: 0.01m, 0.02m, 0.03m, and 0.04m. The height of each cylinder is consistent with the height of the three-dimensional bounding box of the corresponding object, with an error of ≤10mm. To ensure that the extracted features remain similar to the contour and are not sensitive to noise during the continuous deformation of the perception domain with different radii, a homotopy layer is introduced to realize homotopy mapping. The homotopy layer is a core component for realizing topologically continuous mapping, composed of N cascaded linear layers and nonlinear activation functions, used to map the target from the geometric features of the perception domain to the homotopy space, and then to the Riemannian manifold, thereby realizing smooth mapping between Euclidean space and Riemannian manifold. The homotopy function is:

[0105] (1)

[0106] wherein, represents the homotopy mapping function, describing the continuous transformation process from mapping to mapping ; represents the point input into the mapping function, usually representing the geometric features of the perception domain; is a continuously changing parameter, when its norm changes from 0 to 1, the mapping changes continuously from to ; , is a continuous mapping between two topological spaces. As a topologically continuous transformation, homotopy mapping ensures the consistency of the topological structure during the mapping process, so that different contours form continuous curve elements in the Riemannian manifold, thereby improving the effectiveness of contour similarity calculation.

[0107] Subsequently, the cylinder feature vectors with different radii r and different heights h after homotopy mapping are projected into the Riemannian manifold space, and the similarity loss of the perception domain features is calculated in the Riemannian manifold. The Lorenz model is used as the Riemannian manifold, and a dimensional Lorenz model with curvature is defined , where It is a set of points that satisfies the following definition , diagonal matrix Unlike vector addition and multiplication in Euclidean space, the scalar product in the Lorentz model is defined as... , among them Let represent two vectors in the Lorentz model. and scalar product. and They are vectors and The zeroth component (time dimension component); and They are vectors and The component in the i-th spatial dimension. It is a manifold in (N+1)-dimensional Minkowski space, with its origin at point O. . In the Lorentz model The point defined above. For coordinate points... Coordinates can be used on an (N+1)-dimensional Euclidean subspace (tangent space). To approximate The N-dimensional Lorentz model vector space at coordinate points is defined as follows: ,in, Represents a point on a manifold Tangent space at the point; It is a vector in the tangent space, and is a dimensional vector; Represents the tangent vector Points on the manifold The Lorentz inner product, which is zero is lie in Constraints on the tangent space. Next, points on the Euclidean subspace and manifold space are connected using exponential and logarithmic mappings. The exponential mapping allows any tangent vector in the Euclidean subspace to be mapped onto the manifold space; the logarithmic mapping maps back to the tangent space. These two mappings can be written as:

[0108] (2)

[0109] In the formula, Indicates from point Starting from the tangent vector The exponential mapping of directions results in the tangent vector being... Mapped to a point on the manifold; cosh and sinh are hyperbolic cosine and hyperbolic sine functions, respectively; denotes negative curvature Take square root, make sure the inside of the root is positive; denotes tangent vector Lorentz norm of

[0110] (3)

[0111] where denotes the logarithmic map from point to point , the result is a tangent vector representing point in the tangent space at point; denotes the geodesic distance between two points and on the Lorentz model with curvature ; is the curvature of the manifold; is the Lorentz inner product between points and ; denotes the Lorentz norm. denotes the distance on the Lorentz model, which is calculated as:

[0112] (4)

[0113] where denotes the geodesic distance between two points and on the Lorentz model with curvature ; is the inverse hyperbolic cosine function; is the curvature of the manifold; is the Lorentz inner product between points and .

[0114] Finally, in multi-size perception domain regression, the similarity loss of three-dimensional contours of the same object on multi-scale Riemannian manifolds is calculated:

[0115] (5)

[0116] where denotes the multi-scale contour similarity loss calculated on the Riemannian manifold; denotes the number of different size perception domains (cylinders); denotes the Riemannian distance between two points calculated on the Lorentz model with curvature c; The vector represents the feature mapping of the i-th and j-th different size cylindrical perception domain to the Riemannian manifold space. In this way, the model can learn the self-adjusting distribution of the geometric topological structure of the object on the manifold, so that when encountering a completely new shape of the object, it can analyze its contour features from the learned topological structure, construct high-precision motion generation candidate nodes, and provide high-precision expression for specific actions downstream.

[0117] In addition, in the perception domain self-attention component, a corresponding seed feature is also introduced as a guide for the fusion of different scale local features, and a point cloud three-dimensional weight attention-based gate filtering mechanism is used to filter the information in the seed feature. The specific filtering rule is: by calculating the importance weight of each neuron relative to other neurons in the channel, an attention weight matrix is constructed, and then a gate signal is generated through a sigmoid activation function to adaptively weight and filter the information in the seed feature, retain important geometric features, and suppress noise and redundant information.

[0118] 4. Construct a point cloud encoder based on a three-dimensional attention mechanism.

[0119] Referring to Figure 3 , Figure 4 and Figure 5 , the application constructs a point cloud encoder based on a three-dimensional attention mechanism, which is used to convert 3D point cloud data into a feature vector and provide three-dimensional spatial perception capability for a multi-modal large model.

[0120] As shown in Figure 3 , the point cloud 3D attention filtering module designed by the application combines a depth separable convolution to strengthen local attention, and uses a 3D point cloud attention to strengthen local and global attention, and realizes layer-by-layer modeling through a gate filtering mechanism based on point cloud three-dimensional weight attention.

[0121] First, a three-dimensional attention module for point clouds is proposed, which can assign a unique weight to each neuron. For the input local receptive field feature vector

[0122] Next, the importance of the t-th neuron relative to other neurons is calculated by an energy function :

[0123] (6)

[0124] In the formula, represents the importance measure of the t-th neuron relative to other neurons; respectively represent the weight and bias for linear transformation of the t-th neuron; represents the feature vector of the t-th target neuron; The feature vectors of other neurons in the same channel besides the target neuron; This indicates the total number of neurons within that channel; This represents the weight regularization coefficient. And it is achieved by minimizing the energy function. , making it about and The partial derivatives are zero, and the weights are derived analytically. and deviation The optimal closed-form solution is:

[0125] (7)

[0126] (8)

[0127] in, and These are the mean and variance of other neurons within the channel. Based on the assumption that all neurons follow the same distribution, the minimum energy can be obtained.

[0128] (9)

[0129] in and These are the mean and variance of all neurons. The importance of a neuron is... This forms the weight matrix E. Finally, the feature vector X is multiplied by the 3D attention weights to obtain the processed feature vector:

[0130] (10)

[0131] In the formula, This represents the output feature vector after 3D attention weight processing; Indicates the activation function; Indicates the importance of all neurons The weight matrix formed; This represents the input feature vector.

[0132] Furthermore, the gate filtering mechanism is incorporated into the point cloud Transformer to achieve complete self-attention, such as... Figure 4 and Figure 5 As shown, Figure 4 In the diagram, (a) is the original point cloud transformer layer, and (b) is the fully attention transformer layer after applying a gating mechanism that uses 3D weighted attention of the point cloud. Figure 5Point cloud encoder using full attention transformer layers that combine to filter the most distinctive channels, promote more combined representations, and enable channel reweighting, thereby enhancing the model's ability to transfer knowledge to similar scenarios.

[0133] 5. Design a hybrid physical metric method guided by physical prior knowledge.

[0134] Referring to Figure 6 , a hybrid physical metric method guided by physical prior knowledge is designed to accurately evaluate and optimize the quality of predicted motion poses. By fusing multiple physical meaningful metrics, the method overcomes the label noise problem caused by a single evaluation scale and enhances the model's perception of complex three-dimensional physical relationships. In this embodiment, when grabbing a bottle, the surface with the largest contact area and the largest contact surface is selected for grabbing under the condition of satisfying the force closure and collision constraint.

[0135] First, the force closure metric loss is introduced by calculating the relationship between the vector connecting the two robot contact points and the normal vector of each contact point to evaluate the stability of the grabbing action and ensure that the gripper contact complies with the mirror rule constraint. The calculation formula is:

[0136] (11)

[0137] In the formula, represents the force closure metric loss based on the mirror rule; represents the position coordinates of the two contact points in three-dimensional space, respectively; represents the vector from point to point ; , represent the unit outer normal vectors at contact points and , respectively; represents the L2 norm of the vector.

[0138] Next, the flatness metric loss is introduced by calculating the cosine distance between the adjacent normal vectors of the contact points to measure the curvature near the contact points and reflect the flatness of the contact surface. The calculation formula is:

[0139] (12)

[0140] In the formula, represents the flatness metric loss of the curvature near the contact point; denotes the number of neighboring points of each contact point; denotes the unit normal vector of the i-th query contact point; denotes the unit normal vector of the j-th neighboring point of the i-th contact point; denotes the dot product operation of vectors.

[0141] Subsequently, a contact area metric loss is introduced to approximate the contact area by counting the number of sampled points within the contact region, which reflects the influence of the size of the contact area on stability, and the calculation formula is:

[0142] (13)

[0143] wherein, denotes the metric loss of the contact area; denotes the number of point cloud sampling points within the contact region, which is used to approximate the contact area.

[0144] Then, a collision constraint indicator is introduced, which is used to avoid the predicted robot contact point being too close to the object surface, and ensure that the contact point is close enough to the object surface and does not collide, and the calculation formula is:

[0145] (14)

[0146] wherein, denotes the constraint indicator for avoiding collision between the robot and the object surface; denotes the minimum value of the distance between the two contact points and the object surface; denotes the lower limit of the constraint distance; the upper limit of the constraint distance; denotes the hyperbolic tangent activation function.

[0147] Finally, the action detection network predicts the action score. According to the i-th point of the grasping action score , the overall physical constraint loss of the i-th point is calculated:

[0148] (15)

[0149] wherein, denotes the overall physical constraint loss of the i-th point; denotes the grasping action prediction score of the i-th point; denotes the weight coefficient of the flatness, contact area, and collision constraint loss, respectively; denotes the total number of points.

[0150] Thus, the complete Riemannian manifold prior guided point cloud network training loss is obtained , which is calculated as the sum of the baseline loss , the overall physical constraint loss R and the multi-scale contour similarity constraint loss on the Riemannian manifold

[0151] (16)

[0152] In the formula, , the total training loss of the point cloud network is represented; , the grasp detection loss of the baseline method is represented; , the overall physical constraint loss is represented and calculated by formula (9); , the Riemannian manifold multi-scale contour similarity constraint loss is represented, and the weighting coefficient is represented.

[0153] 6. The model is jointly trained using a cross-attention reinforcement learning algorithm.

[0154] The point cloud perception network and the multi-modal large model are jointly trained using a cross-attention reinforcement learning algorithm, aiming to fuse multi-modal information and realize high-precision motion pose generation.

[0155] Firstly, after the point cloud data is encoded by a complete attention Transformer established based on a point cloud three-dimensional weight attention gate filtering mechanism, the spatial distribution features of the object are established. This feature is used for subsequent action candidate generation.

[0156] Then, the view loss is calculated from the spherical space of the object, and the most suitable observation angle is selected. At each viewpoint, a set of action candidates is generated, including grid sampling of the gripper depth and in-plane rotation angle.

[0157] In this process, the Riemannian manifold multi-scale perception domain regression module based on homotopy mapping excavates the invariance of multi-scale contours through the conformal property of Riemannian manifold, so that the model can select appropriate radius and depth to adapt to objects of different sizes. Finally, the six-degree-of-freedom pose coordinates of the grasping action are output through a multi-layer perception machine.

[0158] Subsequently, cross-modal fusion is performed in the cross-attention layer. The action pose coordinates transmitted by the point cloud perception network are taken as the query, and the visual language features centered on the target object and having a bounding box encoded by the multi-modal large model are taken as the value. To assist the alignment of three-dimensional points and visual language features, the visual position features with the target object bounding box are taken as the key of cross-attention.

[0159] ​Specifically, the 3D position of the bounding box center is projected into a nonlinear space, and the position embedding of the bounding box is encoded by a multilayer perceptron. This position embedding is then fused with the visual features of the bounding box. After cross-attention processing involving vision, language, and action pose, the fused cross-attention features are obtained.

[0160] Finally, the obtained cross-attention features are used as input and fed into the policy network and critic network for reinforcement learning in the discrete SAC reinforcement learning algorithm. The cross-attention features of the steps as states The first output after passing through the policy network Step-by-step strategy selection as decision The strategy of reinforcement learning is defined as... The critic network outputs the Q function Q. The SAC algorithm maximizes the objective through soft policy iteration, alternating between policy evaluation and policy improvement within a maximum entropy framework. The soft-state value function is defined as:

[0161] (17)

[0162] In the formula, Representation strategy The parameters are The soft-state value function loss at that time; Indicates the state Take action below The strategy probability distribution; Indicates the state Take action below Q-value function; This represents the temperature parameter, used to balance the exploration and utilization of control strategies.

[0163] The soft Q function is trained by minimizing the soft Bellman residual:

[0164] (18)

[0165] In the formula, This represents the training loss of the Q function; This represents the parameters of the Q-function network; Indicates the state Take action below When, the parameter is The value output by the Q function; This represents the target Q value calculated based on the Bellman equation; This represents the environmental reward obtained at time step t; and These represent the state and action at the next time step, respectively.

[0166] Among them, the target Defined by the Bellman equation:

[0167] (19)

[0168] In the formula, This represents the target Q value calculated based on the Bellman equation; This represents the environmental reward obtained at time step t; Indicates the state Take action below The strategy probability distribution; Indicates the next state and actions The soft-state value function.

[0169] Among them, the soft state value function Defined as:

[0170] (20)

[0171] In the formula, Represents the soft-state value function. Indicates the next state and actions When, the parameter is The value output by the Q function; This represents a temperature parameter used to balance the control strategy between exploration and utilization. Represents the natural logarithm function; Indicates the next state Take action below The strategy probability distribution.

[0172] The policy update is guided by the soft Q function calculated in the policy evaluation step. This is achieved by multiplying the expected KL divergence by the temperature parameter. Ignoring the partitioning function, the policy parameters are learned to obtain the final loss function. :

[0173] (twenty one)

[0174] In the formula, Representing temperature parameters The loss function; represents the target entropy, used to control the degree of randomness of the policy, the algorithm encourages the agent to strike a balance between exploration and exploitation, thereby improving the stability and performance of the policy, and the spatial perception information such as rotation in three-dimensional space, proximity, collision constraint, overlap, occlusion, etc. is fused into the visual language features to realize high-precision generation of downstream action three-dimensional coordinates and poses.

[0175] 7. Perform three-dimensional scene perception and grasping action generation using the trained model.

[0176] A multi-modal large model trained in combination is used to perform three-dimensional scene perception and grasping action generation in combination with a point cloud perception network. In the inference stage, the model first receives down-sampled point cloud data, and based on the guidance of multi-modal information fusion and physical prior knowledge, synthesizes and outputs a set of initial grasping pose parameters. Then, in order to obtain the optimal grasping candidate, non-maximum suppression is performed on these grasping pose parameters, and collision checking is performed on the filtered candidate poses, while evaluating whether they satisfy the force closure condition under the current friction coefficient to ensure physical feasibility. Finally, through this series of screening and evaluation, the six-degree-of-freedom action pose that can guide the robot to accurately grasp the target object is inferred, and a grasping action sequence that can be executed in a real or simulated environment is generated.

[0177] Table 1 is a comparison of the present application with the state-of-the-art method for the RealSense camera captured scene of the GraspNet-billion benchmark. The evaluation indicator AP refers to the average precision, and the values behind it are the friction coefficients, which are set to 0.2, 0.4, 0.6, 0.8, 1.0, and 1.2, respectively. Therefore, AP refers to the total average precision, and AP0.8 and AP0.4 refer to the average precision with a friction coefficient of 0.8 and a friction coefficient of 0.4. In terms of network details, the upper and lower limits of the contact distance are set to 0.02 and 0.005, respectively. The learning rate for training is 0.001, and the batch size is 2. The proposed method is implemented using the Pytorch deep learning framework, version 1.7.1+cu110, with CUDA 11.0 and Python 3.7.9. The experiment is conducted on an Ubuntu 20.04.2 LTS operating system, and the main hardware includes an NVIDIA GeForce RTX 3090 GPU, an Intel(R) Core(TM) i9-10900K processor, and 128GB of memory.

[0178] Table 1

[0179]

[0180] In Table 1, the other methods are as follows:

[0181] GPD corresponds to the method proposed by Ten et al. (Ten Pas A, Gualtieri M, Saenko K, et al. Grasp pose detection in point clouds [J]. The International Journal of Robotics Research, 2017, 36(13-14): 1455-1473.).

[0182] PointnetGPD corresponds to the method proposed by Liang et al. (Liang H, Ma X, Li S, et al. Pointnetgpd: Detecting grasp configurations from point sets [C] / / International Conference on Robotics and Automation. IEEE, 2019: 3629-3635.).

[0183] GraspNet-1Billion corresponds to the method proposed by Fang et al. (Fang H S, Wang C, Gou M, et al. Graspnet-1billion: a large-scale benchmark for general object grasping [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 11444-11453).

[0184] RGB Matters corresponds to the method proposed by Gou et al. (Gou M, Fang H S, Zhu Z, et al. Rgbmatters: Learning 7-dof grasp poses on monocular rgbd images [C] / / IEEE International Conference on Robotics and Automation. IEEE, 2021: 13459-13466.).

[0185] TransGrasp corresponds to the method proposed by Liu et al. (Liu Z, Chen Z, Xie S, et al. Transgrasp: a multi-scale hierarchical point transformer for 7-dof grasp detection [C] / / International Conference on Robotics and Automation. IEEE, 2022: 1533-1539.).

[0186] HGGD corresponds to the method proposed by Chen et al. (Chen S, Tang W, Xie P, et al. Efficient heatmap-guided 6-dof grasp detection in cluttered scenes [J]. IEEE Robotics and Automation Letters, 2023.).

[0187] Scale-balanced Grasp corresponds to the method proposed by Ma et al. (Ma H, Huang D. Towards scale balanced 6-dof grasp detection in cluttered scenes [C] / / Conference on Robot Learning. PMLR, 2023: 2004-2013.).

[0188] GSNet corresponds to the method proposed by Wang et al. (Wang C, Fang H S, Gou M, et al. Graspness discovery in clutters for fast and accurate grasp detection [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. IEEE, 2021: 15964-15973.).

[0189] Generalizing-Grasp corresponds to the method proposed by Ma et al. (Ma H, Shi M, Gao B, et al. Generalizing 6-dof grasp detection via domain prior knowledge [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2024: 18102-18111.).

[0190] The present application provides a multi-modal large model three-dimensional perception method based on Riemannian manifold prior guidance, which solves the problems of insufficient understanding of point cloud three-dimensional space information by existing large models and performance limitations in cluttered scenes and different size object grasping. The method effectively captures the deep geometric topological information of the object contour by introducing a Riemannian manifold multi-size perception domain regression module based on homotopy mapping; by constructing a point cloud encoder based on a three-dimensional attention mechanism, the accuracy of point cloud feature extraction and the global-local semantic connection are enhanced; and a hybrid physical metric method is designed to fuse force closure, flatness, contact area and collision constraints to provide physical prior guidance for grasp pose evaluation; finally, a cross-attention reinforcement learning algorithm is used to jointly train visual, language and action poses to realize deep fusion of multi-modal information, so as to accurately generate a six-degree-of-freedom action pose to guide a robotic arm to perform a target grasping task. The experimental results on the GraspNet-1Billion dataset and the simulation environment show that the method can significantly improve the generalization ability and grasping success rate of the model in complex three-dimensional scenes, verifying its effectiveness and usability in embodied perception.

[0191] The above embodiments are only preferred embodiments of the present application and cannot be considered as limiting the scope of the present application. Any equivalent changes and improvements made within the scope of the present application should still belong to the scope of the present application.

Claims

1. A multimodal large-model 3D perception method based on Riemannian manifold prior guidance, characterized in that... Includes the following steps: 1) Prepare a multimodal 3D grasping and sensing dataset; 2) Construct a multimodal large-scale model 3D perception overall framework; 3) Design a multi-size receptive domain regression module for Riemannian manifolds based on homotopy mapping; 4) Construct a point cloud encoder based on a 3D attention mechanism; 5) Design a hybrid physical measurement method guided by prior physical knowledge; 6) The model is jointly trained using a cross-attention reinforcement learning algorithm; 7) Use the trained model to perform 3D scene perception and grasping action generation.

2. The multimodal large-model 3D perception method based on Riemannian manifold prior guidance as described in claim 1, characterized in that... In step 1), the specific method for preparing the multimodal 3D grasping perception dataset is as follows: the publicly available large-scale six-degree-of-freedom grasping dataset GraspNet-1 Billion is used for model training and evaluation; this dataset contains a total of 97,280 RGB-D images and more than 1 billion six-degree-of-freedom grasping poses. The RGB-D images are taken from different perspectives of 190 cluttered scenes, with each scene containing 512 RGB-D images; the dataset provides 3D mesh models for 88 types of objects and densely annotates the six-degree-of-freedom poses and grasping postures of the objects in each scene; 32 objects are selected from the YCB dataset, 13 adversarial objects are selected from the Dex-Net 2.0 method, and another 43 objects are collected independently; each object has an accurate computer-aided design model, which is automatically annotated through force analysis; in each scene, approximately 10 objects are randomly selected from the entire object set and randomly stacked; The dataset is divided into a training set, a validation set of "seen" objects, a validation set of "familiar" objects, and a test set of "unseen" objects. Each scene consists of 256 images taken by two cameras attached to a robotic arm, covering 256 different viewpoints on a quarter sphere. To overcome the interference caused by the incompleteness of the point cloud and improve the detection capability of new objects, the depth images of multiple views in a single scene are reconstructed into a truncated signed distance function for training and evaluation.

3. The multimodal large-model 3D perception method based on Riemannian manifold prior guidance as described in claim 1, characterized in that... In step 2), the specific method for constructing the overall 3D perception framework of the multimodal large model is as follows: The multimodal large model receives language commands and raw images containing object clutter; The multimodal large model encodes language instructions into language features and extracts bounding boxes of target objects from the original image; The cropped target object is input into the image encoder of the multimodal large model to generate object-centered bounding box visual features, and the linguistic features are fused with the bounding box visual features to form a jointly trained value vector; A point cloud perception network guided by Riemannian manifold priors is introduced. This network takes scene point clouds as input, predicts a set of six-DOF action poses, and encodes them as spatial features, which serve as query vectors for the next stage of joint training. The motion pose features generated by the point cloud perception network are used as queries, and the bounding box visual language features centered on the target object encoded by the multimodal large model are used as values. To further combine spatial features with visual language features, the visual position features of the bounding box are used as key vectors. The three-dimensional position of the center of the bounding box is projected into a nonlinear space and encoded by a multilayer perceptron. Then, the position embedding is fused with the visual features of the bounding box as the key vector of the cross-attention layer. The cross-attention features, which are a fusion of visual, linguistic, and action pose information, obtained after processing by the cross-attention layer, are input into the cross-attention reinforcement learning network. This network is trained by model-free deep reinforcement learning to predict a series of actions to remove obstructing objects and complete the grasping of the specified target.

4. The multimodal large-model 3D perception method based on Riemannian manifold prior guidance as described in claim 1, characterized in that... In step 3), the specific method for designing the Riemannian manifold multi-size receptive domain regression module based on homotopy mapping is as follows: Multiple cylinders with different radii and cutting-related points are set as the sensing field, with the radii evenly distributed within the maximum width range of the robot arm; By introducing homotopy mapping, the target is mapped from the geometric features of the receptive field to the homotopy space, and then to the Riemannian manifold, achieving a smooth mapping between Euclidean space and the Riemannian manifold. The homotopy function is constructed as follows: In the formula, Denotes the homotopy mapping function, which describes the mapping from... to mapping The continuous transformation process; The points that are input into the mapping function typically represent the geometric features of the receptive field. It is a continuously changing parameter, when its norm When the mapping changes from 0 to 1, it changes from... Continuous change ; , It is a continuous mapping between two topological spaces; The eigenvectors of cylinders with different radii and heights after homotopy mapping are projected onto the Riemannian manifold space, and the Lorentz model is used as the Riemannian manifold. Similarity loss is calculated on multi-scale Riemannian manifolds based on the 3D contours of the same object. The calculation method is as follows: In the formula, This represents the multi-scale contour similarity loss computed on the Riemannian manifold; Indicates the number of receptive fields of different sizes; This represents the Riemann distance between two points calculated on a Lorentz model with curvature c; Let represent the vectors after mapping the receptive field features of the i-th and j-th cylinders of different sizes to the Riemannian manifold space; The module also introduces corresponding seed features as guidance for the fusion of local features at different scales, and uses a gating mechanism based on point cloud 3D weighted attention to filter the information in the seed features.

5. The multimodal large-model 3D perception method based on Riemannian manifold prior guidance as described in claim 1, characterized in that... In step 4), the specific method for constructing a point cloud encoder based on a 3D attention mechanism is as follows: We propose a 3D attention module for point clouds, which assigns unique weights to each neuron and focuses on the local receptive field feature vector of the input. Through energy function Calculate the first The importance of each neuron: In the formula, This represents a measure of the importance of the t-th neuron relative to the other neurons; Let represent the weights and biases for the linear transformation of the t-th neuron, respectively; This represents the feature vector of the t-th target neuron; The feature vectors of neurons other than the target neuron within the same channel; This indicates the total number of neurons within that channel; This represents the weight regularization coefficient; The weights are solved by derivation. and deviation Finally, the importance of neurons is calculated. Constructing the weight matrix Multiplying the processed feature vector by the feature vector X yields the processed feature vector: In the formula, This represents the output feature vector after 3D attention weight processing; Indicates the activation function; Indicates the importance of all neurons The weight matrix formed; This represents the input feature vector.

6. The multimodal large-model 3D perception method based on Riemannian manifold prior guidance as described in claim 1, characterized in that... In step 5), the specific method for designing a hybrid physical metric guided by prior physical knowledge is as follows: Introducing force closure measurement loss: In the formula, This represents the force closure metric loss based on the mapping rule; These represent the position coordinates of the two contact points in three-dimensional space; Indicates from point Point of view ; , They represent the contact points respectively. and The unit outward normal vector at that location; The L2 norm of a vector; Introduce a flatness metric loss: In the formula, The flatness measure loss represents the curvature near the contact point; This indicates the number of neighboring points for each contact point; Represents the unit normal vector of the i-th query contact point; Represents the unit normal vector of the j-th neighboring point of the i-th contact point; Represents the dot product operation of vectors; Introducing contact area measurement loss: In the formula, A measure of the loss of contact area; This indicates the number of point cloud sampling points within the contact area, used to approximate the contact area; Introducing collision constraint indices: In the formula, These are the constraint parameters used to prevent the robotic arm from colliding with the surface of an object. This represents the minimum distance between the two contact points and the object surface. Indicates the lower limit of the constraint distance; Limit the upper limit of the constraint distance; Represents the hyperbolic tangent activation function; The action scores predicted by the action detection network are weighted and regularized to calculate the... Overall physical constraint loss at each point: In the formula, This represents the overall physical constraint loss at the i-th point; This represents the prediction score for the grabbing action at the i-th point; These represent the weighting coefficients for flatness, contact area, and collision constraint loss, respectively. Indicates the total number of points; In summary, the training loss of point cloud networks Basic loss Overall physical constraint loss R and multi-scale contour similarity constraint loss on Riemannian manifold sum: In the formula, This represents the total training loss of the point cloud network; This represents the capture detection loss of the baseline method; The overall physical constraint loss is represented by formula (9); Represents the multi-scale contour similarity constraint loss of Riemannian manifolds. Weighting coefficients.

7. The multimodal large-model 3D perception method based on Riemannian manifold prior guidance as described in claim 1, characterized in that... In step 6), the specific method for jointly training the model using the cross-attention reinforcement learning algorithm is as follows: The point cloud data processed by the point cloud encoder based on the 3D attention mechanism is used to generate action candidates, and the appropriate receptive domain parameters are selected by the Riemannian manifold multi-size receptive domain regression module based on homotopy mapping. The six-degree-of-freedom pose coordinates of the grasping action are output through a multilayer perceptron. In the cross-attention layer, the action pose coordinates are used as the query vector, the visual language features of the target object encoded by the multimodal large model are used as the value vector, and the bounding box visual position features are used as the key vector to achieve the fusion of multimodal information and obtain cross-attention features. Cross-attention features are input into the policy network and the critic network, and reinforcement learning training is performed using the discrete SAC algorithm. The policy network is defined as follows: The critics' network outputs the Q function. ; In the strategy evaluation step, the soft state value function is defined as: In the formula, Representation Strategy The parameters are The soft-state value function loss at that time; Indicates the state Take action below The strategy probability distribution; Indicates the state Take action below Q-value function; This represents a temperature parameter used to balance the control strategy between exploration and utilization. The soft Q-function is trained by minimizing the soft Bellman residual, and its calculation formula is as follows: In the formula, This represents the training loss of the Q function; This represents the parameters of the Q-function network; Indicates the state Take action below When, the parameter is The value output by the Q function; This represents the target Q value calculated based on the Bellman equation; This represents the environmental reward obtained at time step t; and These represent the state and action at the next time step, respectively. In the formula, This represents the target Q value calculated based on the Bellman equation; This represents the environmental reward obtained at time step t; Indicates the state Take action below The strategy probability distribution; Indicates the next state and actions The soft-state value function under the given conditions; In the formula, Represents the soft-state value function. Indicates the next state and actions When, the parameter is The value output by the Q function; This represents a temperature parameter used to balance the control strategy between exploration and utilization. Represents the natural logarithm function; Indicates the next state Take action below The strategy probability distribution; Policy updates are guided by the soft Q-function calculated in the policy evaluation step; this is achieved by multiplying the expected KL divergence by the temperature parameter. Ignoring the partitioning function, the policy parameters are learned to obtain the final loss function. : In the formula, Representing temperature parameters The loss function; It represents the target entropy, used to control the degree of randomness in the strategy.

8. The multimodal large-model 3D perception method based on Riemannian manifold prior guidance as described in claim 1, characterized in that... In step 7), the specific method for generating 3D scene perception and grasping actions using the trained model is as follows: A multimodal large model trained in conjunction with a point cloud perception network is used to generate perception and grasping actions in a 3D scene. In the inference phase, the model first receives downsampled point cloud data and, guided by multimodal information fusion and prior physical knowledge, synthesizes and outputs a set of initial grasping pose parameters. Then, in order to obtain the optimal grasping candidates, nonmaximum suppression is performed on these grasping pose parameters, and collision checks are performed on the selected candidate poses. At the same time, it is evaluated whether they meet the force closure condition under the current friction coefficient to ensure physical feasibility. Finally, through this series of screening and evaluation, a six-degree-of-freedom motion pose that can guide the robotic arm to accurately grasp the target object is obtained, and a grasping action sequence that is inferred to be executed in a real or simulation environment is generated.

Images