Multi-modal large model scene perception method based on point cloud multiple manifolds

Abstract

The invention discloses a multi-modal large model scene perception method based on point cloud multiple manifolds, and relates to robot vision and 3D scene understanding. Three kinds of heterogeneous input of RGB images, point cloud and natural language instructions are received, and scene-level multi-mode alignment, object-relation joint inference and the like are completed at a time through a parallel semantic-geometry double-branch architecture. The semantic branch performs cross-modal matching on a text instruction and an image region by using a large vision-language pre-training model to generate a language-vision consistent three-dimensional semantic attention body space; a geometric branch design shares an attention point cloud Transform, and a point cloud feature body with global-local collaboration is output. And multi-manifold consistency feature learning is proposed: multi-scale point neighborhood features are respectively embedded into a hyperbolic manifold and a spherical manifold, a semantic body space and a geometric feature body are fused, uniform multi-modal scene representation is output, and multiple downstream tasks such as support grabbing and the like are supported. And the method has high-precision and high-robustness multi-modal scene understanding capability in a complex and disordered environment.

Description

Technical Field

[0001] This invention belongs to the field of robot vision and multimodal scene understanding technology, and specifically relates to a multimodal large model scene perception method based on point cloud multimanifolds. Background Technology

[0002] Embodied intelligence requires agents to complete a closed loop of "understanding-decision-execution" in complex physical environments in real time. The core of the "understanding" stage is the multimodal, fine-grained, and geometrically consistent perception of the 3D scene. However, existing scene understanding methods for grasping still follow a "cascaded" paradigm: first, instance detection and segmentation are performed in a high-dimensional RGB space, then the results are mapped to a 3D point cloud for pose estimation, and finally, grasping planning is initiated. This serial process leads to amplified front-end visual errors at each stage, and the scene's geometric topology information is diluted in multiple projections, ultimately resulting in inaccurate semantic-geometric consistency judgments of unknown or occluded objects by the robot. On the other hand, mainstream methods uniformly embed multimodal features into Euclidean space, ignoring the nonlinear hierarchical relationships of real objects in the multi-level structure of "category-part-point," as well as the Riemannian properties in attributes such as direction, angle, and curvature, making it difficult to explicitly model complex geometric topologies. Furthermore, existing sampling strategies generally lack scene-adaptive semantic-geometric joint guidance, resulting in candidate regions that neither conform to the semantic focus of human language instructions nor possess graspable geometric advantages. Therefore, there is an urgent need for a new multimodal scene perception framework for embodied intelligence that can perform semantic-geometric joint reasoning in parallel within a unified network and make full use of non-Euclidean manifold priors to explicitly model the hierarchical structure and directional topology, thereby providing high-precision and robust 3D scene representation for subsequent downstream tasks such as grasping, navigation, and interaction. Summary of the Invention

[0003] The purpose of this invention is to address the problems faced by existing robot scene understanding methods in complex and chaotic scenes, such as cascading architecture error accumulation, insufficient cross-layer semantic correlation, low reliability of motion logic generation, and bias in understanding geometric and topological information. This invention provides a multimodal large-scale model scene perception method based on point cloud multi-manifolds. By introducing non-Euclidean geometry (Riemannian manifold) priors, this method jointly models semantic, visual, and geometric information within a unified framework. It aims to improve the agent's ability to perceive the hierarchical structure, directional topology, and implicit relationships of complex 3D scenes, providing high-precision and highly consistent scene representations for downstream tasks such as grasping, navigation, and human-computer interaction.

[0004] To achieve the above-mentioned objectives, the present invention provides the following technical solutions.

[0005] This invention employs a multimodal semantic-guided parallel processing framework to jointly train a visual language model and a geometric motion logic network. It also introduces geometric priors based on hyperbolic and spherical manifolds to address the problems of object size imbalance and contour topology understanding. Furthermore, by combining a point cloud encoder based on shared attention with a motion logic generation module that integrates scene perception and geometric enhancement, the invention enables the robot to understand and accurately perceive specific targets in complex and cluttered environments.

[0006] A multimodal large-scale model scene perception method based on point cloud multi-manifolds includes the following steps:

[0007] 1) Construct a multimodal 3D scene understanding dataset;

[0008] 2) Construct a parallel unified perception framework for 3D scenes with multimodal semantic-geometric collaboration;

[0009] 3) Design a point cloud encoder based on shared attention;

[0010] 4) Construct a high-dimensional feature reasoning module that integrates multi-granularity geometric topology perception and explicit structural prior enhancement for "scene-object-component";

[0011] 5) A consistency feature learning method for multi-manifold perception is proposed;

[0012] 6) The model is trained end-to-end using cross-attention and policy optimization algorithms;

[0013] 7) Utilize the trained model to generate stable logic for 3D scene perception and human intent capture.

[0014] In step 1), the specific method for constructing the multimodal 3D scene understanding dataset is as follows: the publicly available large-scale six-degree-of-freedom grasping dataset GraspNet-1 Billion is used for model training and evaluation; this dataset contains 97,280 RGB-D images, covering 190 real cluttered scenes, involving 88 everyday grasping objects; the dataset provides high-precision 3D mesh models, object instance segmentation masks, and generates dense six-degree-of-freedom grasping pose annotations through an automated process, which have undergone force closure analysis and collision detection; the dataset is divided into a training set and a test set, and the test set is further divided into three subsets: "known objects", "similar objects", and "unknown objects" to systematically evaluate the model's performance under different generalization requirements; the model's input includes scene point clouds, RGB images, and natural language instructions describing the target objects.

[0015] In step 2), the specific method for constructing a parallel 3D scene unified perception framework with multimodal semantic-geometric collaboration is as follows: the framework includes a multimodal semantic understanding branch and a geometric grasping pose generation branch that work in parallel; the multimodal semantic understanding branch takes a pre-trained visual language model as its core and receives RGB images and natural language instructions; first, it obtains the two-dimensional bounding boxes of all potential targets in the image through a target detector; then, it combines the bounding boxes with the corresponding visual features extracted by the image encoder to generate a visual representation containing target appearance information; simultaneously, the language instructions are processed by a text encoder to obtain text features; by calculating the cosine similarity between the visual features and the text features, visual-language features that quantify the degree of semantic matching between each position in the scene and the target indicated by the instruction are generated, ultimately forming a semantic attention map indicating the potential grasping area in 3D space that conforms to human intention; the geometric... The grasping pose generation branch takes scene point cloud as input and generates a set of physically feasible six-DOF grasping pose candidates that depend only on scene geometry information through a point cloud encoder and subsequent modules. Then, the outputs of the two branches are fused through a cross-attention module: the grasping pose candidates output by the geometric grasping pose generation branch are encoded as query vectors; the visual-linguistic features contained in the semantic attention map output by the multimodal semantic understanding branch are used as value vectors; the position information of the target bounding box in three-dimensional space is encoded as key vectors; the value vectors are weighted and aggregated by calculating the similarity between the query and the key, enabling each grasping pose to dynamically focus on task-related semantic regions and obtain a preliminary score accordingly; finally, the fused features are mapped by a lightweight decoder to the final execution priority probability distribution of each grasping pose candidate, and the system selects the grasping pose that best matches human intent based on this distribution.

[0016] In step 3), the specific method for designing the point cloud encoder based on shared attention is as follows: the encoder adopts a point cloud Transformer with a U-Net structure; to alleviate the semantic continuity disruption caused by independent learning of attention parameters between corresponding layers of the encoder and decoder, a shared learnable vector is introduced as an attention bias for each pair of encoder and decoder layers connected by skip connections; this shared bias is added to the initial attention score before attention score normalization, and its calculation method is as follows:

[0017]

[0018] in, Indicates the center point The output characteristics, It is a point The local neighborhood point set, , , , It is a point-level feature transformation function. It is a positional encoding. It is a normalization function. This represents element-wise multiplication. These are the shared, learnable vectors. It is The method transforms the mapping function into a bias term; it enhances cross-level feature consistency by establishing explicit geometric semantic alignment constraints between corresponding levels through parameter-level sharing.

[0019] In step 4), the specific method for constructing the high-dimensional feature inference module with multi-granularity geometric topology perception and explicit structural prior enhancement for "scene-object-part" is as follows: the module sequentially performs utility-aware point cloud sampling, explicit geometric primitive construction, and adaptive feature aggregation; firstly, it uses the features extracted from the backbone network to predict the objectivity score for each point. and capture probability score Foreground object points are filtered using objectivity scores, and the weighted distance in the farthest point sampling is redefined based on the grasp probability score:

[0020]

[0021] in It is a candidate point. It is the set of selected points. The shortest Euclidean distance is used as the seed point, and points with high grasping potential are sampled first. Then, for each seed point... In its fixed radius neighborhood set The flatness factor is calculated using principal component analysis. Linearity factor Scattering factor Local normal direction Local curvature The local point density F and the relative offset vector to the neighborhood centroid. Concatenate to form explicit geometric feature vectors Finally, the explicit geometric features are adaptively fused with the implicit features extracted by the backbone network through a dynamic gating mechanism to generate an enhanced feature representation for predicting the grasping approximation direction.

[0022] In step 5), the specific method for proposing the consistency feature learning method for multi-manifold perception is as follows: the method targets the local features extracted from the seed point under different receptive fields. Through two independent lightweight network branches, it is mapped to a hyperbolic space suitable for modeling hierarchical structures and a spherical space suitable for representing directional information, respectively; in the hyperbolic space, a Poincaré sphere model is adopted. For features projected into this space Calculate the hyperbolic space consistency loss :

[0023]

[0024] Where N is the number of multi-scale receptive fields. It is the hyperbolic geodesic distance between two points; in spherical space In the context of features projected into this space... Calculate the consistency loss of spherical space :

[0025]

[0026] in, It is the spherical geodesic distance between two points; hyperbolic geodesic distance. The calculation formula is

[0027]

[0028] Finally, features from hyperbolic and spherical spaces are stitched together and weighted, and then combined with the loss from the main grasping pose estimation task. Combined, they constitute the overall training loss. :

[0029] (6)

[0030] in, and It is a balancing weight.

[0031] In step 6), the specific method for end-to-end training of the model using the cross-attention and policy optimization algorithm is as follows:

[0032] The training process combines behavioral cloning with interactive policy optimization; in the expert imitation pre-training phase, based on offline expert demonstration data, the model prediction probability distribution is minimized. Unique heat encoding of real expert motion Training is performed using the cross-entropy loss between the two:

[0033]

[0034] In step 7), the specific method for generating stable grasping logic for 3D scene perception and human intent using the trained model is as follows: During the inference phase, the model receives scene point clouds, RGB images, and natural language instructions; the multimodal semantic understanding branch outputs a semantic attention map, and the geometric grasping pose generation branch generates a set of physically feasible six-degree-of-freedom grasping pose candidates in parallel; the cross-attention module scores the geometric candidates based on semantic information to obtain the final probability distribution. The system selects the posture with the highest score as the grasping action to achieve accurate and stable grasping of specific targets in cluttered scenes.

[0035] This invention receives three heterogeneous inputs: RGB images, point clouds, and natural language commands. It employs a parallel semantic-geometric dual-branch architecture to simultaneously perform scene-level multimodal alignment, object-relation joint inference, and fine-grained geometric topology characterization, avoiding error accumulation in traditional cascaded processes. In the semantic branch, a large-scale vision-language pre-trained model is used to perform cross-modal matching between text commands and image regions, generating a consistent 3D semantic attention volume space. In the geometric branch, a shared attention point cloud Transformer is designed, using learnable bias vectors to enhance cross-layer feature consistency at U-Net skip connections, outputting a point cloud feature volume with global-local collaboration. A multi-manifold consistency feature learning approach is proposed: multi-scale point neighborhood features are embedded into hyperbolic and spherical manifolds respectively, achieving cross-manifold feature alignment through geodesic distance consistency constraints, thereby explicitly mining non-Euclidean geometric relationships in complex scenes. Finally, the semantic volume space and geometric feature volume are fused through cross-attention, outputting a unified multimodal scene representation that can directly support various downstream tasks such as grasping, navigation, and interaction.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] This invention effectively alleviates the error accumulation problem caused by the serial processing of semantic recognition and geometric planning modules in traditional cascaded architectures by constructing a parallel unified 3D scene perception framework with multimodal semantic-geometric collaboration. It achieves synergistic optimization of semantic intent and physical feasibility, improving the overall performance and reliability of grasping based on human instructions in cluttered scenes. In particular, this invention innovatively proposes a consistency feature learning method for multi-manifold perception. By embedding multi-scale local features into hyperbolic and spherical manifolds respectively, it fully utilizes the sensitivity of hyperbolic space to hierarchical topology and spherical space to directional information, effectively overcoming the inherent limitations of Euclidean space in representing complex geometric structures. This significantly enhances the understanding of rich geometric topological information within objects, thereby improving the accuracy and generalization of predicting the grasping posture of unknown objects. Simultaneously, by designing a point cloud encoder based on shared attention, parameter-level shared constraints are introduced into the corresponding layers of the encoder and decoder in the U-Net architecture, enhancing the semantic continuity and geometric consistency of cross-level features and providing higher-quality feature representations for subsequent motion logic generation. Furthermore, the scene-aware and geometry-enhanced motion logic generation module constructed in this invention achieves collaborative optimization of grasping point selection and approximation direction prediction through a utility-aware point cloud sampling strategy and the introduction of explicit geometric primitives, laying a solid foundation for generating stable and reliable initial grasping poses. In summary, this invention achieves accurate, robust, and human-intention-compliant six-DOF grasping in complex and cluttered scenes, providing effective technical support for the widespread application of intelligent robots in home services, industrial sorting, and other fields. Experiments on the public dataset GraspNet-1 Billion and self-collected multimodal scene datasets show that the method of this invention achieves leading performance in 6-DoF grasping prediction metrics, verifying its high-precision and robust multimodal scene understanding capability in complex and cluttered environments. Experiments on the public dataset GraspNet-1 Billion and self-collected multimodal scene datasets show that the average grasping accuracy of the method of this invention reaches 72.77%, 71.53%, and 39.64% in scenes with known objects, similar objects, and unknown objects, respectively, representing a 3%-8% improvement over the existing best methods, demonstrating significant technical advantages. Attached Figure Description

[0038] Figure 1 This is a diagram illustrating the overall framework of the multimodal large-scale scene perception method based on point cloud multi-manifolds proposed in this invention.

[0039] Figure 2 The schematic diagram of the scene perception and geometry enhancement motion logic generation module designed for this invention shows in detail the collaborative relationship of the three core sub-processes: utility perception point cloud sampling, explicit geometric primitive construction, and adaptive feature aggregation.

[0040] Figure 3 This is a schematic diagram of the consistency feature learning module based on multi-manifold perception designed in this invention. It illustrates how multi-scale Euclidean features are projected onto hyperbolic and spherical manifolds respectively, consistency constraints are applied, and finally feature fusion is performed. Detailed Implementation

[0041] To make the above-mentioned objectives, features and advantages of the present invention more apparent and understandable, the method of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. These embodiments are implemented based on the technical solution of the present invention, and provide implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0042] See Figure 1 The implementation of this invention includes the following steps:

[0043] 1. Construct a multimodal 3D scene understanding dataset

[0044] This embodiment uses the GraspNet-1 Billion dataset for model training and evaluation. This dataset contains 97,280 RGB-D images, covering 190 realistic cluttered scenes and involving 88 everyday grasping objects, and provides dense six-DOF grasping pose annotations. The test set of the dataset is divided into three subsets: "known objects," "similar objects," and "unknown objects," to systematically evaluate the model's generalization ability. The model's input includes scene point clouds, RGB images, and natural language commands.

[0045] 2. Construct a parallel unified perception framework for 3D scenes with multimodal semantic-geometric collaboration.

[0046] like Figure 1 As shown, this invention constructs a multimodal semantically guided grasping framework. Its core idea is to combine the semantic understanding capability of the visual language model CLIP with the geometric reasoning capability of the grasping network through parallel processing and attention alignment mechanisms. This framework consists of two parallel branches. The multimodal semantic understanding branch uses the pre-trained visual language model CLIP as its core, receiving RGB images and natural language instructions. This branch obtains the bounding boxes of potential targets through the object detector DETR, combines visual features extracted by the image encoder with text features processed by the text encoder, calculates the semantic matching degree, and finally generates a semantic attention map indicating the potential grasping region in 3D space that conforms to human intent. The geometric grasping pose generation branch takes scene point clouds as input, extracts features through a point cloud encoder based on shared attention, and processes them using a scene-aware and geometry-enhanced motion logic generation module and a multi-manifold-aware consistency feature learning module. Finally, it outputs a set of physically feasible six-DOF grasping pose candidates that depend only on scene geometry information. .

[0047] Subsequently, the two branches are fused using a cross-attention module. Geometric grasping pose candidates are encoded as query vectors, visual language features from the semantic attention map are used as value vectors, and the 3D positional features of the target bounding box are used as key vectors. Through cross-attention computation, each grasping pose candidate is assigned a relevance score to the semantic intent. The fused features are then mapped to the final probability distribution by a lightweight decoder. Based on this, the system selects the posture with the highest execution priority as the final grasping action.

[0048] The training process is optimized by minimizing the cross-entropy loss between the model's predicted distribution and the real expert action labels. The entire framework achieves the goal of generating stable grasping postures based on human intent in cluttered scenes through parallel semantic and geometric reasoning and precise cross-attention fusion.

[0049] 3. Design a point cloud encoder based on shared attention.

[0050] This point cloud Transformer encoder employs a symmetric U-Net structure with a depth of 4. To enhance cross-layer feature consistency, a shared learnable vector is introduced as an attention bias for each pair of encoder and decoder layers connected by skip connections. The improved attention calculation method is as follows:

[0051]

[0052] in, Indicates the center point The output characteristics, It is a point The local neighborhood point set, , , , It is a point-level feature transformation function. It is a positional encoding. It is a normalization function. This represents element-wise multiplication. These are the shared, learnable vectors. It is The method transforms the mapping function into a bias term; it enhances cross-level feature consistency by establishing explicit geometric semantic alignment constraints between corresponding levels through parameter-level sharing.

[0053] 4. Construct a high-dimensional feature reasoning module that integrates multi-granularity geometric topology perception and explicit structural prior enhancement for "scene-object-component" relationships.

[0054] like Figure 2As shown, this module first performs utility-aware point cloud sampling, and then uses the grasp probability score predicted by the backbone network. The weighted distance in the farthest point sampling is redefined as:

[0055] Prioritize high-potential capture points Subsequently, an explicit geometric primitive is constructed for each seed point, in its neighborhood set. Calculate the flatness factor Linearity factor Scattering factor Local curvature Local point density and relative offset vector And concatenate them into explicit geometric feature vectors. Finally, adaptive feature aggregation is used to fuse with implicit features, generating enhanced features for predicting the grasping approximation direction. Represents the variance of the maximum extension direction. Represents the variance of the secondary extension direction. This represents the variance in the direction of minimum variance, and is also interpreted as the normal direction of that local neighborhood. , Represents a point set The geometric centroid.

[0056] 5. A consistency feature learning method for multi-manifold sensing is proposed.

[0057] like Figure 3 The multi-manifold perception consistency feature learning method shown targets the local features extracted from the seed point in the receptive field at different scales. Through two independent lightweight network branches, it is mapped to a hyperbolic space suitable for modeling hierarchical structures and a spherical space suitable for representing directional information, respectively; in the hyperbolic space, a Poincaré sphere model is adopted. The curvature is set to 1 for features projected into this space. Calculate the hyperbolic space consistency loss :

[0058]

[0059] Where N is the number of multi-scale receptive fields, which was set to 4 in the experiment. The values ​​of each receptive field ranged from [-0.02, 0.01], [-0.02, 0.02], [-0.02, 0.03], to [-0.02, 0.04]. It is the hyperbolic geodesic distance between two points; in spherical space In the context of features projected into this space... Calculate the consistency loss of the spherical space:

[0060]

[0061] in, It is the spherical geodesic distance between two points; hyperbolic geodesic distance. The calculation formula is:

[0062]

[0063] in, Let represent the geodesic distance between points p and q in hyperbolic space, and let c represent the curvature in hyperbolic space. This represents the inverse hyperbolic cosine function.

[0064] Finally, features from hyperbolic and spherical spaces are stitched together and weighted, and then combined with the loss from the main grasping pose estimation task. Combined, they constitute the overall training loss. :

[0065] (6)

[0066] in, and These are the balancing weights, and in the experiment, they were all set to 0.05.

[0067] 6. The model is trained end-to-end using cross-attention and policy optimization algorithms.

[0068] This method employs cross-attention and policy optimization algorithms for end-to-end model training. The training process combines behavior cloning and interactive policy optimization. In the expert imitation pre-training phase, the cross-entropy loss is minimized.

[0069]

[0070] in, It is the unique heat encoding of expert actions. It represents the probability predicted by the model, and K is the number of candidate grasping poses, with a value of 128.

[0071] For model training configuration, the Adam optimizer was used for parameter optimization, with an initial learning rate of 0.001, a batch size of 2, and a total of 360 training epochs. Each input to the network contained 20,000 point clouds, with an initial viewpoint sampling count of 300 per point. A step-by-step learning rate scheduling strategy was employed, decaying by a factor of 0.1 in epochs 8, 12, and 16. The decay period for the batch normalization layer was set to 2 epochs, with a decay rate of 0.5.

[0072] 7. Utilize the trained model to generate stable logic for 3D scene perception and human intent capture.

[0073] The system utilizes a trained model to generate stable grasping logic for 3D scene perception and human intent. During the inference phase, the system receives scene point clouds, RGB images, and natural language commands. Through parallel processing and fusion of the above steps, it outputs the probability distribution of grasping pose candidates and selects the pose with the highest score as the final grasping action, thereby achieving accurate and stable grasping of specific targets in cluttered scenes.

[0074] Table 1 shows the performance comparison results of this invention and existing state-of-the-art methods in the RealSense acquisition scenario of the GraspNet-1Billion dataset. In terms of experimental setup, the RealSense test scenario from the publicly available GraspNet-1Billion dataset was used for performance evaluation. The main evaluation metric was the average precision (AP), which was calculated by combining the grasping success rate under five different friction coefficients from 0.2 to 1.0 (in 0.2 increments). AP0.8 and AP0.4 represent the average precision when the friction coefficients are 0.8 and 0.4, respectively.

[0075] This method is implemented based on the PyTorch 1.7.1 framework and runs on the Ubuntu 20.04.2 LTS operating system. The experimental hardware platform configuration includes an NVIDIA GeForce RTX 3090 GPU, an Intel Core i9-10900K processor, and 128GB of memory. The software environment uses Python 3.7.9 and CUDA 11.7.0.

[0076] Table 1

[0077]

[0078] As shown in Table 1, on the GraspNet-1 Billion dataset, our method achieved the best overall performance across most settings in the three test subsets: "known objects," "similar objects," and "unknown objects," surpassing all comparable methods. Specifically, in the most challenging "unknown objects" generalization test, our method achieved an average precision of 39.64, a significant improvement over the previous best, SGSIN (26.69), demonstrating excellent generalization ability when facing entirely new objects. Even with low friction, our method still achieved an AP0.4 of 24.58 on unknown objects, far exceeding other methods, indicating that it maintained a high success rate even with low friction. In the "similar objects" subset, our method also performed exceptionally well, with significantly higher AP (71.53) and AP0.4 (67.74), showcasing its adaptability to slight variations in object shape and category.

[0079] Other methods are listed in Table 1:

[0080] GraspNet-1Billion corresponds to the method proposed by Fang et al. (Fang HS, Wang C, Gou M, et al. Graspnet-1billion: A large-scale benchmark for general object grasping[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 11444-11453.)

[0081] SSCL corresponds to the method proposed by Li et al. (Li Y, Kong T, Ma Y, et al. SSCL: Self-supervised curriculum learning for 6-DoF grasp pose detection[C]. 2021 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE,2021: 3571-3578).

[0082] GraNet corresponds to the method proposed by Wang et al. (Wang H, Niu W, Zhuang C, et al. Granet: A multi-level graph network for 6-dof grasp pose generation in cluttered scenes[C] / / IEEE / RSJ International Conference on Intelligent Robots and Systems. IEEE, 2023: 937-943.)

[0083] Scale-Balanced-Grasp corresponds to the method proposed by Ma et al. (Ma H, Huang D. Towards scale-balanced 6-dot grasp detection in cluttered scenes[C] / / Conference on RobotLearning. PMLR, 2023: 2004-2013.)

[0084] Real-to-Sim Grasp corresponds to the method proposed by Cai et al. (Cai J, Chen Z, Wu X, et al. Real-to-sim grasp: Rethinking the gap between simulation and real world ingrasp detection[C] / / Conference on Robot Learning. PMLR, 2024: 1109-1124.)

[0085] SGSIN corresponds to the method proposed by Wang et al. (Wang W, Zhu H, Ang MH. SGSIN: Simultaneous grasp and suction inference network via attention-based affordance learning[J]. IEEE Transactions on Industrial Electronics, 2024.)

[0086] FastGNet corresponds to the method proposed by Ding et al. (Ding Z, Wang A, Gao M, et al. Fastgnet: An efficient 6-dof grasp detection method with multi-attention mechanisms and point transformer network[J]. Measurement Science and Technology, 2024, 35(4): 045020.)

[0087] ZeroGrasp corresponds to the method proposed by Iwase et al. (Iwase S, Irshad MZ, Liu K, et al. Zerograsp: Zero-shot shape reconstruction enabled robotic grasping[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2025: 17405-17415.)

[0088] Mink-GraSNet corresponds to the method proposed by Lyu et al. (Lyu Y, Li J, Wang A, et al. Mink-grasnet: Computationally efficient 6-dof grasp detection for large-scale AI-powered robotics[J]. The Journal of Supercomputing, 2025: 1-23.)

[0089] This invention proposes a multimodal large model scene perception method based on point cloud multi-manifolds, aiming to solve key problems in existing grasping technologies such as cascaded architecture error accumulation, insufficient cross-layer semantic correlation, unreliable motion logic generation, and bias in understanding geometric topology information. This method constructs a parallel unified 3D scene perception framework with multimodal semantic-geometric collaboration. By parallel processing of semantic understanding branches and geometric grasping pose generation branches, combined with a cross-attention fusion mechanism, it effectively alleviates the error propagation problem of traditional cascaded methods. A point cloud encoder based on shared attention is designed, which enhances the cross-level semantic continuity of point cloud features in the U-Net architecture by introducing learnable shared attention bias vectors between layers. A motion logic generation module for scene perception and geometry enhancement is proposed. A utility-aware point cloud sampling strategy is used to select high-potential grasping points and explicitly construct local geometric primitive features. An adaptive fusion mechanism provides reliable geometric cues for grasping approximation direction prediction. An innovative consistency feature learning method for multi-manifold perception is proposed, which maps multi-scale local features to hyperbolic manifolds and spherical manifolds respectively. By utilizing the sensitivity of hyperbolic manifolds to hierarchical structure and spherical manifolds to directional angle information, and by achieving cross-scale feature alignment through geodesic distance constraints, the ability to represent the geometric topological information of complex objects is significantly enhanced. Through end-to-end training using cross-attention and policy optimization algorithms, the method of this invention can generate six-DOF grasping postures that conform to both semantic intent and physical stability in highly cluttered scenes based on human natural language commands. Experimental results on the GraspNet-1 Billion dataset demonstrate that the method achieves optimal overall performance in scenarios with known objects, similar objects, and unknown objects, validating the effectiveness and advancement of this method in the field of embodied intelligence in robots.

[0090] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A multimodal large-scale scene perception method based on point cloud multi-manifolds, characterized in that... Includes the following steps: 1) Construct a multimodal 3D scene understanding dataset; 2) Construct a parallel unified perception framework for 3D scenes with multimodal semantic-geometric collaboration; 3) Design a point cloud encoder based on shared attention; 4) Construct a high-dimensional feature reasoning module that integrates multi-granularity geometric topology perception and explicit structural prior enhancement for "scene-object-component"; 5) A consistency feature learning method for multi-manifold perception is proposed; 6) The model is trained end-to-end using cross-attention and policy optimization algorithms; 7) Utilize the trained model to generate stable logic for 3D scene perception and human intent capture.

2. The multimodal large model scene perception method based on point cloud multimanifolds as described in claim 1, characterized in that... In step 1), the specific method for constructing the multimodal 3D scene understanding dataset is as follows: the publicly available large-scale six-degree-of-freedom grasping dataset GraspNet-1 Billion is used for model training and evaluation; this dataset contains 97,280 RGB-D images, covering 190 real cluttered scenes, involving 88 everyday grasping objects; the dataset provides high-precision 3D mesh models, object instance segmentation masks, and generates dense six-degree-of-freedom grasping pose annotations through an automated process, which have undergone force closure analysis and collision detection; the dataset is divided into a training set and a test set, and the test set is further divided into three subsets: "known objects", "similar objects" and "unknown objects" to systematically evaluate the model's performance under different generalization requirements; the model's input includes scene point clouds, RGB images, and natural language instructions describing the target objects.

3. The multimodal large model scene perception method based on point cloud multimanifolds as described in claim 1, characterized in that... In step 2), the specific method for constructing a parallel 3D scene unified perception framework with multimodal semantic-geometric collaboration is as follows: the framework includes a multimodal semantic understanding branch and a geometric grasping pose generation branch that work in parallel; the multimodal semantic understanding branch takes a pre-trained visual language model as its core and receives RGB images and natural language instructions; first, it obtains the two-dimensional bounding boxes of all potential targets in the image through a target detector; then, it combines the two-dimensional bounding boxes with the corresponding visual features extracted by the image encoder to generate a visual representation containing target appearance information; simultaneously, the language instructions are processed by a text encoder to obtain text features; by calculating the cosine similarity between the visual features and the text features, visual-language features that quantify the degree of semantic matching between each position in the scene and the target indicated by the instruction are generated, ultimately forming a semantic attention map indicating the potential grasping area in 3D space that conforms to human intent; the aforementioned... The geometric grasping pose generation branch takes scene point cloud as input and generates a set of physically feasible six-DOF grasping pose candidates that depend only on scene geometry information through a point cloud encoder and subsequent modules. Then, the outputs of the two branches are fused through a cross-attention module: the grasping pose candidates output by the geometric grasping pose generation branch are encoded as query vectors; the visual-linguistic features contained in the semantic attention map output by the multimodal semantic understanding branch are used as value vectors; the position information of the target bounding box in three-dimensional space is encoded as key vectors; the value vectors are weighted and aggregated by calculating the similarity between the query and the key, so that each grasping pose can dynamically focus on the semantic region related to the task and obtain a preliminary score accordingly; finally, the fused features are mapped to the final execution priority probability distribution of each grasping pose candidate by a lightweight decoder, and the system selects the grasping pose that best matches human intent based on this distribution.

4. The multimodal large model scene perception method based on point cloud multimanifolds as described in claim 1, characterized in that... In step 3), the specific method for designing the point cloud encoder based on shared attention is as follows: the encoder adopts a point cloud Transformer with a U-Net structure; to alleviate the semantic continuity disruption caused by independent learning of attention parameters between corresponding layers of the encoder and decoder, a shared learnable vector is introduced as an attention bias for each pair of encoder and decoder layers connected by skip connections; this shared bias is added to the initial attention score before attention score normalization, and its calculation method is as follows: in, Indicates the center point The output characteristics, It is a point The local neighborhood point set, , , , It is a point-level feature transformation function. It is a positional encoding. It is a normalization function. This represents element-wise multiplication. These are the shared, learnable vectors. It is The method transforms the mapping function into a bias term; it enhances cross-level feature consistency by establishing explicit geometric semantic alignment constraints between corresponding levels through parameter-level sharing.

5. The multimodal large model scene perception method based on point cloud multimanifolds as described in claim 1, characterized in that... In step 4), the specific method for constructing the high-dimensional feature reasoning module for "scene-object-part" multi-granularity geometric topology perception and explicit structural prior enhancement is as follows: the module sequentially performs utility-aware point cloud sampling, explicit geometric primitive construction, and adaptive feature aggregation; firstly, it uses the features extracted from the backbone network to predict the objectivity score for each point. and capture probability score Foreground object points are filtered using objectivity scores, and the weighted distance in the farthest point sampling is redefined based on the grasp probability score: in It is a candidate point. It is the set of selected points. It is the shortest Euclidean distance, and points with high grasping potential are sampled first as seed points; Then, for each seed point In its fixed radius neighborhood set The flatness factor is calculated using principal component analysis. Linearity factor Scattering factor Local normal direction Local curvature The local point density F and the relative offset vector to the neighborhood centroid. Concatenate to form explicit geometric feature vectors Finally, the explicit geometric features are adaptively fused with the implicit features extracted by the backbone network through a dynamic gating mechanism to generate an enhanced feature representation for predicting the grasping approximation direction.

6. The multimodal large model scene perception method based on point cloud multimanifolds as described in claim 1, characterized in that... In step 5), the specific method for proposing the consistency feature learning method for multi-manifold perception is as follows: the method targets the local features extracted from the seed point under different receptive fields. Through two independent lightweight network branches, it is mapped to a hyperbolic space suitable for modeling hierarchical structures and a spherical space suitable for representing directional information, respectively; in the hyperbolic space, a Poincaré sphere model is adopted. For features projected into this space Calculate the hyperbolic space consistency loss : Where N is the number of multi-scale receptive fields. It is the hyperbolic geodesic distance between two points; in spherical space In the context of features projected into this space... Calculate the consistency loss of spherical space : in, It is the spherical geodesic distance between two points; hyperbolic geodesic distance. The calculation formula is: Finally, features from hyperbolic and spherical spaces are stitched together and weighted, and then combined with the loss from the main grasping pose estimation task. Combined, they constitute the overall training loss. : in, and It is a balancing weight.

7. The multimodal large model scene perception method based on point cloud multimanifolds as described in claim 1, characterized in that... In step 6), the specific method for end-to-end training of the model using the cross-attention and policy optimization algorithm is as follows: The training process combines behavioral cloning with interactive policy optimization; in the expert imitation pre-training phase, based on offline expert demonstration data, the model prediction probability distribution is minimized. Unique heat encoding of real expert motion Training is performed using the cross-entropy loss between the two: 。 8. The multimodal large model scene perception method based on point cloud multimanifolds as described in claim 1, characterized in that... In step 7), the specific method for generating stable grasping logic for 3D scene perception and human intent using the trained model is as follows: During the inference phase, the model receives scene point clouds, RGB images and natural language instructions; the multimodal semantic understanding branch outputs a semantic attention map, and the geometric grasping posture generation branch generates a set of physically feasible six-degree-of-freedom grasping posture candidates in parallel. The cross-attention module scores geometric candidates based on semantic information to obtain the final probability distribution. The system selects the posture with the highest score as the grasping action to achieve accurate and stable grasping of specific targets in cluttered scenes.

Images