Mechanical arm generalization operation method and system based on mixed geometric representation pre-training

Abstract

The invention discloses a mechanical arm generalization operation method and system based on mixed geometric representation pre-training, and belongs to the field of mechanical arm generalization operation. The method comprises the steps that a 3D-MOV data set containing an object, a fine-grained scene and desktop-level 3D point cloud is constructed, and a mixed geometric representation encoder is trained based on the pre-training; and after mapping to the hyperbolic space optimization learning, returning to the Euclidean space to complete self-supervised reconstruction. The scene observation point cloud is rendered into an RGB image, the RGB image is subjected to double-path distribution after being processed, strategy network fusion information is subjected to fine adjustment, and the action of the mechanical arm is predicted. Hyperbolic geometric representation is introduced into visual self-supervision pre-training based on 3D multi-view angles, 3D multi-view representation pre-training based on mixed geometric space is used for mechanical arm operation, the next operation action is effectively predicted, and the generalization operation capacity of a mechanical arm in different scenes is greatly enhanced.

Description

Technical Field

[0001] This invention belongs to the field of robotic arm generalization operation, specifically relating to a robotic arm generalization operation method and system based on hybrid geometric representation pre-training. Background Technology

[0002] In recent years, developing robot manipulation methods with versatility and high generalization capabilities has been crucial for the deployment of robots in real-world environments. To address this challenge, researchers have drawn on advancements in computer vision, exploring the use of visual pre-training techniques to learn robust visual representations and then fine-tuning them for application in downstream robotic arm manipulation strategies. This "pre-training-fine-tuning" paradigm has proven effective in improving the model's ability to handle diverse manipulation tasks and scene perturbations.

[0003] Existing visual pre-training methods for robotic arm operations are mainly divided into supervised pre-training and self-supervised pre-training. Supervised pre-training methods primarily rely on labeled 2D image or video datasets. Self-supervised pre-training methods, however, do not require labeled information and have therefore attracted considerable attention. Among these, the current state-of-the-art self-supervised visual pre-training method for robotic arm operations, 3D-MVP (3D Multiview Pretraining), demonstrates strong sample efficiency and competitiveness in downstream task performance by explicitly constructing 3D multiview representations. Specifically, 3D-MVP renders 3D point clouds into multiple orthogonal viewpoint images as input, uses a multiview Transformer to construct a pre-trained encoder, and then combines the pre-trained encoder with the downstream robotic arm operation strategy for fine-tuning on specific operation tasks. Although existing methods have made progress in downstream multi-operation tasks through visual pre-training, they still face several shortcomings in practical applications: 1. Existing robotic arm operation methods exhibit a significant performance drop when environmental disturbances are applied, severely limiting their deployment in real-world environments. This indicates that the model's generalization ability needs improvement.

[0004] 2. Existing vision-based pre-training methods for robotic arm manipulation all confine the representation embedding space to Euclidean space. The geometric properties of Euclidean space are "flat" (with zero curvature), and its uniform linear distance metric limits its ability to model complex data topologies. In particular, for structured relationships crucial to robot spatial perception (e.g., the hierarchical relationship between local components and the overall scene), Euclidean space struggles to effectively model and capture them using a unified metric.

[0005] 3. Existing robotic arm operation methods based on 3D multi-view pre-training require the same number of input views in both the pre-training and fine-tuning stages. For example, 3D-MVP requires 5 input view images in both the pre-training and fine-tuning stages, which greatly limits the scalability and flexibility of model deployment.

[0006] 4. Existing robotic arm operation methods based on 3D self-supervised pre-training are mainly pre-trained on point clouds at the single object level. However, the robotic arm perception process involves not only a single object, but also complex scene information and interactions between objects. Therefore, the singularity of this pre-training data limits the potential of the pre-training method.

[0007] The above analysis demonstrates a pressing need for a pre-training method capable of capturing robust 3D spatial representations for robotic arm operations. This method should significantly improve the generalization of operational strategies under various perturbation conditions. Furthermore, it should possess scalability and facilitate the construction of a robust and generalizable robotic arm operating system. Summary of the Invention

[0008] The purpose of this invention is to overcome the problem of poor generalization in existing pre-training methods for robotic arm operation, and to propose a robotic arm generalization operation method and system based on hybrid geometric representation pre-training.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for generalizing robotic arm operations based on hybrid geometric representation pre-training, comprising the following steps: Construct a 3D-MOV dataset, which includes object-level 3D point clouds, fine-grained scene-level 3D point clouds, and desktop-level 3D point clouds; The hybrid geometric representation encoder is pre-trained based on 3D point clouds from the 3D-MOV dataset to obtain a pre-trained hybrid geometric representation encoder, including the following steps: Multi-view rendering preprocessing is performed on 3D point cloud instances to obtain multi-view Euclidean representations; The multi-view Euclidean representation is mapped onto hyperbolic space to obtain the mapped feature representation; In hyperbolic space, the mapped feature representation is optimized, learned, and constrained. The optimization introduces the Top-K neighborhood rank correlation loss to obtain the hyperbolic learned features. The hyperbolic learned features are returned to Euclidean space through logarithmic mapping and used for reconstruction tasks to complete self-supervised reconstruction task pre-training, resulting in a pre-trained hybrid geometric representation encoder. Based on the pre-trained hybrid geometric representation encoder, fine-tuning of the robotic arm operation task is performed to complete the generalization operation of the robotic arm based on the hybrid geometric representation pre-training, including the following steps: The point cloud of the robotic arm scene observation is rendered into RGB images from multiple perspectives through orthogonal projection. The RGB images from each perspective are then segmented and embedded to obtain a multi-view embedded representation of the scene. The scene multi-view embedding representation is processed by dual-path distribution. One path is input into the pre-trained hybrid geometric representation encoder to obtain the encoded scene feature representation. The other path, together with the natural language description of the robotic arm operation task, is input into the robotic arm operation policy network. The encoded scene feature representation is then input into the robotic arm operation policy network. After fusing scene feature representation encoded by the robotic arm operation strategy network, scene multi-view embedding expression, and natural language description of the robotic arm operation task, fine-tuning is completed by following the optimization constraint logic of the robotic arm operation strategy network to predict the next operation action of the robotic arm.

[0010] Furthermore, the 3D point cloud instance undergoes multi-view rendering preprocessing to obtain a multi-view Euclidean representation, specifically: The 3D point cloud in the 3D-MOV dataset is rendered into RGB images with five orthogonal viewpoints through orthogonal projection. The RGB images of each viewpoint are then divided into blocks, randomly masked, and embedded to obtain a multi-view image block embedding representation. Multi-view image patch embedding representation is input into a feature encoding network along with CLS embedding and mask embedding, and outputs a multi-view Euclidean representation. Five orthogonal RGB images, namely top, front, back, left, and right views; Mapping the multi-view Euclidean representation onto hyperbolic space yields the mapped feature representation, specifically: The Euclidean features from each viewpoint are regarded as tangent vectors at the origin of the Lorentz hyperbolic model. The tangent vectors are mapped onto the hyperbolic space through exponential mapping to obtain hyperbolic features, including CLS embedding, image patch embedding and Mask embedding in the hyperbolic space.

[0011] Furthermore, the mapped feature representation is optimized, learned, and constrained in the hyperbolic space to obtain the hyperbolic learned features, specifically: In hyperbolic space, topological consistency constraint loss and hierarchical structure constraint loss are applied to hyperbolic features to complete hyperbolic feature optimization and obtain optimized hyperbolic features. The topology consistency constraint loss is the Top-K neighborhood rank correlation loss, as shown in the following equation:

[0012] in, Represents the loss related to the rank of the Top-K neighborhood, where and Let represent the Top-K neighborhood rank correlation matrices in Euclidean space and hyperbolic space, respectively; The hierarchical structure constraint loss is the implication loss, which constrains the image patch embedding and mask embedding within the implication cone defined by hyperbolic CLS embedding. The hyperbolic learned features are returned to Euclidean space through a logarithmic mapping and used for the reconstruction task, completing the pre-training of the self-supervised reconstruction task and obtaining a pre-trained hybrid geometric representation encoder; specifically: The optimized hyperbolic features are mapped back to Euclidean space by logarithmic mapping to obtain mapped feature vectors. The mapped feature vectors are then used to perform a self-supervised reconstruction task, thus completing the pre-training of the hybrid geometric representation encoder. The self-supervised reconstruction task is shown in the following formula:

[0013]

[0014] in, and represents the pixel reconstruction loss within and between viewpoints, respectively, and W and H represent the width and height of the image, respectively. and Let represent the ground truth value and the predicted value of the j-th pixel in the i-th view, respectively. This represents the ground truth value of the j-th pixel from the perspective of the a-th anchor. This represents the predicted value of the j-th pixel from the i-th viewpoint, as seen from the a-th anchor viewpoint.

[0015] Furthermore, self-supervised reconstruction tasks include intra-view reconstruction and inter-view reconstruction; The reconstruction process between viewpoints is as follows: Randomly select one of the multiple rendering perspectives as the anchor perspective, perform cross-attention calculation on the image patch representations of the other perspectives and the representation of the anchor perspective, obtain the calculation result, input the calculation result into the decoder of the standard self-supervised pre-training paradigm framework, and output the reconstructed image of the anchor perspective for the other perspectives.

[0016] Furthermore, the construction process of the 3D-MOV dataset is as follows: Collect object-level 3D point clouds from the Objaverse-XL dataset; We collect raw indoor scene point clouds from the ScanNet dataset and segment each raw indoor scene to obtain fine-grained scene-level 3D point clouds. We collected desktop scene point clouds from the TO-Scene dataset, including vanilla-level and crowd-level desktop scene point clouds.

[0017] Furthermore, the RGB images from multiple perspectives are RGB images from three orthogonal perspectives: top, front, and right.

[0018] Furthermore, the robotic arm manipulation strategy network is a multi-view robotic arm manipulation strategy RVT.

[0019] Secondly, the present invention provides a robotic arm generalization operating system based on hybrid geometric representation pre-training, comprising: The dataset construction module is used to build 3D-MOV datasets, which include object-level 3D point clouds, fine-grained scene-level 3D point clouds, and desktop-level 3D point clouds. The pre-trained encoder module is used to pre-train the hybrid geometric representation encoder based on the 3D point cloud in the 3D-MOV dataset to obtain a pre-trained hybrid geometric representation encoder, including the following steps: Multi-view rendering preprocessing is performed on 3D point cloud instances to obtain multi-view Euclidean representations; The multi-view Euclidean representation is mapped onto hyperbolic space to obtain the mapped feature representation; The mapped feature representation is optimized, learned, and constrained in hyperbolic space to obtain hyperbolic learned features; The hyperbolic learned features are returned to Euclidean space through logarithmic mapping and used for reconstruction tasks to complete self-supervised reconstruction task pre-training, resulting in a pre-trained hybrid geometric representation encoder. The robotic arm generalization operation module is used to perform fine-tuning of robotic arm operation tasks based on the pre-trained hybrid geometric representation encoder, and to complete the robotic arm generalization operation based on the hybrid geometric representation pre-training, including the following steps: The point cloud of the robotic arm scene observation is rendered into RGB images from multiple perspectives through orthogonal projection. The RGB images from each perspective are then segmented and embedded to obtain a multi-view embedded representation of the scene. The scene multi-view embedding representation is processed by dual-path distribution. One path is input into the pre-trained hybrid geometric representation encoder to obtain the encoded scene feature representation. The other path, together with the natural language description of the robotic arm operation task, is input into the robotic arm operation policy network. The encoded scene feature representation is then input into the robotic arm operation policy network. After fusing scene feature representation encoded by the robotic arm operation strategy network, scene multi-view embedding expression, and natural language description of the robotic arm operation task, fine-tuning is completed by following the optimization constraint logic of the robotic arm operation strategy network to predict the next operation action of the robotic arm.

[0020] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a robotic arm generalization operation method based on hybrid geometric representation pre-training.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a robotic arm generalization operation method based on hybrid geometric representation pre-training.

[0022] Compared with the prior art, the present invention has the following beneficial technical effects: This invention proposes a generalization method for robotic arm operations based on hybrid geometric representation pre-training. Existing visual pre-training methods for robotic arm operations typically use embedding representations in Euclidean space. This flat geometric space makes it difficult for models to effectively model the hierarchical and structured relationships required for robot spatial perception, resulting in insufficient robustness. This invention introduces hyperbolic geometric representation into 3D multi-view visual self-supervised pre-training for the first time. It innovatively proposes a Top-K neighborhood rank correlation loss to constrain the relative order of image patch embeddings in Euclidean and hyperbolic spaces, thereby maintaining the structural consistency of image patch embeddings across geometric spaces. Simultaneously, it introduces an implication loss to constrain the consistency of global and local semantic information. The Top-K neighborhood rank correlation loss is designed to constrain the consistency of the relative arrangement order of image patch embeddings across geometric spaces. Existing multi-view representation pre-training techniques for robotic arm operations (such as 3D-MVP) require the same number of views during pre-training and fine-tuning, lacking scalability and flexibility. This invention designs a method for 3D multi-view representation pre-training based on hybrid geometric space for robotic arm operation. The decoder of the standard self-supervised pre-training paradigm framework can adopt the decoder of the MAE framework, and is extended based on the MAE (Mask Autoencoder) framework. The designed hybrid geometric representation encoder adopts a viewpoint decoupling design, allowing the model to use different numbers of input viewpoint images during the fine-tuning and pre-training stages. This flexibility greatly improves the scalability and applicability of the model in actual deployment. This invention introduces a large-scale 3D-MOV dataset, which integrates high-quality 3D point cloud data from four different sources (including object-level, indoor scene-level, and desktop scene-level). Each type of data contributes different structural layouts and background information. By increasing data diversity rather than simply increasing the scale, the representational ability of the model is significantly enhanced. This diverse data integration (rather than simply increasing the scale) helps to capture more robust pre-trained representations and helps to study which types of data are more helpful for downstream tasks. Attached Figure Description

[0023] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of the invention in any way. Furthermore, the shapes and proportions of the components in the drawings are merely illustrative to aid in understanding the invention and do not specifically limit the shapes and proportions of the components of the invention.

[0024] In the attached diagram: Figure 1 This is a flowchart of the robotic arm generalization operation method based on hybrid geometric representation pre-training according to the present invention.

[0025] Figure 2 This is a simplified structural diagram of the robotic arm generalization operating system based on hybrid geometric representation pre-training of the present invention.

[0026] Figure 3 This is a schematic diagram of an electronic device for the robotic arm generalization operation method based on hybrid geometric representation pre-training according to the present invention.

[0027] Figure 4 This is a schematic diagram of the 3D multi-view self-supervised pre-training process based on hybrid geometric space representation in an embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram illustrating the process of fine-tuning a pre-trained hybrid geometric representation encoder for downstream robotic arm operations in an embodiment of the present invention.

[0029] Figure 6 The 3D-MOV dataset in this embodiment was constructed by collecting high-quality 3D datasets from existing data sources.

[0030] Figure 7 This refers to the inter-view reconstruction process during the pre-training process in this embodiment of the invention.

[0031] Figure 8 This is a schematic diagram showing the comparative experimental results of the method of this invention with other methods on 18 tasks in the RLBench benchmark. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] Example 1 See Figure 1 A generalization method for robotic arms based on hybrid geometric representation pre-training includes the following steps: Construct a 3D-MOV dataset, which includes object-level 3D point clouds, fine-grained scene-level 3D point clouds, and desktop-level 3D point clouds; The hybrid geometric representation encoder is pre-trained based on 3D point clouds from the 3D-MOV dataset to obtain a pre-trained hybrid geometric representation encoder, including the following steps: Multi-view rendering preprocessing is performed on 3D point cloud instances to obtain multi-view Euclidean representations; the multi-view Euclidean representations are mapped to hyperbolic space to obtain mapped feature representations; the mapped feature representations are optimized, learned, and constrained in hyperbolic space, with optimization introducing Top-K neighborhood rank correlation loss to obtain hyperbolic learned features; the hyperbolic learned features are returned to Euclidean space through logarithmic mapping and used for reconstruction tasks to complete self-supervised reconstruction task pre-training, resulting in a pre-trained hybrid geometric representation encoder; Based on the pre-trained hybrid geometric representation encoder, fine-tuning of the robotic arm operation task is performed to complete the generalization operation of the robotic arm based on the hybrid geometric representation pre-training, including the following steps: The point cloud of the robotic arm scene observation is rendered into RGB images from multiple perspectives through orthogonal projection. The RGB images from each perspective are then segmented and embedded to obtain a multi-view embedding representation of the scene. This multi-view embedding representation is then processed by a dual-path distribution. One path is input into a pre-trained hybrid geometric representation encoder to obtain the encoded scene feature representation. The other path, along with the natural language description of the robotic arm operation task, is input into the robotic arm operation policy network. The encoded scene feature representation, the other path's multi-view embedding representation, and the natural language description of the robotic arm operation task are then fused by the robotic arm operation policy network. Fine-tuning is performed according to the optimization constraint logic of the robotic arm operation policy network to predict the next operation action of the robotic arm.

[0034] This embodiment constructs a rich 3D-MOV dataset, covering 3D point clouds at different levels, providing comprehensive data support for model training. Through unique pre-training methods such as multi-view rendering and hyperbolic space mapping, the resulting hybrid geometric representation encoder can more accurately capture geometric features and improve feature extraction capabilities. In the fine-tuning stage of the robotic arm's operation tasks, the scene observation point cloud is converted into multi-view RGB images and embedded in blocks. Combined with dual-path distribution processing, the advantages of the pre-trained encoder are fully utilized, and natural language descriptions are introduced to enable the robotic arm to better understand the operation tasks. Finally, by fusing multiple pieces of information and following optimization constraint logic, fine-tuning is completed, effectively predicting the next operation action. This greatly enhances the robotic arm's generalization ability in different scenarios, reduces dependence on specific scenarios, improves operational flexibility and accuracy, and promotes the application of robotic arms in complex environments.

[0035] Example 2 See Figure 2 A robotic arm generalization operating system based on hybrid geometric representation pre-training includes: The dataset construction module is used to build 3D-MOV datasets, which include object-level 3D point clouds, fine-grained scene-level 3D point clouds, and desktop-level 3D point clouds. The pre-trained encoder module is used to pre-train the hybrid geometric representation encoder based on the 3D point cloud in the 3D-MOV dataset to obtain a pre-trained hybrid geometric representation encoder, including the following steps: Multi-view rendering preprocessing is performed on 3D point cloud instances to obtain multi-view Euclidean representations; The multi-view Euclidean representation is mapped onto hyperbolic space to obtain the mapped feature representation; The mapped feature representation is optimized, learned, and constrained in hyperbolic space to obtain hyperbolic learned features; The hyperbolic learned features are returned to Euclidean space through logarithmic mapping and used for reconstruction tasks to complete self-supervised reconstruction task pre-training, resulting in a pre-trained hybrid geometric representation encoder. The robotic arm generalization operation module is used to perform fine-tuning of robotic arm operation tasks based on the pre-trained hybrid geometric representation encoder, and to complete the robotic arm generalization operation based on the hybrid geometric representation pre-training, including the following steps: The point cloud of the robotic arm scene observation is rendered into RGB images from multiple perspectives through orthogonal projection. The RGB images from each perspective are then segmented and embedded to obtain a multi-view embedded representation of the scene. The scene multi-view embedding representation is processed by dual-path distribution. One path is input into the pre-trained hybrid geometric representation encoder to obtain the encoded scene feature representation. The other path, together with the natural language description of the robotic arm operation task, is input into the robotic arm operation policy network. The encoded scene feature representation is then input into the robotic arm operation policy network. After fusing scene feature representation encoded by the robotic arm operation strategy network, scene multi-view embedding expression, and natural language description of the robotic arm operation task, fine-tuning is completed by following the optimization constraint logic of the robotic arm operation strategy network to predict the next operation action of the robotic arm.

[0036] Example 3 See Figure 3An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a generalized operation method for a robotic arm based on pre-trained hybrid geometric representations. This method involves: constructing a 3D-MOV dataset, which includes object-level 3D point clouds, fine-grained scene-level 3D point clouds, and desktop-level 3D point clouds; pre-training a hybrid geometric representation encoder based on the 3D point clouds in the 3D-MOV dataset to obtain a pre-trained hybrid geometric representation encoder, including the following steps: performing multi-view rendering preprocessing on 3D point cloud instances to obtain multi-view Euclidean representations; mapping the multi-view Euclidean representations to hyperbolic space to obtain mapped feature representations; optimizing, learning, and constraining the mapped feature representations in hyperbolic space, introducing Top-K neighborhood rank correlation loss for optimization to obtain hyperbolic learned features; returning the hyperbolic learned features to Euclidean space through a logarithmic mapping and using them for reconstruction tasks to complete self-supervised reconstruction task pre-training, resulting in... A pre-trained hybrid geometric representation encoder is used to fine-tune the robotic arm operation task based on the pre-trained hybrid geometric representation encoder, completing the robotic arm generalization operation based on the hybrid geometric representation pre-training. This includes the following steps: rendering the robotic arm scene observation point cloud into RGB images from multiple perspectives through orthogonal projection; segmenting and embedding the RGB images from each perspective to obtain a multi-view embedding representation of the scene; performing dual-path distribution on the multi-view embedding representation of the scene, inputting one path to the pre-trained hybrid geometric representation encoder to obtain the encoded scene feature representation; inputting the other path along with the natural language description of the robotic arm operation task into the robotic arm operation policy network, and inputting the encoded scene feature representation into the robotic arm operation policy network; after fusing the encoded scene feature representation, the other path of the multi-view embedding representation of the scene, and the natural language description of the robotic arm operation task by the robotic arm operation policy network, fine-tuning is completed according to the optimization constraint logic of the robotic arm operation policy network to predict the next operation action of the robotic arm.

[0037] Example 4 A computer-readable storage medium stores a computer program that, when executed by a processor, implements a generalized operation method for a robotic arm based on pre-trained hybrid geometric representations. The method involves constructing a 3D-MOV dataset, which includes object-level 3D point clouds, fine-grained scene-level 3D point clouds, and desktop-level 3D point clouds. A hybrid geometric representation encoder is pre-trained based on the 3D point clouds in the 3D-MOV dataset to obtain a pre-trained hybrid geometric representation encoder. This includes the following steps: performing multi-view rendering preprocessing on 3D point cloud instances to obtain multi-view Euclidean representations; mapping the multi-view Euclidean representations to hyperbolic space to obtain mapped feature representations; optimizing, learning, and constraining the mapped feature representations in hyperbolic space, introducing Top-K neighborhood rank correlation loss for optimization to obtain hyperbolic learned features; returning the hyperbolic learned features to Euclidean space through a logarithmic mapping and using them for reconstruction tasks to complete self-supervised reconstruction task pre-training, resulting in a pre-trained hybrid geometric representation encoder. Hybrid geometric representation encoder; Based on the pre-trained hybrid geometric representation encoder, fine-tuning of the robotic arm operation task is performed to complete the generalization operation of the robotic arm based on the hybrid geometric representation pre-training, including the following steps: The observation point cloud of the robotic arm scene is rendered into RGB images of multiple views through orthogonal projection; the RGB images of each view are segmented and embedded to obtain the scene multi-view embedding representation; the scene multi-view embedding representation is processed by dual-path distribution, one path is input to the pre-trained hybrid geometric representation encoder to obtain the encoded scene feature representation; the other path and the natural language description of the robotic arm operation task are input to the robotic arm operation policy network, and the encoded scene feature representation is input to the robotic arm operation policy network; the robotic arm operation policy network fuses the encoded scene feature representation, the other path scene multi-view embedding representation and the natural language description of the robotic arm operation task, and then performs fine-tuning according to the optimization constraint logic of the robotic arm operation policy network to predict the next operation action of the robotic arm.

[0038] Example 5 A generalization method for robotic arms based on hybrid geometric representation pre-training, see [link to relevant documentation]. Figure 4 A schematic diagram of the 3D multi-view self-supervised pre-training process based on hybrid geometric space representation. The pre-training process of this invention includes the following steps: S1. First, input a 3D point cloud instance (from the 3D-MOV dataset), and then perform multi-view rendering preprocessing on it.

[0039] S1.1 Render the input 3D point cloud into five (top, front, back, left, right) RGB images using orthographic projection.

[0040] S1.2. Perform image patch segmentation on each orthogonal projection image. To achieve self-supervised learning, apply random masking to the image patch of each viewpoint and perform mapping, retaining only the embedded multi-view image patches that are not masked.

[0041] S1.3. The preprocessed multi-view image block embedding representation, CLS embedding, and mask embedding are input together into multiple ViT Blocks to obtain a multi-view Euclidean representation, including CLS embedding. Image patch embedding and mask embedding .

[0042] S2. The multi-view Euclidean representation obtained in step S1 is mapped to a non-Euclidean space (hyperbolic space) for further processing.

[0043] S2.1 Using an exponential map to represent Euclidean features Mapped into hyperbolic space.

[0044] S2.2, Euclidean characteristics Consider it as the tangent vector at the origin of the Lorentz hyperbolic model .

[0045] S2.3, through exponential mapping , tangent vector Mapped to hyperbolic space The hyperbolic feature is obtained above. Including CLS token embedding in hyperbolic space Image patch embedding and Mask embedding .

[0046] Features obtained by representation in Euclidean space are Euclidean features; multi-view Euclidean representation means representation in Euclidean space based on multiple perspectives, that is, representation of multi-view images in Euclidean space is called multi-view Euclidean representation.

[0047] S3, in hyperbolic space Chinese feature representation Optimize, learn, and constrain S3.1, in hyperbolic features Apply Top-K neighborhood rank correlation loss This is to ensure that the characteristic topology (neighborhood order) in Euclidean space is consistent with that in hyperbolic space.

[0048]

[0049] in, and Let represent the Top-K neighborhood rank correlation matrices in Euclidean and hyperbolic spaces, respectively.

[0050] S3.2, in hyperbolic features Apply implied loss By constraining image patch embedding and Mask embedding Embedded in Hyperbolic CLS Within the defined implication cone, structural hierarchy and local-global semantic alignment are enhanced.

[0051] S4. The hyperbolic learned features are returned to Euclidean space through a logarithmic mapping and used for the reconstruction task to complete the self-supervised pre-training.

[0052] S4.1, through logarithmic mapping hyperbolic space Representations learned in Mapping back to Euclidean space yields eigenvectors. .

[0053] S4.2, will It is used for two self-supervised tasks: Intra-view Reconstruction and Inter-view Reconstruction. It jointly optimizes the hybrid geometric representation encoder and completes the pre-training.

[0054]

[0055]

[0056] in, and represents the pixel reconstruction loss within and between viewpoints, respectively, and W and H represent the width and height of the image, respectively. and Let represent the ground truth value and the predicted value of the j-th pixel in the i-th view, respectively. This represents the ground truth value of the j-th pixel from the perspective of the a-th anchor. This represents the predicted value of the j-th pixel from the i-th viewpoint, as seen from the a-th anchor viewpoint.

[0057] See Figure 5 A flowchart illustrating the fine-tuning process of a pre-trained hybrid geometric representation encoder for downstream robotic arm operations. The fine-tuning process of this invention includes the following steps: S001. First, input the scene observation point cloud of the robotic arm and render it as an orthogonal projection image of any number of viewpoints.

[0058] S001.1 Render the input 3D point cloud into three (top, front, and right) RGB images using orthogonal projection.

[0059] S001.2 Preprocessing of multi-view images, including segmenting the images into blocks and embedding mapping representations.

[0060] S002. Introduce a pre-trained hybrid geometric representation encoder for fine-tuning in downstream robotic arm operation tasks.

[0061] S002.1 Input the multi-view embedded representation output by S1 into the hybrid geometric representation encoder and the RoboticView Transformer (RVT) (RVT is an existing multi-view-based robotic arm operation strategy), and also input the natural language description of the robotic arm operation task into the RVT.

[0062] S002.2 The RVT predicts the next operation of the robotic arm, and the optimization and constraint process is consistent with the original RVT.

[0063] See Figure 6 To support pre-training, high-quality 3D datasets were collected from existing data sources to construct the 3D-MOV dataset. The data collection process of this invention includes the following steps: S101. First, 180K point clouds at the individual object level were collected, which were obtained from the Objaverse-XL dataset.

[0064] S102. Collect 6052 fine-grained scene-level point clouds, which are derived from the ScanNet dataset.

[0065] S102.1 Collect 1513 original indoor scenes provided in the ScanNet dataset.

[0066] The indoor scenes in S102.2 and ScanNet are quite complex and large. Directly rendering them as multi-view images makes it difficult to provide fine-grained scene information. Therefore, each indoor scene is divided into 4 parts, resulting in 6052 fine-grained scene point clouds.

[0067] S103. Then, collect 14K desktop scene point clouds, which are derived from the TO-Scene dataset.

[0068] S103.1 Collect 3999 vanilla-level desktop scene point clouds.

[0069] S103.2 Collect 10001 crowd-level point clouds of the desktop scene.

[0070] See Figure 7 The specific steps of the inter-view reconstruction process during pre-training are as follows: S1111: Randomly select one of the five perspectives as the anchor perspective.

[0071] S1112. Perform cross-attention calculation on the image patch representations of the remaining views output by the hybrid geometric representation encoder and the representations of the anchor view (including image patch representations and mask representations).

[0072] S1113. Input the features output by cross-attention into the standard MAE decoder.

[0073] S1114. Output the reconstructed images of the anchor view from the other viewpoints.

[0074] The proposed 3D multi-view representation pre-training HyperMVP based on hybrid geometric space was compared with existing methods such as PolarNet, PerAct, RVT, SAM2Act, and 3D-MVP on 18 tasks of the RLBench benchmark. The comparison results are as follows: Figure 8 As shown, the average success rate of the present invention, 71.1%, is significantly better than other methods.

[0075] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A robotic arm generalization operation method based on hybrid geometric representation pre-training, characterized in that, Includes the following steps: Construct a 3D-MOV dataset, which includes object-level 3D point clouds, fine-grained scene-level 3D point clouds, and desktop-level 3D point clouds; The hybrid geometric representation encoder is pre-trained based on 3D point clouds from the 3D-MOV dataset to obtain a pre-trained hybrid geometric representation encoder, including the following steps: Multi-view rendering preprocessing is performed on 3D point cloud instances to obtain multi-view Euclidean representations; The multi-view Euclidean representation is mapped onto hyperbolic space to obtain the mapped feature representation; The mapped feature representation is optimized, learned, and constrained in hyperbolic space. The optimization introduces Top-K neighborhood rank correlation loss to obtain hyperbolic learned features. The hyperbolic learned features are returned to Euclidean space through logarithmic mapping and used for reconstruction tasks to complete self-supervised reconstruction task pre-training, resulting in a pre-trained hybrid geometric representation encoder. Based on the pre-trained hybrid geometric representation encoder, fine-tuning of the robotic arm operation task is performed to complete the generalization operation of the robotic arm based on the hybrid geometric representation pre-training, including the following steps: The point cloud of the robotic arm scene observation is rendered into RGB images from multiple perspectives through orthogonal projection. The RGB images from each perspective are then segmented and embedded to obtain a multi-view embedded representation of the scene. The scene multi-view embedding representation is processed by dual-path distribution. One path is input to the pre-trained hybrid geometric representation encoder to obtain the encoded scene feature representation. The other path, together with the natural language description of the robotic arm operation task, is input to the robotic arm operation policy network. The encoded scene feature representation is then input to the robotic arm operation policy network. After fusing scene feature representation encoded by the robotic arm operation strategy network, scene multi-view embedding expression, and natural language description of the robotic arm operation task, fine-tuning is completed by following the optimization constraint logic of the robotic arm operation strategy network to predict the next operation action of the robotic arm.

2. The robotic arm generalization operation method based on hybrid geometric representation pre-training according to claim 1, characterized in that, The multi-view rendering preprocessing of the 3D point cloud instance to obtain a multi-view Euclidean representation is specifically as follows: The 3D point cloud in the 3D-MOV dataset is rendered into RGB images with five orthogonal viewpoints through orthogonal projection. The RGB images of each viewpoint are then divided into blocks, randomly masked, and embedded to obtain a multi-view image block embedding representation. The multi-view image patch embedding representation is input into the feature encoding network along with CLS embedding and mask embedding to output a multi-view Euclidean representation. The five orthogonal RGB images are the top, front, back, left, and right views, respectively. The process of mapping the multi-view Euclidean representation to hyperbolic space to obtain the mapped feature representation is as follows: The Euclidean features from each viewpoint are regarded as tangent vectors at the origin of the Lorentz hyperbolic model. The tangent vectors are mapped onto the hyperbolic space through exponential mapping to obtain hyperbolic features, including CLS embedding, image patch embedding and Mask embedding in the hyperbolic space.

3. The robotic arm generalization operation method based on hybrid geometric representation pre-training according to claim 2, characterized in that, The optimization, learning, and constraint of the mapped feature representation in hyperbolic space to obtain hyperbolic learned features are specifically as follows: In hyperbolic space, topological consistency constraint loss and hierarchical structure constraint loss are applied to the hyperbolic features to complete the hyperbolic feature optimization and obtain the optimized hyperbolic features. The topological consistency constraint loss is the Top-K neighborhood rank correlation loss, as shown in the following formula: in, Represents the loss related to the rank of the Top-K neighborhood, where and Let represent the Top-K neighborhood rank correlation matrices in Euclidean space and hyperbolic space, respectively; The hierarchical constraint loss is an implication loss, which constrains the image patch embedding and mask embedding within the implication cone defined by hyperbolic CLS embedding. The hyperbolic learned features are returned to Euclidean space through a logarithmic mapping and used for the reconstruction task, completing the pre-training of the self-supervised reconstruction task and obtaining a pre-trained hybrid geometric representation encoder; specifically: The optimized hyperbolic features are mapped back to Euclidean space by logarithmic mapping to obtain the mapped feature vector. The mapped feature vector is then used to perform a self-supervised reconstruction task to complete the pre-training of the hybrid geometric representation encoder. The self-supervised reconstruction task is shown in the following equation: in, and represents the pixel reconstruction loss within and between viewpoints, respectively, and W and H represent the width and height of the image, respectively. and Let represent the ground truth value and the predicted value of the j-th pixel in the i-th view, respectively. This represents the ground truth value of the j-th pixel from the perspective of the a-th anchor. This represents the predicted value of the j-th pixel from the i-th viewpoint, as seen from the a-th anchor viewpoint.

4. The robotic arm generalization operation method based on hybrid geometric representation pre-training according to claim 1, characterized in that, The self-supervised reconstruction task includes intra-view reconstruction and inter-view reconstruction; The inter-view reconstruction process is specifically as follows: Randomly select one of the multiple rendering perspectives as the anchor perspective, perform cross-attention calculation on the image patch representations of the other perspectives and the representation of the anchor perspective, obtain the calculation result, input the calculation result into the decoder of the standard self-supervised pre-training paradigm framework, and output the reconstructed image of the anchor perspective for the other perspectives.

5. The robotic arm generalization operation method based on hybrid geometric representation pre-training according to claim 1, characterized in that, The specific process of constructing the 3D-MOV dataset is as follows: Collect object-level 3D point clouds from the Objaverse-XL dataset; We collect raw indoor scene point clouds from the ScanNet dataset and segment each raw indoor scene to obtain fine-grained scene-level 3D point clouds. Desktop scene point clouds were collected from the TO-Scene dataset, including vanilla-level desktop scene point clouds and crowd-level desktop scene point clouds.

6. The robotic arm generalization operation method based on hybrid geometric representation pre-training according to claim 1, characterized in that, The RGB images from multiple perspectives are RGB images from three orthogonal perspectives: top, front, and right.

7. The robotic arm generalization operation method based on hybrid geometric representation pre-training according to claim 1, characterized in that, The robotic arm operation strategy network is a multi-view robotic arm operation strategy RVT.

8. A robotic arm generalization operating system based on hybrid geometric representation pre-training, characterized in that, include: The dataset construction module is used to construct a 3D-MOV dataset, which includes object-level 3D point clouds, fine-grained scene-level 3D point clouds, and desktop-level 3D point clouds. The pre-trained encoder module is used to pre-train the hybrid geometric representation encoder based on the 3D point cloud in the 3D-MOV dataset to obtain a pre-trained hybrid geometric representation encoder, including the following steps: Multi-view rendering preprocessing is performed on 3D point cloud instances to obtain multi-view Euclidean representations; The multi-view Euclidean representation is mapped onto hyperbolic space to obtain the mapped feature representation; The mapped feature representation is optimized, learned, and constrained in hyperbolic space to obtain hyperbolic learned features; The hyperbolic learned features are returned to Euclidean space through logarithmic mapping and used for reconstruction tasks to complete self-supervised reconstruction task pre-training, resulting in a pre-trained hybrid geometric representation encoder. The robotic arm generalization operation module is used to perform fine-tuning of robotic arm operation tasks based on the pre-trained hybrid geometric representation encoder, and to complete the robotic arm generalization operation based on the hybrid geometric representation pre-training, including the following steps: The point cloud of the robotic arm scene observation is rendered into RGB images from multiple perspectives through orthogonal projection. The RGB images from each perspective are then segmented and embedded to obtain a multi-view embedded representation of the scene. The scene multi-view embedding representation is processed by dual-path distribution. One path is input to the pre-trained hybrid geometric representation encoder to obtain the encoded scene feature representation. The other path, together with the natural language description of the robotic arm operation task, is input to the robotic arm operation policy network. The encoded scene feature representation is then input to the robotic arm operation policy network. After fusing scene feature representation encoded by the robotic arm operation strategy network, scene multi-view embedding expression, and natural language description of the robotic arm operation task, fine-tuning is completed by following the optimization constraint logic of the robotic arm operation strategy network to predict the next operation action of the robotic arm.

9. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the robotic arm generalization operation method based on hybrid geometric representation pre-training as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the robotic arm generalization operation method based on hybrid geometric representation pre-training as described in any one of claims 1-7.

Images