A method for training a three-dimensional visual perception model and performing a visual task
By performing structured feature extraction and point cloud reconstruction training on the initial encoder, a 3D visual perception model with both geometric structure information and good generalization ability is generated, which solves the problem that robots have difficulty in accurately perceiving the position and shape of target objects, and improves the success rate and robustness of robot operation tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING HUMANOID ROBOTICS INNOVATION CENTER CO LTD
- Filing Date
- 2026-04-22
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, robots have difficulty accurately perceiving the position and shape of target objects, leading to the failure of operation tasks. This is especially true when there is a significant difference between the actual operation scenario and the pre-training scenario, making it easy for point cloud feature extraction to deviate.
By acquiring sample visual data, the initial encoder in the initial 3D visual perception model is used to extract structured features from the sample depth image, generating structured latent features. These features are then combined with the initial decoder for point cloud reconstruction and 3D perception rendering. A self-supervised training signal is constructed to train the initial encoder, generating a target 3D visual perception model that combines geometric structure information and good generalization ability.
It improves the robot's perception accuracy and environmental adaptability in the work environment, and enhances the success rate and robustness of the robot's operation tasks.
Smart Images
Figure CN122435409A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robotics technology, and more specifically, to a method for training a three-dimensional visual perception model and performing visual tasks. Background Technology
[0002] In recent years, with the development of artificial intelligence and sensor technology, robots have been increasingly widely used in people's lives. Among them, robot manipulation is a core technology in the field of robotics, which aims to enable robots to understand operational task instructions and perform environmental perception, grasping planning, and action execution in real-world environments to complete the operational tasks specified in the instructions.
[0003] In related technologies, after acquiring an operational task, the robot uses a depth camera to obtain point cloud data of the work scene, extracts features from each point in the point cloud using a point cloud encoder, and combines this with the robot's proprioceptive state to generate the motion trajectory of the robot's actuators, controlling the robot to execute the operational task along the aforementioned motion trajectory. However, the features extracted directly from the point cloud encoder in these technologies are not accurate enough, making it difficult for the robot to accurately perceive the position and shape of the target object, thus leading to the failure of the operational task.
[0004] Therefore, how to provide an accurate 3D visual perception model to improve the success rate and robustness of robot operation tasks is a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0005] The purpose of this application is to address the shortcomings of the prior art by providing a method for training a three-dimensional visual perception model and executing visual tasks, thereby improving the robot's perception accuracy and environmental adaptability in the work scene, and thus improving the success rate and robustness of the robot's operation tasks, and solving the problem in related technologies that robots have difficulty accurately perceiving the position and shape of target objects.
[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a method for training a three-dimensional visual perception model, the method comprising: Acquire sample visual data, which includes: sample color image and sample depth image; The initial encoder in the initial 3D visual perception model is used to extract structured features from the sample depth image to obtain structured latent features; The initial decoder in the initial 3D visual perception model is used to reconstruct the point cloud of the structured latent features to obtain the reconstructed target point cloud data. Based on the target point cloud data, perform 3D perception rendering to generate a rendering feature map and a rendering depth map; The initial encoder is trained based on the sample color image, the sample depth image, the rendered feature map, and the rendered depth map until a preset stopping condition is met. The target encoder that meets the preset stopping conditions is identified as the target 3D visual perception model.
[0007] In some embodiments, the initial encoder includes a point cloud encoder and a latent representation encoder, wherein the initial encoder in the initial 3D visual perception model extracts structured latent features based on the structured features of the sample depth image to obtain structured latent features, including: The point cloud encoder is used to extract features from the original point cloud data corresponding to the sample depth image to obtain a sparse feature point cloud. The latent representation encoder is used to perform 3D perception based on the sparse feature point cloud to generate the structured latent features.
[0008] In some embodiments, the feature data includes: global features and local features; the latent representation encoder includes: a backbone network, an attention pooling-based aggregation module, a splicing module, and a structured prediction module; The step of using the latent representation encoder to perform 3D perception based on the coordinates and features of the sparse feature point cloud to generate the structured latent features includes: The backbone network is used to extract local features of each point in the sparse feature point cloud; The aggregation module is used to aggregate the local features of each point to obtain the global features; The stitching module is used to stitch together the local features and the global features of each point to obtain the joint features of each point. The structured prediction module performs structured prediction based on the joint features of each point to obtain the structured latent features.
[0009] In some embodiments, the structured prediction module includes a perceptron and a sampling module, wherein the structured prediction module performs structured prediction based on the joint features of each point to obtain the structured latent features, including: The Gaussian distribution parameters of each point are predicted by using the perceptron to analyze the joint features of each point, thereby obtaining the Gaussian distribution parameters of each point. The sampling module is used to sample the Gaussian distribution parameters of each point to obtain the structured latent features.
[0010] In some embodiments, the Gaussian distribution parameters include a first Gaussian distribution parameter in the feature dimension and a second Gaussian distribution parameter in the coordinate dimension. The step of using the perceptron to predict the Gaussian distribution parameters of the joint features of each point to obtain the Gaussian distribution parameters of each point includes: Two perceptrons are used to predict the Gaussian distribution parameters of the joint features of each point in terms of feature dimension and coordinate dimension, respectively, to obtain the first Gaussian distribution parameter and the second Gaussian distribution parameter of each point.
[0011] In some embodiments, the structured latent features include: latent features in the feature dimension and latent features in the coordinate dimension; the step of sampling the Gaussian distribution parameters of each point using the sampling module to obtain the structured latent features includes: The sampling module uses the reparameterization technique to sample the first Gaussian distribution parameters and the second Gaussian distribution parameters of each point as latent variables, thereby obtaining the latent features of each point in the feature dimension and the latent features in the coordinate dimension.
[0012] In some embodiments, the step of performing 3D perception rendering based on the target point cloud data to generate a rendering feature map and a rendering depth map includes: Geometric reasoning is performed based on the target point cloud data to obtain the Gaussian sputtering properties of the three-dimensional Gaussian points corresponding to each point in the target point cloud data. Three-dimensional Gaussian rendering is performed based on the Gaussian sputtering properties of the three-dimensional Gaussian points to generate the rendering feature map and the rendering depth map.
[0013] In some embodiments, training the initial encoder based on the sample color image, the sample depth image, the rendered feature map, and the rendered depth map includes: The rendering alignment loss is determined based on the sample color image, the sample depth image, the rendered feature map, and the rendered depth map; The reconstruction loss is determined based on the original point cloud data and the target point cloud data; The divergence loss is determined based on the Gaussian distribution parameters and standard Gaussian distribution of each point; The initial encoder is trained based on the rendering alignment loss, the reconstruction loss, and the divergence loss until the preset stopping condition is met.
[0014] In some embodiments, determining the reconstruction loss based on the original point cloud data and the target point cloud data includes: The auxiliary reconstruction loss is calculated based on the coordinates and color of each original point cloud in the original point cloud data and the coordinates and color of each target point cloud in the target point cloud data. The potential reconstruction loss is calculated based on the coordinates of each original point cloud, the coordinates of each target point cloud, the local features of each original point cloud, and the local features of each target point cloud. The reconstruction loss is determined based on the auxiliary reconstruction loss and the potential reconstruction loss.
[0015] In some embodiments, determining the rendering alignment loss based on the sample color image, the sample depth image, the rendered feature map, and the rendered depth map includes: The rendered feature map is decoded to obtain the target semantic feature map and the target color feature map; The first loss is calculated based on the target semantic feature map and the original semantic feature map corresponding to the sample color image; The second loss is calculated based on the sample depth image and the rendered depth map; The third loss is calculated based on the target color feature map and the original color feature map corresponding to the sample color image; The rendering alignment loss is determined based on the first loss, the second loss, and the third loss.
[0016] Secondly, one embodiment of this application provides a method for performing a visual task, the method comprising: Based on the acquired natural language instructions, acquire image data captured by the robot in the current working scenario; Obtain the point cloud data corresponding to the image data; The structured latent features corresponding to the point cloud data are generated using a three-dimensional visual perception model; the three-dimensional visual perception model is a target three-dimensional visual perception model generated using any of the methods described in the first aspect above. Based on the structured latent features and the robot's state information, a target action is generated; Control the robot to perform the target action in order to complete the target task indicated by the natural language command.
[0017] In some embodiments, the three-dimensional visual perception model includes: a point cloud encoder and a latent representation encoder, wherein generating structured latent features corresponding to the point cloud data using the three-dimensional visual perception model includes: The point cloud encoder is used to extract features from the point cloud data to obtain a sparse feature point cloud. The latent representation encoder is used to perform 3D perception based on the sparse feature point cloud to generate the structured latent features.
[0018] Thirdly, another embodiment of this application provides a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in either the first or second aspect above.
[0019] Fourthly, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of the method as described in either the first or second aspect above.
[0020] The beneficial effects of this application are as follows: This application extracts structured features from sample depth images using an initial encoder to generate structured latent features, and then performs point cloud reconstruction and 3D perception rendering based on these structured latent features to construct a self-supervised training signal for training the initial encoder. The trained target 3D visual perception model can extract structured latent features with geometric structure information and good generalization ability. The robot generates action commands based on these structured latent features with higher accuracy, thereby improving the success rate and robustness of robot operation tasks. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A flowchart illustrating a training method for a three-dimensional visual perception model according to an embodiment of this application is shown. Figure 2 A schematic diagram of a process for determining structured potential features according to an embodiment of this application is shown; Figure 3 A schematic diagram of the training process of an initial encoder provided in an embodiment of this application is shown; Figure 4 A flowchart illustrating a method for performing a visual task according to an embodiment of this application is shown; Figure 5 A schematic diagram of the structure of a training device for a three-dimensional visual perception model provided in an embodiment of this application is shown; Figure 6 A schematic diagram of the structure of a vision task execution device provided in an embodiment of this application is shown; Figure 7This application shows a schematic diagram of the structure of a robot according to an embodiment of the present application; Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.
[0024] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0025] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.
[0026] The technical solution provided in this application is applicable to task scenarios requiring robot visual perception and manipulation, such as home service scenarios, industrial manufacturing scenarios, and medical rehabilitation scenarios. In these scenarios, the robot needs to execute actions corresponding to the user's natural language commands in a real-world environment, such as grasping, placing, and opening / closing drawers.
[0027] Currently, when robots perform operational tasks, they generally use point cloud-based 3D visual perception methods.
[0028] However, during the pre-training process of 3D visual perception using related technologies, there is a problem that robots struggle to accurately perceive the spatial position and shape of objects. Furthermore, when the actual operational scenario differs significantly from the pre-training scenario, the point cloud features extracted using these technologies are prone to deviation. Therefore, when using related technologies for 3D visual perception, the generated robot movements are not accurate enough, leading to a low success rate in the robot's task execution.
[0029] Based on this, this application provides a training method for a 3D visual perception model and a method for executing visual tasks. In the training phase, this application iteratively trains an initial encoder based on sample visual data, enabling the trained target encoder to generate structured latent features that possess both geometric structural information and good generalization ability. In the execution phase, the trained 3D visual perception model processes the point cloud data of the current work scene to generate structured latent features of the point cloud data, and combines this with the robot's state information to generate the target action. Because the structured latent features obtained based on the trained 3D visual perception model retain both the geometric structural information of the current work scene and have good generalization ability, the target action generated based on the structured latent features and the robot's state information is more accurate, thereby improving the success rate and robustness of the robot's operation tasks.
[0030] It should be noted that the above description of application scenarios is only one of the multiple use scenarios provided in this application. Those skilled in the art should understand that when the robot motion generation method and the robot provided in this application are applied to other use scenarios, their implementation methods and technical effects are similar.
[0031] Figure 1 A flowchart illustrating a training method for a three-dimensional visual perception model according to an embodiment of this application is shown. Figure 1 As shown, the training method for the 3D visual perception model provided in this application may include steps 110-160, wherein: Step 110: Obtain sample visual data, which includes: sample color image and sample depth image.
[0032] The sample visual data includes sample color images and sample depth images for the same scene. The sample color images are used to record appearance information such as color, texture, and lighting of objects in the scene. Each pixel value in the sample depth image represents the distance from that point to the camera plane, and is used to record the three-dimensional spatial information of objects in the scene.
[0033] The sample visual data can be collected by the robot in real working scenarios using a red-green-blue-depth (RGB-D) camera; or it can be generated in advance using a physical simulation engine; or it can be obtained directly from publicly available red-green-blue-depth (RGB-D) datasets, etc.
[0034] Step 120: Use the initial encoder in the initial 3D visual perception model to extract structured features from the sample depth image to obtain structured latent features.
[0035] In some embodiments, the sample depth image is first converted into raw point cloud data, and then the raw point cloud data is input into the initial encoder. The initial encoder performs multi-level feature extraction and transformation on the raw point cloud to extract structured latent features that simultaneously contain semantic information and spatial structure information.
[0036] In some embodiments, the initial encoder includes a point cloud encoder and a latent representation encoder. Step 120 may include: using the point cloud encoder to extract features from the original point cloud data corresponding to the sample depth image to obtain a sparse feature point cloud; and using the latent representation encoder to perform three-dimensional perception based on the sparse feature point cloud to generate structured latent features.
[0037] Among them, the point cloud encoder can be a PTv3 encoder. Used for raw point cloud data Perform downsampling and feature extraction from the raw point cloud data. Representative key points are selected from the data, and a sparse feature point cloud is constructed based on these key points. sparse feature point cloud The number of points contained in it is far less than that in the original point cloud data. The number of points contained in the cloud. Furthermore, the point cloud encoder extracts features from each keypoint, ensuring that each point in the sparse feature point cloud contains both coordinate and feature information.
[0038]
[0039] in, It contains M sparse feature points, each of which is represented by three-dimensional coordinates. and high-dimensional features composition.
[0040] The method provided in this application inserts a latent representation encoder between the output of the initial encoder and the initial decoder for processing sparse feature point cloud data. Regularization is performed. The latent representation encoder can be a PL-VAE, used for 3D perceptual modeling of sparse feature point clouds by capturing sparse feature point cloud data. Spatial relationships between midpoints generate structured latent features .
[0041] Step 130: Use the initial decoder in the initial 3D visual perception model to reconstruct the point cloud of the structured latent features to obtain the reconstructed target point cloud data.
[0042] The target point cloud data is the point cloud output by the initial decoder after reconstructing the structured latent features. Each point in the target point cloud data contains coordinate information and feature information.
[0043] The structured latent features are input into the initial decoder, which upsamples these features, gradually increasing the number of points to reconstruct a dense target feature cloud from a sparse point cloud. Because the structured latent features contain both semantic and spatial structural information, the reconstructed target point cloud data retains the geometric structure and details of the original point cloud data.
[0044] In some embodiments, the initial decoder can be a PTv3 decoder. Structured latent features Input the initial decoder to obtain the reconstructed target point cloud features. .
[0045]
[0046] Step 140: Perform 3D perception rendering based on the target point cloud data to generate a rendering feature map and a rendering depth map.
[0047] The rendered feature map provides semantic feature information for each location on the two-dimensional image plane, while the rendered depth map provides spatial distance information for each location on the two-dimensional plane.
[0048] 3D perceptual rendering simulates the imaging principle of a camera, projecting 3D point cloud data onto a 2D plane based on preset camera intrinsic and extrinsic parameters. For each pixel location after projection, a depth map is calculated based on the distance between the 3D coordinates of each point in the 3D point cloud data and the camera center, generating a depth rendering map. The rendered depth map provides spatial distance information for each location. Furthermore, based on the semantic features of each point after projection, a feature vector is generated for each pixel location to obtain a rendered feature map. The rendered feature map provides a semantic feature description of each location on the 2D image plane.
[0049] In some embodiments, step 140 may include: performing geometric reasoning based on the target point cloud data to obtain the Gaussian sputtering attribute of the three-dimensional Gaussian point corresponding to each point in the target point cloud data; performing three-dimensional Gaussian rendering based on the Gaussian sputtering attribute of the three-dimensional Gaussian point to generate a rendering feature map and a rendering depth map.
[0050] A 3D Gaussian point is a point represented by a Gaussian distribution, and each point has attributes such as position, shape, size, and opacity. Gaussian sputtering properties are used to describe the geometric parameters of a 3D Gaussian point, including position, rotation, scaling, opacity, and color. Gaussian sputtering properties determine the shape, orientation, and transparency of a 3D Gaussian point in space.
[0051] Geometric Reasoning Module Based on target point cloud data Geometric reasoning is performed on the 3D coordinates and semantic features of each point to obtain the Gaussian sputtering attribute G of the 3D Gaussian point for each point:
[0052] The geometric reasoning method described above expands each point in the target point cloud data from a discrete, sizeless point into a continuous Gaussian distribution region, allowing the points to overlap and merge with each other, thus more smoothly representing the geometric structure in the scene.
[0053] 3D Gaussian rendering is the process of projecting 3D Gaussian points onto a 2D image plane. Specifically, based on preset camera parameters, all 3D Gaussian points are projected onto the 2D image plane. During the projection process, the depth value of each pixel is calculated based on the distance information of the 3D Gaussian points to obtain a rendering depth map. The semantic features of the 3D Gaussian points are then weighted and fused to generate a feature vector for each pixel location, resulting in a rendering feature map.
[0054] In one embodiment of this application, a pure feature sputtering strategy is employed, rendering only feature information and depth information point by point, without rendering color. Given a camera viewpoint v, feature sputtering is performed:
[0055] in, To render the feature map, To render the depth map, It is a Gaussian rasterization renderer.
[0056] Step 150: Train the initial encoder based on the sample color image, sample depth image, rendered feature map, and rendered depth map until the preset stopping condition is met.
[0057] The preset stopping conditions can be such as the loss function falling below a preset threshold or reaching the maximum number of training rounds.
[0058] Obtain sample color and depth images from the visual samples as ground truth during training. Obtain rendered feature maps and rendered depth maps as the prediction results output by the initial encoder. Calculate the difference between the ground truth and the prediction results, and update the parameters of the initial encoder based on the difference calculation results. Repeat the above process until the difference between the ground truth and the prediction results is less than a preset threshold, or until the difference between the ground truth and the prediction results no longer decreases after multiple consecutive training epochs, or until a preset maximum number of training epochs is reached.
[0059] Step 160: Determine the target encoder that meets the preset stopping conditions as the target 3D visual perception model.
[0060] The target encoder after training is identified as the target 3D visual perception model. The target 3D visual perception model can extract structured potential features from point cloud data that have both geometric structure information and good generalization ability.
[0061] The training method for the 3D visual perception model provided in this application involves acquiring sample visual data, including sample color images and sample depth images; using the initial encoder in the initial 3D visual perception model to extract structured features from the sample depth images to obtain structured latent features; using the initial decoder in the initial 3D visual perception model to reconstruct point clouds from the structured latent features to obtain reconstructed target point cloud data; performing 3D perception rendering based on the target point cloud data to generate a rendered feature map and a rendered depth map; training the initial encoder based on the sample color images, sample depth images, rendered feature maps, and rendered depth maps until a preset stopping condition is met; and determining the target encoder that meets the preset stopping condition as the target 3D visual perception model. In other words, this application uses the initial encoder to extract structured features from the sample depth images to generate structured latent features, and then performs point cloud reconstruction and 3D perception rendering based on the structured latent features to construct a self-supervised training signal for training the initial encoder. The trained target 3D visual perception model can extract structured latent features with geometric structure information and good generalization ability. The robot generates action commands based on these structured latent features with higher accuracy, thereby improving the success rate and robustness of robot operation tasks.
[0062] Figure 2 This application provides a schematic diagram of a process for determining structured potential features according to an embodiment of the present application. Figure 2 As shown, the latent representation encoder includes: a backbone network, an attention pooling-based aggregation module, a splicing module, and a structured prediction module; the process for determining structured latent features may include: steps 121-124, wherein: Step 121: Use a backbone network to extract local features of each point in the sparse feature point cloud.
[0063] Backbone network for sparse feature point clouds For a point i in the given space, search for neighboring points in its three-dimensional neighborhood. By aggregating the coordinate differences and feature differences of neighboring points, generate local features that reflect the local geometry of point i. : =
[0064] in, The backbone network encodes the geometric information such as the local shape and orientation around each point. MLP stands for Multilayer Perceptron, which is used to perform nonlinear transformations and mappings on the features.
[0065] Step 122: Use the aggregation module to aggregate the local features of each point to obtain the global features.
[0066] The aggregation module uses symmetric aggregation functions (such as average pooling, max pooling, or attention pooling) to aggregate sparse feature point clouds. Local features of all points are aggregated into global features. : .
[0067] Agg is an aggregation module based on attention pooling. Global features provide a holistic description of the entire point cloud scene, including high-level semantic information such as the distribution of object categories and spatial layout within the scene.
[0068] Step 123: Use the stitching module to stitch together the local and global features of each point to obtain the joint features of each point.
[0069] For sparse feature point clouds For point i in the image, the local features of point i are combined using the stitching module cat. With global features By concatenating the features, we obtain the joint features of point i. :
[0070] Through the above splicing operation, the joint features of each point contain both the local geometric information of that point and the global context information of the entire point cloud scene, making the features between points interconnected.
[0071] Step 124: Use the structured prediction module to perform structured prediction based on the joint features of each point to obtain structured latent features.
[0072] The structured prediction module transforms and maps the joint features of each point to output structured latent features for each point. These structured latent features contain the 3D coordinates and semantic features of each point.
[0073] In some embodiments, the structured prediction module includes a perceptron and a sampling module. Step 124 may include: using the perceptron to predict the Gaussian distribution parameters of the joint features of each point to obtain the Gaussian distribution parameters of each point; and using the sampling module to sample the Gaussian distribution parameters of each point to obtain structured latent features.
[0074] The joint features of each point are input into the perceptron. The perceptron processes the joint features through multiple nonlinear transformations and outputs Gaussian distribution parameters. These Gaussian distribution parameters include the mean and standard deviation. A larger standard deviation indicates lower accuracy in the perceptron's prediction of that point, while a smaller standard deviation indicates higher accuracy.
[0075] The predicted Gaussian distribution parameters for each point are input into the sampling module. The sampling module first randomly extracts a noise value from the standard Gaussian distribution, and then calculates the sampled value based on the mean and standard deviation of each point. The sampled value is used as the structured latent feature of that point.
[0076] Through the above sampling process, the structured latent features of each point are sampling results with a certain degree of randomness. This randomness enhances the robustness of the initial encoder to noise and scene changes, and improves the generalization ability of the initial encoder.
[0077] In some embodiments, the Gaussian distribution parameters include a first Gaussian distribution parameter in the feature dimension and a second Gaussian distribution parameter in the coordinate dimension. Two perceptrons are used to predict the Gaussian distribution parameters in the feature dimension and coordinate dimension for the joint features of each point, respectively, to obtain the first and second Gaussian distribution parameters for each point.
[0078] Among them, the two sensors can be independent first sensors. Second sensor Output the mean and standard deviation of the feature distribution and the coordinate distribution, respectively.
[0079] Still focusing on sparse feature point clouds Taking the prediction of feature dimensions for point i in the first perceptron as an example: Joint features of point i Predicting the feature dimensions yields the parameters of the first Gaussian distribution used to describe the feature distribution at point i:
[0080] The parameters of the first Gaussian distribution include: the characteristic mean. and characteristic standard deviation This reflects the uncertainty estimation of the features of point i by the first perceptron.
[0081] Still focusing on sparse feature point clouds Taking the prediction of coordinate dimensions of point i in the second perceptron as an example: Joint features of point i By predicting the coordinate dimensions, we obtain the second Gaussian distribution parameters used to describe the coordinate distribution of point i:
[0082] The second Gaussian distribution parameters include: coordinate mean. and coordinate standard deviation This reflects the uncertainty estimate of the position of point i by the second perceptron.
[0083] In some embodiments, a sampling module uses a reparameterization technique to sample the first Gaussian distribution parameters and the second Gaussian distribution parameters of each point as latent variables, thereby obtaining the latent features of each point in the feature dimension. and latent features of coordinate dimensions .
[0084] The sampling module randomly selects a noise value from the standard Gaussian distribution. Randomly selecting a noise value introduces some volatility into the sampling results. Subsequently, using a reparameterization technique, the sampling module processes the parameters of the first Gaussian distribution with the randomly selected noise value: the mean and standard deviation of the first Gaussian distribution parameters are multiplied by the noise value, and the results are summed to obtain the sampled feature dimensions and latent features. Furthermore, the sampling module, using a reparameterization technique, processes the second Gaussian distribution parameters with the same randomly selected noise value to obtain the latent features of the sampled coordinate dimensions. .
[0085] In other words, structured latent features The calculation method can be:
[0086] Where M represents the number of sparse point cloud data points, through the above sampling process, each point obtains latent features in both the feature dimension and the coordinate dimension. The latent features in the feature dimension reflect the semantic attributes of the point, while the latent features in the coordinate dimension reflect its spatial location. (Compared to the original sparse feature point cloud) In comparison, structured latent features More compact and closer to the Gaussian prior.
[0087] In some embodiments, step 130 may include: processing the sampled structured latent features based on the latent representation encoder. Reconstruction yields the reconstructed sparse feature point cloud data. Furthermore, the initial decoder processes the reconstructed sparse feature point cloud data. Perform point cloud reconstruction to obtain the reconstructed target point cloud data. .
[0088]
[0089] Figure 3 This illustration shows a schematic diagram of the training process of an initial encoder according to an embodiment of this application, as follows: Figure 3 As shown, the initial encoder training process may include steps 151-154, wherein: Step 151: Determine the rendering alignment loss based on the sample color image, sample depth image, rendering feature map, and rendering depth map.
[0090] Based on the sample color image and the rendered feature image, feature differences are determined; based on the sample depth image and the rendered depth image, depth differences are determined. Rendering alignment loss is then determined based on the feature differences and depth differences. The rendering alignment loss reflects the difference between the rendered result of the 3D visual perception model and the sample scene. A smaller rendering alignment loss indicates that the structured latent features extracted by the 3D visual perception model can more accurately reconstruct the appearance, geometry, and semantic information in the scene.
[0091] In some embodiments, step 151 may include: decoding the rendered feature map to obtain a target semantic feature map and a target color feature map; calculating a first loss based on the target semantic feature map and the original semantic feature map corresponding to the sample color image; calculating a second loss based on the sample depth image and the rendered depth map; calculating a third loss based on the target color feature map and the original color feature map corresponding to the sample color image; and determining a rendering alignment loss based on the first loss, the second loss, and the third loss.
[0092] Among them, the feature map will be rendered. The input is fed to two projectors to decode and obtain the target semantic feature map. and target color feature map :
[0093] in, The first projector is used to render the feature map. Decoding generates target semantic feature map , As a second projector, it is used to render feature maps. Decoding generates target color feature map .
[0094] Rendering alignment loss The calculation method can be:
[0095] Where V is the color image of the sample from the Vth viewpoint. , , These are preset constant values, for example , , =0.1. For the sample color image of the v-th viewpoint, This is the target color feature map from the v-th viewpoint. Let be the error between the sample color image and the target color feature image at the i-th viewpoint. This is the sample depth map for the v-th viewpoint. Render the depth map for the v-th viewpoint. denoted as the error between the sample depth map and the rendered depth map of the v-th viewpoint. This is the original semantic feature map of the v-th viewpoint. This is the target semantic feature map from the v-th viewpoint. Let be the error between the original semantic feature map and the target semantic feature map at the v-th viewpoint.
[0096] Step 152: Determine the reconstruction loss based on the original point cloud data and the target point cloud data.
[0097] Reconstruction loss reflects the ability of a 3D perception vision model to recover target point cloud data from structured latent features. The smaller the reconstruction loss, the more complete the original point cloud information is preserved by the structured latent features, and the stronger the ability to recover target point cloud data from the structured latent features.
[0098] In some embodiments, step 152 may include: calculating an auxiliary reconstruction loss based on the coordinates and color of each original point cloud in the original point cloud data and the coordinates and color of each target point cloud in the target point cloud data; calculating a potential reconstruction loss based on the coordinates of each original point cloud, the coordinates of each target point cloud, the local features of each original point cloud, and the local features of each target point cloud; and determining the reconstruction loss based on the auxiliary reconstruction loss and the potential reconstruction loss.
[0099] Auxiliary reconstruction loss The calculation method can be:
[0100] Where S is the number of points contained in the original point cloud data. Let i be the original coordinates of the i-th point. Let i be the predicted coordinates of the i-th point. Let be the error between the predicted coordinates and the original coordinates of the i-th point. The original color of the i-th point. The predicted color for the i-th point. Let be the error between the original color and the predicted color of the i-th point.
[0101] Potential reconstruction losses Potential reconstruction loss is used to measure the degree of information loss in structured latent features during compression and decompression. The calculation method can be:
[0102] Where M is the number of sparse feature point cloud data, and the value of M is less than S. , These are preset weights used to balance coordinate reconstruction and feature reconstruction. The original features of the i-th point, For the reconstructed features of the i-th point, Let be the error between the original features and the reconstructed features of the i-th point.
[0103] In some embodiments, It can be 1, It can be 0.1.
[0104] Step 153: Determine the divergence loss based on the Gaussian distribution parameters and standard Gaussian distribution for each point.
[0105] Divergence loss forces the Gaussian distribution predicted by the 3D visual perception model to converge towards the standard Gaussian distribution, thereby enhancing the generalization ability of the 3D visual perception model.
[0106] Divergence loss The calculation method can be:
[0107] in, Used to measure the degree of difference between two distributions. is the predicted distribution of the feature dimension, and is the Gaussian distribution of the predicted i-th point in the feature space. It is a prediction of the coordinate dimension, and it is the Gaussian distribution of the predicted i-th point in the coordinate space. It is a standard Gaussian distribution, a fixed reference distribution with a mean of 0 and a standard deviation of 1, used to constrain the distribution predicted by the model.
[0108] Step 154: Train the initial encoder based on the rendering alignment loss, reconstruction loss, and divergence loss until the preset stopping condition is met.
[0109] The rendering alignment loss, reconstruction loss, and divergence loss are weighted and summed to obtain the total loss. The initial encoder is then trained based on the total loss until a preset stopping condition is met.
[0110] In the embodiments of this application, a total loss function is determined based on rendering alignment loss, potential reconstruction loss, auxiliary reconstruction loss, and divergence loss. The initial encoder is then trained based on this total loss function until a preset stopping condition is met. Total Loss Function The calculation method can be:
[0111] in, Here, is the weighted annealing scheduling parameter, and t is the current training iteration round. In the embodiments of this application, The initial value is 1, gradually decreasing to 0.1. This method of adjusting the weight annealing scheduling parameters during model training allows for a higher weighting of the auxiliary reconstruction loss during the early training phase when rendering features are immature, enabling the 3D vision reconstruction model to quickly learn basic point cloud reconstruction capabilities. As training stabilizes, the weighting of the auxiliary reconstruction loss is gradually reduced, while the weighting of the rendering alignment loss is increased, allowing the 3D vision reconstruction model to learn visual representations with generalization capabilities.
[0112] Figure 4 A flowchart illustrating a method for performing a vision task according to an embodiment of this application is shown. Figure 4 As shown, the method for performing a visual task provided in this application may include steps 210-250, wherein: Step 210: Based on the acquired natural language instructions, acquire the image data captured by the robot in the current working scene.
[0113] Natural language commands can be input by users to the robot through voice input, text input, or other human-computer interaction interfaces, describing specific tasks that the robot needs to perform by executing physical operations. For example, natural language commands could be "go to the end of the corridor," "unplug the plug," "pick up the water glass," or "hammer the nail."
[0114] After receiving natural language commands, the robot begins data acquisition. Specifically, the robot uses depth cameras fixedly mounted on its head, robotic arm, or wrist to capture images of the current work scene, obtaining image data of that scene. The image data can be red-green-blue-depth (RGB-D) images or depth images.
[0115] Step 220: Obtain the point cloud data corresponding to the image data.
[0116] Point cloud data is a dataset representing the three-dimensional spatial information of the surface of an object in a robot's working scene, acquired from depth images. Point cloud data contains multiple points, each with three-dimensional coordinates (x, y, z), and can be supplemented with color features (r, g, b), normal features, texture features, etc.
[0117] Step 230: Use a 3D visual perception model to generate structured latent features corresponding to the point cloud data.
[0118] The three-dimensional visual perception model adopts the above Figures 1-3 The target 3D visual perception model generated by any method.
[0119] In some embodiments, the three-dimensional visual perception model includes a point cloud encoder and a latent representation encoder. Step 230 may include: using the point cloud encoder to extract features from the point cloud data to obtain a sparse feature point cloud; and using the latent representation encoder to perform three-dimensional perception based on the sparse feature point cloud to generate structured latent features.
[0120] A sparse feature extractor downsamples the point cloud data to select representative keypoints, forming a sparse feature point cloud. Local features are then extracted from each keypoint, and the global features of the sparse feature point cloud are obtained through aggregation. The global features are then concatenated with the local features of each keypoint to obtain the joint features of each keypoint, which serve as the feature data. This feature data is then input into an implicit regularization network to generate structured latent features that retain global geometric information and possess strong generalization capabilities.
[0121] Step 240: Generate the target action based on the structured latent features and the robot's state information.
[0122] The robot's state information is the robot's own proprioceptive information, including but not limited to: the angles of each joint, the pose (position and orientation) of the end effector (gripper, manipulator) in three-dimensional space, and the opening and closing degree of the gripper / the angle of the manipulator's finger joints, etc.
[0123] The target motion is the control command generated by the robot and sent to the underlying actuator to drive the robot to complete specific physical operations. The target motion includes, but is not limited to: the target position and orientation of the end effector, the gripping force or opening and closing degree of the gripper / manipulator, and the motion trajectory of each joint of the manipulator.
[0124] Policy Network Receive structured latent features Robot status information Generate target action :
[0125] Step 250: Control the robot to perform the target action to complete the target task indicated by the natural language command.
[0126] The robot performs corresponding operations based on the target action, such as moving the robotic arm to the target position, closing the gripper / manipulator to grasp an object, rotating the wrist to flip the object, and translating backward to unplug a plug. This application uses a pre-trained 3D visual perception model based on point cloud data to obtain structured latent features containing semantic and spatial structural information. The target action is generated based on the structured latent features and the robot's state information, improving the accuracy of the target action and thus increasing the success rate and execution precision of the robot in executing target tasks indicated by natural language commands.
[0127] Based on the same inventive concept, this application also provides a training device for a three-dimensional visual perception model corresponding to the training method for a three-dimensional visual perception model. Since the principle of the device in this application is similar to the training method for a three-dimensional visual perception model described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0128] Figure 5 This illustration shows a structural schematic diagram of a training device for a three-dimensional visual perception model provided in an embodiment of this application, as shown below. Figure 5 As shown, the device includes: an acquisition module 310, a reconstruction module 320, a generation module 330, a training module 340, and a determination module 350; wherein: The acquisition module 310 is used to acquire sample visual data, which includes sample color images and sample depth images. The initial encoder in the initial 3D visual perception model is used to extract structured features from the sample depth images to obtain structured latent features. The reconstruction module 320 is used to reconstruct the point cloud of the structured latent features using the initial decoder in the initial 3D visual perception model, so as to obtain the reconstructed target point cloud data. The generation module 330 is used to perform 3D perception rendering based on the target point cloud data, and generate a rendering feature map and a rendering depth map. The training module 340 is used to train the initial encoder based on the sample color image, sample depth image, rendered feature map and rendered depth map until a preset stopping condition is met. The determination module 350 is used to determine the target encoder as the target three-dimensional visual perception model when the preset stopping conditions are met.
[0129] In one possible implementation, the initial encoder includes a point cloud encoder and a latent representation encoder. The acquisition module 310 is further configured to use the point cloud encoder to extract features from the original point cloud data corresponding to the sample depth image to obtain a sparse feature point cloud; and to use the latent representation encoder to perform three-dimensional perception based on the sparse feature point cloud to generate structured latent features.
[0130] In one possible implementation, the feature data includes global features and local features; the latent representation encoder includes a backbone network, an attention pooling-based aggregation module, a stitching module, and a structured prediction module. The acquisition module 310 is specifically used to: extract local features of each point in the sparse feature point cloud using the backbone network; aggregate the local features of each point using the aggregation module to obtain global features; stitch together the local and global features of each point using the stitching module to obtain joint features of each point; and perform structured prediction based on the joint features of each point using the structured prediction module to obtain structured latent features.
[0131] In one possible implementation, the structured prediction module includes a perceptron and a sampling module. The acquisition module 310 is specifically used to: use the perceptron to predict the Gaussian distribution parameters of the joint features of each point to obtain the Gaussian distribution parameters of each point; and use the sampling module to sample the Gaussian distribution parameters of each point to obtain structured latent features.
[0132] In one possible implementation, the Gaussian distribution parameters include a first Gaussian distribution parameter in the feature dimension and a second Gaussian distribution parameter in the coordinate dimension. The acquisition module 310 is specifically used to use two perceptrons to predict the Gaussian distribution parameters in the feature dimension and coordinate dimension of the joint features of each point, respectively, to obtain the first Gaussian distribution parameter and the second Gaussian distribution parameter of each point.
[0133] In one possible implementation, the structured latent features include: latent features in the feature dimension and latent features in the coordinate dimension; the acquisition module 310 is specifically used to: use the sampling module to sample the latent variables of the first Gaussian distribution parameter and the second Gaussian distribution parameter of each point according to the reparameterization technique, so as to obtain the latent features of each point in the feature dimension and the latent features in the coordinate dimension.
[0134] In one possible implementation, the generation module 330 is specifically used to perform geometric reasoning based on the target point cloud data to obtain the Gaussian sputtering attribute of the three-dimensional Gaussian point corresponding to each point in the target point cloud data; and to perform three-dimensional Gaussian rendering based on the Gaussian sputtering attribute of the three-dimensional Gaussian point to generate a rendering feature map and a rendering depth map.
[0135] In one possible implementation, the training module 340 is specifically used to determine the rendering alignment loss based on the sample color image, sample depth image, rendering feature map, and rendering depth map; determine the reconstruction loss based on the original point cloud data and the target point cloud data; determine the divergence loss based on the Gaussian distribution parameters and standard Gaussian distribution of each point; and train the initial encoder based on the rendering alignment loss, reconstruction loss, and divergence loss until a preset stopping condition is met.
[0136] In one possible implementation, the determining module 350 is specifically used to calculate the auxiliary reconstruction loss based on the coordinates and color of each original point cloud in the original point cloud data and the coordinates and color of each target point cloud in the target point cloud data; calculate the potential reconstruction loss based on the coordinates of each original point cloud, the coordinates of each target point cloud, the local features of each original point cloud, and the local features of each target point cloud; and determine the reconstruction loss based on the auxiliary reconstruction loss and the potential reconstruction loss.
[0137] In one possible implementation, the determining module 350 is specifically used to decode the rendered feature map to obtain a target semantic feature map and a target color feature map; calculate a first loss based on the target semantic feature map and the original semantic feature map corresponding to the sample color image; calculate a second loss based on the sample depth image and the rendered depth map; calculate a third loss based on the target color feature map and the original color feature map corresponding to the sample color image; and determine the rendering alignment loss based on the first loss, the second loss, and the third loss.
[0138] Figure 6 This illustration shows a structural schematic diagram of a vision task execution device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes: an acquisition module 410, used to acquire image data captured by the robot in the current working scene according to the acquired natural language instructions; and to acquire point cloud data corresponding to the image data; Generation module 420 is used to generate structured latent features corresponding to the point cloud data using a three-dimensional visual perception model; the three-dimensional visual perception model is the one used in the above-mentioned... Figures 1-3 A 3D visual perception model of the target is generated by any method; the target motion is generated based on the structured latent features and the robot's state information. The control module 430 is used to control the robot to perform target actions in order to complete the target task indicated by the natural language instructions.
[0139] In one possible implementation, the 3D visual perception model includes: a point cloud encoder and a latent representation encoder. The generation module 420 is specifically used to extract features from the point cloud data using the point cloud encoder to obtain a sparse feature point cloud; and to perform 3D perception based on the sparse feature point cloud using the latent representation encoder to generate the structured latent features.
[0140] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.
[0141] This application also provides a robot. Figure 7 A schematic diagram of the structure of a robot according to an embodiment of this application is shown, as follows: Figure 7 As shown, the robot includes a robot body 510 and a controller 520 disposed within the robot body. The controller 520 is used to execute the steps of the above-described vision task execution method.
[0142] For example, the robot body 510 can be a humanoid robot or a non-humanoid robot.
[0143] In other embodiments of this application, the controller 520 may also be disposed outside the robot body.
[0144] This application also provides a computer device. Figure 8 This application provides a schematic diagram of the structure of a computer device according to an embodiment of the present application. Figure 8 As shown, the system includes a processor 61 and a memory 62, and optionally, a bus 63. The memory 62 stores machine-readable instructions executable by the processor 61. When the computer device is running, the processor 61 communicates with the memory 62 via the bus 63. When the machine-readable instructions are executed by the processor 61, the steps of the above-described training method for the three-dimensional visual perception model or the execution method for the visual task are performed.
[0145] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the above-described training method for a three-dimensional visual perception model or the execution method for a visual task.
[0146] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.
[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0148] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A training method for a three-dimensional visual perception model, characterized in that, The method includes: Acquire sample visual data, which includes: sample color image and sample depth image; The initial encoder in the initial 3D visual perception model is used to extract structured features from the sample depth image to obtain structured latent features; The initial decoder in the initial 3D visual perception model is used to reconstruct the point cloud of the structured latent features to obtain the reconstructed target point cloud data. Based on the target point cloud data, perform 3D perception rendering to generate a rendering feature map and a rendering depth map; The initial encoder is trained based on the sample color image, the sample depth image, the rendered feature map, and the rendered depth map until a preset stopping condition is met. The target encoder that meets the preset stopping conditions is identified as the target 3D visual perception model.
2. The method according to claim 1, characterized in that, The initial encoder includes a point cloud encoder and a latent representation encoder. The initial encoder in the initial 3D visual perception model extracts structured latent features based on the structured features of the sample depth image, including: The point cloud encoder is used to extract features from the original point cloud data corresponding to the sample depth image to obtain a sparse feature point cloud. The latent representation encoder is used to perform 3D perception based on the sparse feature point cloud to generate the structured latent features.
3. The method according to claim 2, characterized in that, The latent representation encoder includes: a backbone network, an attention pooling-based aggregation module, a splicing module, and a structured prediction module; The step of using the latent representation encoder to perform 3D perception based on the coordinates and features of the sparse feature point cloud to generate the structured latent features includes: The backbone network is used to extract local features of each point in the sparse feature point cloud; The aggregation module is used to aggregate the local features of each point to obtain the global features; The stitching module is used to stitch together the local features and the global features of each point to obtain the joint features of each point. The structured prediction module performs structured prediction based on the joint features of each point to obtain the structured latent features.
4. The method according to claim 3, characterized in that, The structured prediction module includes a perceptron and a sampling module. The structured prediction module performs structured prediction based on the joint features of each point to obtain the structured latent features, including: The Gaussian distribution parameters of each point are predicted by using the perceptron to analyze the joint features of each point, thereby obtaining the Gaussian distribution parameters of each point. The sampling module is used to sample the Gaussian distribution parameters of each point to obtain the structured latent features.
5. The method according to claim 4, characterized in that, The Gaussian distribution parameters include a first Gaussian distribution parameter in the feature dimension and a second Gaussian distribution parameter in the coordinate dimension. The step of using the perceptron to predict the Gaussian distribution parameters of the joint features of each point to obtain the Gaussian distribution parameters of each point includes: Two perceptrons are used to predict the Gaussian distribution parameters of the joint features of each point in terms of feature dimension and coordinate dimension, respectively, to obtain the first Gaussian distribution parameter and the second Gaussian distribution parameter of each point.
6. The method according to claim 5, characterized in that, The structured latent features include: latent features in the feature dimension and latent features in the coordinate dimension; the sampling module is used to sample the Gaussian distribution parameters of each point to obtain the structured latent features, including: The sampling module uses the reparameterization technique to sample the first Gaussian distribution parameters and the second Gaussian distribution parameters of each point as latent variables, thereby obtaining the latent features of each point in the feature dimension and the latent features in the coordinate dimension.
7. The method according to claim 1, characterized in that, The step of performing 3D perception rendering based on the target point cloud data to generate a rendering feature map and a rendering depth map includes: Geometric reasoning is performed based on the target point cloud data to obtain the Gaussian sputtering properties of the three-dimensional Gaussian points corresponding to each point in the target point cloud data. Three-dimensional Gaussian rendering is performed based on the Gaussian sputtering properties of the three-dimensional Gaussian points to generate the rendering feature map and the rendering depth map.
8. The method according to claim 4, characterized in that, The step of training the initial encoder based on the sample color image, the sample depth image, the rendered feature map, and the rendered depth map includes: The rendering alignment loss is determined based on the sample color image, the sample depth image, the rendered feature map, and the rendered depth map; The reconstruction loss is determined based on the original point cloud data and the target point cloud data; The divergence loss is determined based on the Gaussian distribution parameters and standard Gaussian distribution of each point; The initial encoder is trained based on the rendering alignment loss, the reconstruction loss, and the divergence loss until the preset stopping condition is met.
9. The method according to claim 8, characterized in that, The step of determining the reconstruction loss based on the original point cloud data and the target point cloud data includes: The auxiliary reconstruction loss is calculated based on the coordinates and color of each original point cloud in the original point cloud data and the coordinates and color of each target point cloud in the target point cloud data. The potential reconstruction loss is calculated based on the coordinates of each original point cloud, the coordinates of each target point cloud, the local features of each original point cloud, and the local features of each target point cloud. The reconstruction loss is determined based on the auxiliary reconstruction loss and the potential reconstruction loss.
10. The method according to claim 8, characterized in that, The step of determining the rendering alignment loss based on the sample color image, the sample depth image, the rendered feature map, and the rendered depth map includes: The rendered feature map is decoded to obtain the target semantic feature map and the target color feature map; The first loss is calculated based on the target semantic feature map and the original semantic feature map corresponding to the sample color image; The second loss is calculated based on the sample depth image and the rendered depth map; The third loss is calculated based on the target color feature map and the original color feature map corresponding to the sample color image; The rendering alignment loss is determined based on the first loss, the second loss, and the third loss.
11. A method for performing a visual task, characterized in that, The method includes: Based on the acquired natural language instructions, acquire image data captured by the robot in the current working scenario; Obtain the point cloud data corresponding to the image data; The structured latent features corresponding to the point cloud data are generated using a three-dimensional visual perception model; the three-dimensional visual perception model is a target three-dimensional visual perception model generated using any one of the methods of claims 1-10 above. Based on the structured latent features and the robot's state information, a target action is generated; Control the robot to perform the target action in order to complete the target task indicated by the natural language command.
12. The method according to claim 11, characterized in that, The three-dimensional visual perception model includes a point cloud encoder and a latent representation encoder. Generating structured latent features corresponding to the point cloud data using the three-dimensional visual perception model includes: The point cloud encoder is used to extract features from the point cloud data to obtain a sparse feature point cloud. The latent representation encoder is used to perform 3D perception based on the sparse feature point cloud to generate the structured latent features.