A robot operation method and system based on task-aware virtual perspective re-rendering
By employing a task-aware virtual perspective re-rendering method, and utilizing a hybrid expert model combining multi-viewpoint exploration and task awareness, the problems of occlusion and multi-task learning in robot operation were solved, resulting in a higher success rate and greater robustness in operation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUN YAT SEN UNIV
- Filing Date
- 2026-04-01
- Publication Date
- 2026-05-29
AI Technical Summary
Existing robot operation methods suffer from fixed-viewpoint occlusion in complex scenes, and feature interference and the inability of models to dynamically select the optimal feature processing expert in multi-task learning.
A task-aware virtual perspective re-rendering method is adopted. The virtual exploration perspective is obtained through the multi-view exploration strategy (MVEP), the global scene point cloud is re-rendered, the task-aware hybrid expert (TaskMoE) model is used for feature extraction, the task-specific fine visual features are generated, and the robot operation sequence is generated through the autoregressive action strategy.
It solves the problems of occlusion and limited viewpoint, improves multi-task versatility and robustness, increases the success rate and anti-interference ability of robot operation, and makes the training process efficient without the need for physical environment interaction.
Smart Images
Figure CN121946544B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of robotics and artificial intelligence, and more specifically, to a robot operation method and system based on task-aware virtual perspective re-rendering. Background Technology
[0002] In recent years, Vision-Language-Action (VLA) models have attracted much attention in the field of robot manipulation. Existing end-to-end robot manipulation frameworks typically utilize the Transformer architecture to map natural language commands and visual observations into low-level control actions of the robot. Typical methods include PerAct and RVT (Robotic ViewTransformer), which usually use static viewpoint camera data as input.
[0003] Despite the progress made by the above methods, there are still many shortcomings in practical complex robot operation tasks, mainly in terms of perception perspective, multi-task learning and model architecture.
[0004] Limitations of static perspective perception: Most existing VLA models rely on camera observations from a single or a few fixed perspectives. In complex dynamic scenes or environments with stacked objects, fixed perspectives can easily lead to occlusion of the target object or the robot's end effector, resulting in incomplete scene understanding and consequently, operation failure. For example, the front view may only show the cabinet, while the side view may only show the object being grasped, lacking a globally consistent spatial understanding.
[0005] Feature interference in multi-task learning: Existing multi-task learning methods typically use a shared visual encoder to handle all tasks. When faced with tasks that differ greatly in visual and semantic aspects (e.g., "grab an apple" versus "open a drawer"), shared parameters can lead to gradient conflicts and feature entanglement between tasks, limiting the model's robustness and generalization ability in multi-task scenarios.
[0006] The routing mechanism of expert hybrid models (MoE) is simplistic: Although some works have introduced expert hybrid models (MoE) to expand the model capacity, the existing MoE routing mechanism usually only relies on the task ID to make simple expert selection, ignoring the specific instruction semantics and visual scene information, which makes it impossible to dynamically select the optimal feature processing expert for a specific scenario. Summary of the Invention
[0007] In view of the above problems, the purpose of this invention is to provide a robot operation method and system based on task-aware virtual perspective re-rendering.
[0008] The first aspect of this invention provides a robot operation method based on task-aware virtual perspective re-rendering, the method comprising:
[0009] S1: Acquire multimodal data of the robot's working scenario;
[0010] S2: Construct a global scene point cloud of the robot's working scene based on the acquired multimodal data;
[0011] S3: Predict the next position of the robot's end effector using a coarse grounding model, and extract a local point cloud centered on that position;
[0012] S4: Input the local point cloud into the Multi-View Exploration Strategy (MVEP) network model to obtain a virtual exploration perspective for point cloud rendering;
[0013] S5: Based on the virtual exploration perspective obtained in S4, re-render the global scene point cloud to obtain a re-rendered image and generate 2D image observation data containing complete visual information.
[0014] S6: Input the re-rendered image and natural language instructions into the TaskMoE (Task-Aware Hybrid Expert) model for feature extraction to obtain task-specific fine visual features;
[0015] S7: Input fine visual features into the Autoregressive Action Policy network model to predict the robot's action sequence;
[0016] S8: Complete the robot's task operation according to the robot's action sequence.
[0017] Preferably, the method employs the following training strategy for training:
[0018] First, train a baseline model with a fixed viewpoint: optimize the coarse localization model to improve its coarse and fine localization capabilities; optimize the action loss of the Autoregressive Action Policy network model.
[0019] Then, the Multi-View Exploration Strategy (MVEP) network model is trained: a pseudo-environment interaction mechanism is introduced, and the trained fixed-view baseline model is used as a "shadow network" to provide supervision signals for the training of the MVEP network model.
[0020] The design includes task loss difference. Confidence reward and perspective diversity rewards The composite reward function is used to optimize the multi-view exploration strategy (MVEP) network model using reinforcement learning methods (such as PPO);
[0021] Finally, the Multi-View Exploration Policy (MVEP) network model was fixed, and the coarse localization model, the Task-Aware Hybrid Expert (TaskMoE) model, and the Autoregressive Action Policy network model were adjusted to adapt them to action generation under dynamic perspectives.
[0022] Preferably, adjusting the Task-Aware Hybrid Expert (TaskMoE) model further includes adding load balancing loss and entropy regularization terms to the Task-Aware Hybrid Expert (TaskMoE) model.
[0023] Preferably, the multimodal data includes: multi-view RGB-D (infrared-depth) image data, robot body perception data, and natural language task instructions.
[0024] Preferably, S3 specifically comprises:
[0025] The target position of the robot's end effector at the next moment is predicted using a coarse grounding network model.
[0026] Using the target location at the next moment as the center, the global scene point cloud is cropped and scaled to extract the region of interest (ROI) point cloud containing the manipulated object, which is then used as the local point cloud. (This serves as input for subsequent processing, reducing computation and focusing on key regions.)
[0027] Preferably, S4 specifically comprises:
[0028] The local point cloud is input into the Multi-View Exploration Strategy (MVEP) network model;
[0029] The Multi-View Exploration Strategy (MVEP) network model predicts K optimal virtual viewpoint poses based on the RGB features of the local point cloud using a Multilayer Perceptron (MLP); each optimal virtual viewpoint pose is represented by a 5-dimensional vector. ;
[0030] in Representing the first in spherical coordinate system The polar angle of the optimal virtual viewpoint pose. Representing the first in spherical coordinate system The azimuth angle of the optimal virtual viewpoint pose. Representing the The radius of the optimal virtual viewpoint pose. Representing the The polar angle of the upper vector of the optimal virtual viewpoint pose. Representing the The azimuth angle of the upper vector of the optimal virtual viewpoint pose;
[0031] The virtual exploration perspective is obtained by randomly sampling the poses of K optimal virtual viewpoints using reparameterization techniques. .
[0032] Preferably, S6 specifically includes:
[0033] The re-rendered image and natural language instructions are input into the TaskMoE (Task-Aware Hybrid Expert) model;
[0034] A cross-attention mechanism is used to process the visual features of natural language instruction embedding and re-rendered images to generate contextual features containing instruction and visual context information.
[0035] The task ID is embedded into the context features and then modulated and fused using the FiLM layer, so that the context features carry specific task identification information;
[0036] set up Each gate and There are 10 expert networks, and the number of expert networks is less than the total number of tasks.
[0037] The task router calculates the gating score for each task based on the task identification information, activates the corresponding gating, and routes different tasks to the corresponding shared skill clusters.
[0038] Each activated gating point further filters a specific expert network. The selected expert network extracts task-specific visual representations for the current task, thus completing the final fine visual feature extraction.
[0039] Preferably, activating the corresponding gating specifically involves:
[0040] Based on the gating score, select tasks whose gating score is higher than a preset threshold and activate the corresponding gating for those tasks.
[0041] Preferably, each activated gating point further filters a specific expert network, specifically:
[0042] Each activated gating system combines its gating score with the expert network's skill characteristics to calculate a matching score, thus selecting the expert network that best suits the current task.
[0043] A second aspect of the present invention provides a robot operating system based on task-aware virtual perspective re-rendering, including a memory and a processor. The memory includes a robot operation method program based on task-aware virtual perspective re-rendering. When the robot operation method program based on task-aware virtual perspective re-rendering is executed by the processor, it implements the steps of a robot operation method based on task-aware virtual perspective re-rendering.
[0044] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:
[0045] (1) Solved the problems of occlusion and limited viewpoint: Through MVEP active exploration and re-rendering, the method described in this invention can obtain complete visual information including key operation objects and end effectors, effectively avoiding blind spots and occlusion caused by fixed viewpoints, thereby improving the success rate of operation.
[0046] (2) Improved multi-task versatility and robustness: The TaskMoE module solves the parameter conflict problem in multi-task learning through task-aware dynamic routing.
[0047] (3) Efficient training process: The pseudo-environment interaction mechanism proposed in this invention avoids the expensive reinforcement learning exploration from scratch, and can efficiently optimize the viewpoint strategy using existing data.
[0048] (4) The method described in this invention has stronger robustness and enhances the anti-interference ability of robot operation. Attached Figure Description
[0049] Figure 1 This is a flowchart of a robot operation method based on task-aware virtual perspective re-rendering.
[0050] Figure 2 This is a logic diagram of a robot operation method based on task-aware virtual perspective re-rendering.
[0051] Figure 3 This is a structural diagram of a task-aware hybrid expert model. Detailed Implementation
[0052] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0053] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0054] Example 1
[0055] like Figure 1 As shown, this embodiment discloses a robot operation method based on task-aware virtual perspective re-rendering, the method including:
[0056] S1: Acquire multimodal data of the robot's working scene.
[0057] S2: Construct a global scene point cloud of the robot's working scene based on the acquired multimodal data.
[0058] In this embodiment, the multimodal data includes: multi-view RGB-D (infrared-depth) image data, robot body perception data, and natural language task instructions.
[0059] It should be noted that this embodiment acquires multi-view RGB-D (infrared-depth) image data of the robot's working scene, robot body perception data (such as joint angles and gripper opening), and natural language task commands. Using the camera's intrinsic and extrinsic parameter matrices, the multi-view RGB-D images are converted into 3D point clouds and aggregated into a global scene point cloud in a unified world coordinate system. .
[0060] S3: Predict the next position of the robot's end effector using a coarse grounding model, and extract a local point cloud centered on that position.
[0061] In this embodiment, S3 specifically refers to:
[0062] S3.1: Predict the target position of the robot's end effector at the next moment using a coarse grounding network model;
[0063] S3.2: Using the target position at the next moment as the center, crop and scale the global scene point cloud to extract the point cloud containing the region of interest (ROI) of the operation object, and use it as the local point cloud.
[0064] It should be noted that this embodiment uses a coarse grounding model to predict the approximate target position of the robot's end effector. Centered on this predicted position, the global scene point cloud is cropped and scaled to extract the region of interest (ROI) point cloud containing the manipulated object. This ROI is then used as input for subsequent processing to reduce computational load and focus on key areas.
[0065] S4: Input the local point cloud into the Multi-View Exploration Strategy (MVEP) network model to obtain a virtual exploration perspective for point cloud rendering.
[0066] In this embodiment, S4 specifically refers to:
[0067] S4.1: Input the local point cloud into the Multi-View Exploration Strategy (MVEP) network model.
[0068] It should be noted that the input feature in this embodiment is: ,in Point cloud number, For splicing operations, This represents the aggregate features of a local point cloud. For the RGB visual features corresponding to a local point cloud, It is the field of real numbers.
[0069] S4.2: The Multi-View Exploration Strategy (MVEP) network model predicts K optimal virtual viewpoint poses based on the RGB features of the local point cloud using a Multilayer Perceptron (MLP); each optimal virtual viewpoint pose is represented by a 5-dimensional vector. ;
[0070] in Representing the first in spherical coordinate system The polar angle of the optimal virtual viewpoint pose. Representing the first in spherical coordinate system The azimuth angle of the optimal virtual viewpoint pose. Representing the The radius of the optimal virtual viewpoint pose. Representing the The polar angle of the upper vector of the optimal virtual viewpoint pose. Representing the The upper vector azimuth angle of the optimal virtual viewpoint pose;
[0071] S4.3: Using reparameterization techniques for The virtual exploration perspective is obtained by randomly sampling the optimal virtual viewpoint pose. .
[0072] It should be noted that, in this embodiment, to support end-to-end training, the output is the mean of a Gaussian distribution. and logarithmic standard deviation By using reparameterization techniques, some viewpoints are randomly selected around the "optimal virtual viewpoint" to obtain the final exploration viewpoint. This ensures that the viewpoint does not deviate from the key area, allows the training gradient of the model to be backpropagated normally, and generates multiple different viewpoints.
[0073] It should be noted that the multi-view exploration strategy (MVEP) described in this embodiment can adaptively explore virtual viewpoints that maximize target coverage and solve the occlusion problem.
[0074] S5: Based on the virtual exploration perspective obtained in S4, re-render the global scene point cloud to obtain a re-rendered image and generate 2D image observation data containing complete visual information.
[0075] S6: Input the re-rendered image and natural language instructions into the TaskMoE (Task-Aware Hybrid Expert) model for feature extraction to obtain task-specific fine visual features.
[0076] In this embodiment, S6 specifically refers to:
[0077] S6.1: Input the re-rendered image and natural language instructions into the TaskMoE (Task-Aware Hybrid Expert) model;
[0078] S6.2: Employs a cross-attention mechanism to process the visual features of natural language instruction embeddings and re-rendered images, generating contextual features that contain instruction and visual contextual information;
[0079] S6.3: The task ID is embedded into the context features and then modulated and fused using the FiLM layer, so that the context features carry specific task identification information;
[0080] S6.4: Settings Each gate and There are 10 expert networks, and the number of expert networks is less than the total number of tasks.
[0081] S6.5: The task router calculates the gating score for each task based on the task identification information, activates the corresponding gating, and routes different tasks to the corresponding shared skill clusters;
[0082] S6.6: Each activated gating further filters a specific expert network, and the selected expert network extracts task-specific visual representations for the current task, thus completing the final fine visual feature extraction.
[0083] It should be noted that the TaskMoE (TaskAware Hybrid Expert Model) described in this embodiment introduces instructions and visual cues to guide expert routing, rather than relying solely on task IDs; at the same time, the TaskMoE (TaskAware Hybrid Expert Model) adopts a decoupled gating strategy (the number of gating mechanisms is less than the number of tasks), allowing semantically similar tasks to share parameters, while semantically different tasks are routed in isolation.
[0084] S7: Input fine visual features into the Autoregressive Action Policy network model to predict the robot's action sequence.
[0085] S8: Complete the robot's task operation according to the robot's action sequence.
[0086] It should be noted that the activation of the corresponding gating is specifically as follows:
[0087] Based on the gating score, select tasks whose gating score is higher than a preset threshold and activate the corresponding gating for those tasks.
[0088] It should be noted that each activated gating point further filters a specific expert network, specifically:
[0089] Each activated gating system combines its gating score with the expert network's skill characteristics to calculate a matching score, thus selecting the expert network that best suits the current task.
[0090] It is particularly important to note that the method also employs the following training strategy:
[0091] First, train a baseline model with a fixed viewpoint: optimize the coarse localization model to improve its coarse and fine localization capabilities; optimize the action loss of the Autoregressive Action Policy network model.
[0092] Then, a multi-view exploration strategy (MVEP) network model is trained: a pseudo-environment interaction mechanism is introduced, and a pre-trained fixed-view baseline model is used as a "shadow network" to provide supervision signals.
[0093] The design includes task loss difference. Confidence reward and perspective diversity rewards The composite reward function is used to optimize the multi-view exploration strategy (MVEP) network model using reinforcement learning methods (such as PPO);
[0094] Finally, the Multi-View Exploration Policy (MVEP) network model was fixed, and the coarse localization model, the Task-Aware Hybrid Expert (TaskMoE) model, and the Autoregressive Action Policy network model were adjusted to adapt them to action generation under dynamic perspectives.
[0095] It should be noted that the training strategy described in this embodiment utilizes a pre-trained fixed-viewpoint model as a reference to calculate the reward signal without the need for physical environment interaction, which greatly improves training efficiency.
[0096] In this embodiment, adjusting the TaskMoE model further includes adding load balancing loss and entropy regularization terms to the TaskMoE model.
[0097] As a specific embodiment, the following description uses specific examples to illustrate this embodiment:
[0098] S1: The system receives RGB-D images (128*128) from four fixed viewpoints (front, left shoulder, right shoulder, and wrist).
[0099] S2: Convert the depth map into a point cloud and aggregate it, remove redundant points outside the workspace, and obtain the global scene point cloud.
[0100] S3: Rough positioning.
[0101] PointNet++ is used as a point cloud encoder to predict the next position of the end effector and to capture a local point cloud centered on this position.
[0102] S4: Perspective Exploration.
[0103] Input the local point cloud into the MVEP network (based on MLP). Output the parameter distributions of 3 (K=3) optimal viewpoints.
[0104] The parameters are restricted to a specific range of spherical coordinates, such as the radius range r. [0.90, 1.04]m is used to obtain the best performance.
[0105] S5: Re-render: Re-render a 224*224 resolution image based on the predicted viewpoint.
[0106] S6: Feature extraction.
[0107] The TaskMoE model is configured with 8 gating gates and 16 experts, with the top-2 experts selected for each task.
[0108] like Figure 3 As shown, Figure 3 This is a diagram of the Task-Aware Hybrid Expert (TaskMoE) model structure. The diagram shows the conditional input based on cross-attention and FiLM, as well as the routing process from task ID to gating and then to the expert.
[0109] In this embodiment, the processing procedure for the Task-Aware Hybrid Expert (TaskMoE) model is as follows:
[0110] 1. Instructions – Visual Fusion: Input a sequence of visual tokens and instruction embedding Through the formula:
[0111] , ,
[0112]
[0113] This embodiment obtains the fusion features by calculating cross attention. .
[0114] 2. FiLM Modulation: Embedding globally pooled visual features with the task The data is stitched together, and a scaling factor is generated using an MLP. and bias ,right Modulation is performed to obtain .
[0115]
[0116]
[0117]
[0118] 3. Two-level routing:
[0119] Level 1 (Gate Control): Based on Select the active gating.
[0120]
[0121] Level 2 (Expert): Under the selected gating, calculate...
[0122]
[0123] Select the final expert network.
[0124] The selected expert network extracts task-specific visual representations for the current task, thus completing the final fine visual feature extraction.
[0125] 4. Regularization: Add load balancing loss to the training of the method described above. Entropy regularization term This prevents route collapse and ensures that experts are fully utilized.
[0126] This embodiment uses cross-attention fusion of text instructions such as "put the sugar in the cabinet" and image features to dynamically activate a specific expert network responsible for "opening the cabinet door" or "grabbing the object".
[0127] S7; Action generation: The autoregressive policy network outputs a 7-dimensional action vector (position, rotation, gripper state).
[0128] S8: Real-world execution: On the Dobot Nova 2 robotic arm, the system successfully completed tasks such as "picking grapes" and "stacking bowls." For example, in the "picking grapes" task, even when the grapes were obscured from certain fixed viewpoints, MVEP was able to find an unobstructed viewpoint for observation and successful grasping.
[0129] Example 2
[0130] This embodiment discloses a robot operating system based on task-aware virtual perspective re-rendering, including a memory and a processor. The memory includes a robot operation method program based on task-aware virtual perspective re-rendering. When the robot operation method program based on task-aware virtual perspective re-rendering is executed by the processor, it implements the steps of the robot operation method based on task-aware virtual perspective re-rendering described in Embodiment 1.
[0131] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0132] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0133] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
Claims
1. A robot operation method based on task-aware virtual perspective re-rendering, characterized in that, The method includes: S1: Acquire multimodal data of the robot's working scenario; S2: Construct a global scene point cloud of the robot's working scene based on the acquired multimodal data; S3: Predict the next position of the robot's end effector using a coarse localization model, and extract a local point cloud centered on that position; S4: Input the local point cloud into the multi-view exploration strategy network model to obtain a virtual exploration perspective for point cloud rendering; S5: Based on the virtual exploration perspective obtained in S4, re-render the global scene point cloud to obtain a re-rendered image and generate 2D image observation data containing complete visual information. S6: Input the re-rendered image and natural language instructions into the task-aware hybrid expert model for feature extraction, obtaining task-specific fine visual features; specifically: Input re-rendered images and natural language instructions into a task-aware hybrid expert model; A cross-attention mechanism is used to process the visual features of natural language instruction embedding and re-rendered images to generate contextual features containing instruction and visual context information. The task ID is embedded into the context features and then modulated and fused using the FiLM layer, so that the context features carry specific task identification information; set up individual gates and There are 10 expert networks, and the number of expert networks is less than the total number of tasks. The task router calculates the gating score for each task based on the task identification information, activates the corresponding gating, and routes different tasks to the corresponding shared skill clusters. Each activated gate further filters a specific expert network, and the selected expert network extracts task-specific visual representations for the current task, thus completing the final fine visual feature extraction. S7: Input fine visual features into the autoregressive action policy network model to predict the robot's action sequence; S8: Complete the robot's task operation according to the robot's action sequence.
2. The robot operation method based on task-aware virtual perspective re-rendering according to claim 1, characterized in that, The method employs the following training strategy: First, train a baseline model with a fixed viewpoint: optimize the coarse localization model to improve its coarse and fine localization capabilities; optimize the action loss of the autoregressive action policy network model. Then, a multi-viewpoint exploration strategy network model is trained: a pseudo-environment interaction mechanism is introduced, and a pre-trained baseline model with a fixed viewpoint is used to provide supervision signals for the training of the multi-viewpoint exploration strategy network model. The design includes task loss difference. Confidence reward and perspective diversity rewards The composite reward function is used to optimize the multi-view exploration strategy network model using reinforcement learning methods; Finally, the multi-viewpoint exploration strategy network model was fixed, and the coarse localization model, task-aware hybrid expert model, and autoregressive action strategy network model were adjusted to adapt them to action generation under dynamic perspectives.
3. The robot operation method based on task-aware virtual perspective re-rendering according to claim 2, characterized in that, Adjusting the task-aware hybrid expert model also includes adding load balancing loss and entropy regularization terms to the task-aware hybrid expert model.
4. A robot operation method based on task-aware virtual perspective re-rendering according to claim 2 or 3, characterized in that, The multimodal data includes: multi-view RGB-D image data, robot body perception data, and natural language task instructions.
5. A robot operation method based on task-aware virtual perspective re-rendering according to claim 4, characterized in that, S3 specifically refers to: Predict the target position of the robot's end effector at the next moment using a coarse localization network model; Using the target position at the next moment as the center, the global scene point cloud is cropped and scaled to extract the region of interest point cloud containing the operation object, which is then used as the local point cloud.
6. A robot operation method based on task-aware virtual perspective re-rendering according to claim 5, characterized in that, S4 specifically refers to: Input the local point cloud into the multi-view exploration strategy network model; The multi-viewpoint exploration strategy network model predicts K optimal virtual viewpoint poses based on the RGB features of the local point cloud using a multilayer perceptron; each optimal virtual viewpoint pose is represented by a 5-dimensional vector. ; in Representing the first in spherical coordinate system The polar angle of the optimal virtual viewpoint pose. Representing the first in spherical coordinate system The azimuth angle of the optimal virtual viewpoint pose. Representing the The radius of the optimal virtual viewpoint pose. Representing the The polar angle of the upper vector of the optimal virtual viewpoint pose. Representing the The azimuth angle of the upper vector of the optimal virtual viewpoint pose; The virtual exploration perspective is obtained by randomly sampling the poses of K optimal virtual viewpoints using reparameterization techniques. .
7. The robot operation method based on task-aware virtual perspective re-rendering according to claim 1, characterized in that, The activation of the corresponding gating is specifically as follows: Based on the gating score, select tasks whose gating score is higher than a preset threshold and activate the corresponding gating for those tasks.
8. A robot operation method based on task-aware virtual perspective re-rendering according to claim 1 or 7, characterized in that, Each activated gate further filters a specific expert network, specifically: Each activated gating system combines its gating score with the expert network's skill characteristics to calculate a matching score, thus selecting the expert network that best suits the current task.
9. A robot operating system based on task-aware virtual perspective re-rendering, characterized in that, The system includes a memory and a processor. The memory includes a robot operation method program based on task-aware virtual perspective re-rendering as described in claim 1. When the robot operation method program based on task-aware virtual perspective re-rendering is executed by the processor, it implements the steps of a robot operation method based on task-aware virtual perspective re-rendering.