Robot virtual-real collaborative training method and system based on ar-vla fusion

Abstract

This invention discloses a robot virtual-real collaborative training method and system based on AR-VLA fusion, relating to the fields of robotics, computer vision, and augmentation technology. Addressing the problems of high data costs, reliance on robot presence, and poor generalization ability in existing training methods, this invention collects human demonstration multimodal data pairs (VLA) from operators performing tasks using AR devices. human ,L,A human ), which is converted into robot-executable data pairs (V robot ,L,A robot Based on this, a VLA model is trained and domain-adapted to fit real-world scenarios; the trained model is then deployed to a real robot to generate action outputs based on real visual and verbal commands. This invention can be widely applied in professional fields such as industrial assembly, medical surgery, and hazardous environment inspection.

Description

Technical Field

[0001] This invention relates to the fields of robotics, computer vision, and augmentation technology. More specifically, this invention relates to a method and system for robot virtual-real collaborative training based on AR-VLA fusion. Background Technology

[0002] In high-risk, high-cost professional fields such as industrial assembly, medical surgery, and hazardous environment inspection, the core bottleneck of robot training lies in data acquisition and model generalization ability. Traditional robot training methods heavily rely on robots collecting real-world data on-site. However, high-risk scenarios such as nuclear power plant inspections and surgical procedures cannot be repeatedly collected, with the cost of a single data instance exceeding ten thousand yuan. Furthermore, traditional methods require the simultaneous presence of both robots and humans, resulting in high costs for robot hardware and scheduling. This limits the data scale to the number of robots, available space, and scheduling, making it difficult to expand the data on a large scale through operator demonstrations alone. In addition, annotation for specialized tasks requires the participation of domain experts, such as labeling surgical lesion boundaries, with the cost of a single annotation exceeding one thousand yuan, and the results are highly subjective and unreproducible. Real-world tasks also contain a large number of low-probability but high-impact long-tail scenarios, such as robots grasping slippery parts or encountering temporary obstacles during inspections. Traditional imitation learning cannot cover these scenarios, leading to insufficient generalization ability of the model in practical applications.

[0003] To address these issues, existing technologies attempt to train robot strategies using purely simulated environments. However, a significant "Sim2Real Gap" exists between simulation and the real world, causing models that perform well in simulation to experience a more than 50% drop in task success rate when transferred to real-world scenarios. Furthermore, safety rules in real-world scenarios, such as surgical robots not touching blood vessels and industrial robots not entering dangerous areas, are difficult to effectively encode using traditional training methods, increasing the risk of safety accidents during training. Although some researchers have attempted to apply augmented reality (AR) technology to robot-assisted operation, existing AR solutions remain at the assistance level, failing to integrate multimodal unified learning, and even more so, failing to achieve robot strategy training that is primarily from the operator's perspective and does not rely on the robot's presence.

[0004] On the other hand, existing unified visual-language-action (VLA) learning models are mostly designed with a large number of parameters, reaching billions, resulting in high training and deployment costs. This makes it difficult to complete training and inference on a single GPU or consumer-grade hardware, limiting their widespread application in professional fields. While pure virtual reality (VR) solutions can simulate the training environment, VR is completely detached from the real world, and data acquisition cannot reflect the complex factors of real-world scenes such as lighting, texture, and occlusion, leading to poor transfer learning results. In contrast, AR overlays virtual elements into the real environment, resulting in data acquisition that more closely approximates the distribution of the real world, and its spatial construction and scene building costs are lower. However, existing AR technology has not yet been effectively integrated with VLA model training.

[0005] In summary, existing technologies suffer from the following significant drawbacks: real-world data acquisition is extremely costly and relies on the presence of a robot; simulation training suffers from a significant Sim2Real gap; safety constraints are difficult to embed into the training process; VLA models have a large number of parameters and high training costs; and AR and VLA have not yet been integrated to form a robot training loop primarily based on the operator's perspective. These shortcomings result in low efficiency, high cost, and poor generalization ability in professional robot training. There is an urgent need for a robot training method and system that can use the operator's perspective as the primary data source, collect data on a large scale without the need for a robot's presence, and effectively reduce the gap between virtual and real data. Summary of the Invention

[0006] One object of the present invention is to solve at least the above-mentioned problems and to provide at least the advantages that will be described later.

[0007] Another objective of this invention is to provide a robot virtual-real collaborative training method based on AR-VLA fusion, which collects human demonstrations through AR and transfers them to robot data, thereby reducing data collection costs and robot dependence, and improving task success rate and generalization ability.

[0008] To achieve these objectives and other advantages according to the present invention, a robot virtual-real collaborative training method based on AR-VLA fusion is provided, comprising: Training data pairs (V) of human demonstrations performed by operators during task execution were collected using AR devices. human ,L,A human And convert it into robot-executable data pairs (V) that are aligned with the robot's viewpoint and action space. robot ,L,A robot ); Based on the robot executable data pair (V) robot ,L,A robot Train the VLA model and perform domain adaptation on the trained VLA model to fit the real-world operating scenario; The VLA model, which has completed domain adaptive training, is deployed to a real robot to generate robot motion outputs based on real visual and task language command inputs.

[0009] Preferably, the robot virtual-real collaborative training method based on AR-VLA fusion specifically includes: S1. Utilize AR spatial registration technology to construct an editable virtual-real fusion scene for robot task execution and establish a unified world coordinate system; S2. In the virtual-real fusion scenario, the operator wears an AR device to perform tasks from the operator's perspective, and simultaneously collects and produces multimodal aligned training data pairs (V) from the operator's perspective. human ,L,Ahuman Based on the unified world coordinate system, and through viewpoint transfer and motion transfer mechanisms, the multimodal aligned training data pairs (V) of the operator's viewpoint are... human ,L,A human ) is transformed into robot-executable data pairs (V) aligned with the robot's vision and robot motion space. robot ,L,A robot This enables large-scale data collection without the need for robots to be present. S3, Based on robot executable data pairs (V robot ,L,A robot Joint representation training is performed on the VLA model, enabling the VLA model to learn the mapping relationship from robot-perspective images and task language instructions to robot actions; S4. Perform domain adaptive training on the VLA model to reduce the feature distribution difference between the synthetic visual data generated by AR acquisition and transfer and the real visual data acquired by the robot in the real operation scenario. S5. Deploy the VLA model after domain adaptive training to the robot in a real-world work scenario to generate robot motion output based on real visual and language input. Among them, V human ,L,A human These are respectively operator-view images, task language instructions, and operator action data; V robot ,L,A robot These are robot vision, task language commands, and robot motion data.

[0010] Preferably, step S1, the construction of the editable virtual-real fusion scene, is based on arranging visual marker codes, calculating the real-time six-DOF pose of the AR device through the PnP algorithm, and anchoring virtual elements at specified positions in a unified world coordinate system. Specifically, this includes: Visual markers are deployed in the work environment, and the three-dimensional coordinates of each marker are measured in advance in a unified world coordinate system. The AR device worn by the operator captures images of the work scene in real time through the built-in camera, detects visual marker codes in the images, and obtains their corresponding two-dimensional pixel coordinates; Based on the two-dimensional pixel coordinates and the pre-measured three-dimensional coordinates, the real-time six-degree-of-freedom pose of the AR device in a unified world coordinate system is calculated using the PnP algorithm. Based on the real-time six-degree-of-freedom pose, virtual elements are anchored at a specified position in a unified world coordinate system to construct an editable virtual-real fusion scene.

[0011] Preferably, step S2 specifically includes: S21. In the editable virtual-real fusion scene constructed in step S1, the operator wears an AR device to perform tasks from the operator's perspective, and the AR device's sensors acquire images V from the operator's perspective in real time. human It records the real-time pose of the AR device in a unified world coordinate system, and simultaneously detects and tracks the operator's hand movements from the AR field of view, generating operator motion data A. human Simultaneously acquire task language commands L, which are either verbally entered by the operator or selected from a preset task library, and are bound to the currently executed task during acquisition; simultaneously acquire operator-perspective images V. human Task language instructions (L), operator action data (A) human Perform timestamp alignment to form multimodal aligned training data pairs (V) from the operator's perspective. human , L, A human ); S22. Based on the aforementioned unified world coordinate system, V is transferred through a viewpoint migration mechanism. human Mapped to a robot-view image Vr aligned with the robot's viewpoint obot A through action transfer mechanism human Mapped to robot motion data A aligned with the robot's motion space robot To obtain the robot's executable data pair (V) robot , L, A robot ).

[0012] Preferably, step S3 specifically includes: S31. Construct a VLA model, which includes a visual encoder, a language encoder, a multimodal fusion module, and an action decoder; S32, Transfer robot executable data pairs (V) robot , L, A robot Robot vision image V robot The visual encoder is input to extract features and obtain a visual feature vector; the task language instruction L is input to the language encoder to extract features and obtain a language feature vector; the visual feature vector and the language feature vector are input to the multimodal fusion module for joint representation learning to obtain multimodal joint features. S33. Based on the aforementioned multimodal joint features, predict robot action A using an action decoder. robot and with A robot As a supervisory signal, the VLA model is trained end-to-end using a joint loss function; Wherein, the joint loss function L total for: ; L VLThe visual-language alignment loss is used to maximize the similarity between visual feature vectors and linguistic feature vectors. Its calculation method is as follows: ; L LA The language-action conditional loss is used to ensure that actions are generated from language instructions. It adopts the form of stream matching loss, and its calculation method is as follows: ; L AF The motion feasibility loss is used to constrain motions to conform to robot dynamics and joint limits. Its calculation method is as follows: ; L S Safety constraint loss, used to penalize actions that violate safety rules, is calculated as follows: ; Among them, v i Let be the visual feature vector of the i-th sample; l i and l j Let be the language feature vectors of the i-th and j-th samples, respectively; sim(·) is the cosine similarity function; T is the temperature parameter, with a value of 0.07; N is the batch sample size; σ t This represents the joint representation of multiple modes under current observations. A t For the actual action block at time t; For noise-generating action blocks, ; It is Gaussian noise; τ is the noise scheduling parameter; The vector field predicted by the action decoder; u is the target vector field; o t For current observations; q t Let be the robot joint angle at time t; q lim Limit the movement of robot joints; c represents the tolerance, with a value of 0.01 rad. M is the length of the action sequence; d t Let t be the distance from the robot's end effector to the danger zone at time t; k is the safety threshold, with a value of 0.05m; λ1, λ2, λ3 and λ4 are the weighting coefficients of each loss term, with values ​​of 0.3, 0.4, 0.2 and 0.1 respectively.

[0013] Preferably, step S4 specifically includes: S41. Data sources for domain-adaptive training include: Source Domain Data X s Synthetic visual data generated by AR acquisition and transfer, wherein the synthetic visual data comes from the robot view image Vrobot or its visual feature vector obtained by view transfer in step S2; Target domain data X r Real visual data collected by the robot in real-world work scenarios; S42. Construct a domain adaptive network, wherein the domain adaptive network includes: The feature mapping network G shares the visual encoder of the VLA model in step S3 as a shared encoder, and a mapping head is linked after the shared encoder. The mapping head is a 3-layer fully connected network with a dimension change of 512→256→128. Each layer is followed by a ReLU activation function and a BatchNorm normalization layer. Domain discriminator D is a 4-layer fully connected network with dimensions changing from 128 to 128 to 64 to 1. Each layer is followed by a LeakyReLU activation function and a Dropout layer. The output layer uses a Sigmoid activation function to distinguish between the source domain and the target domain of the input features. A gradient inversion layer GRL is added after the feature mapping network G and before the domain discriminator D. The gradient inversion layer GRL implements the identity mapping during forward propagation and multiplies the gradient by -1 during backward propagation so that the shared encoder learns domain-invariant features. S43. Optimize the domain adaptive loss function L through adversarial domain adaptation training. DA and the VLA joint loss function L from step S3. total Joint optimization aligns the feature distribution of synthetic visual data with that of real visual data; The domain adaptive loss function L DA for: ; The joint optimization objective function is L joint : ; Where, x s The source domain data sample is the synthetic visual data. x r The target domain data sample is the real visual data. G(·) is a feature mapping network that maps input data to the feature space; D(·) is a domain discriminator that outputs the probability that the input feature belongs to the source domain; E represents expectation; λ DA The domain adaptive loss weight coefficient has a value of 0.1 to 0.5.

[0014] Preferably, step S5 is followed by an asynchronous inference step S6: after the VLA model is deployed, an asynchronous inference mechanism is used to decouple action block prediction from action execution, thereby improving the robot's control frequency and responsiveness; the asynchronous inference mechanism specifically includes: S61. Maintain an action queue on the robot side, the action queue storing action blocks A output by the VLA model. t =(a t ,a t+1 ,…,a t+n ), where n is the action block size, a t The underlying action instruction at time t; S62. The robot takes action a from the head of the action queue according to a fixed control cycle. t And execute; S63. Monitor the number of remaining actions in the action queue in real time. When the proportion of the number of remaining actions to the action block size n is lower than the preset threshold g, the robot collects the current observation o. t+1 It is then sent to the policy side where the VLA model is located, requesting the generation of a new action block, while continuing to execute the remaining actions in the current queue; S64, The strategy side receives observation o t+1 Then, the VLA model is run for forward inference to obtain new action blocks. And send the new action block back to the robot; S65. The robot aggregates the new action block with the remaining overlapping time step actions in the current queue and updates the action queue. The preset threshold g has a value range of 0.5 to 0.8.

[0015] This invention further claims a robot virtual-real collaborative training system based on AR-VLA fusion, comprising: AR devices, worn by operators, are used to perform tasks from the operator's perspective in a virtual-real fusion scene, simultaneously acquiring and producing images (V) from the operator's viewpoint. human Task language instructions (L) and operator action data (A) human Human demonstration multimodal alignment training data pairs (V human , L, A human ); The spatial registration module is used to construct an editable virtual-real fusion scene for robot task execution and to establish a unified world coordinate system; The migration module, connected to the spatial registration module, is used to migrate the V coordinate system based on the unified world coordinate system through a view migration mechanism. human Mapped to a robot-view image V aligned with the robot's viewpoint robot A through action transfer mechanism human Mapped to robot motion data A aligned with the robot's motion space robot To obtain the robot's executable data pair (V) robot , L, A robot This enables large-scale data collection without the need for robots to be present. The VLA model training module, connected to the transfer module, is used to train the robot based on the robot executable data pair (VLA). robot , L, A robot Joint representation training is performed on the VLA model, enabling the VLA model to learn the mapping relationship from robot-perspective images and task language instructions to robot actions. The domain adaptive training module, connected to the VLA model training module, is used to perform domain adaptive training on the trained VLA model to reduce the feature distribution differences between the synthetic visual data generated by AR acquisition and transfer and the real visual data acquired by the robot in the real operation scenario; the deployment module is used to deploy the VLA model after domain adaptive training to the robot in the real operation scenario, so that the robot can generate robot action output based on real visual and language input.

[0016] The present invention further claims a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the described AR-VLA fusion-based robot virtual-real collaborative training method.

[0017] The present invention has at least the following beneficial effects: Firstly, the AR device of this invention collects human demonstration data from the operator's perspective, and transforms it into robot-executable data through perspective and action transfer, realizing large-scale data collection without the need for robot presence, significantly reducing data collection costs and robot hardware dependence, reducing single-case data costs by more than 90%, and improving data collection efficiency by several times. Secondly, this invention establishes a unified world coordinate system through spatial registration technology, and combines visual marker codes and PnP pose calculation to achieve precise anchoring of AR virtual elements and real environment and spatiotemporal alignment of human-robot perspective. It provides a high-precision spatial reference for perspective migration and motion migration, ensuring that data alignment error is ≤2cm and ≤2°, meeting the needs of high-precision scenarios such as fine assembly and surgery. Thirdly, this invention designs a joint loss function that includes visual-language alignment loss, language-action conditional loss, action feasibility loss, and safety constraint loss, and uses adversarial domain adaptive training to align the feature distribution of synthetic data and real data, thereby increasing the success rate of the VLA model in real-world tasks to over 85%, reducing the safety incident rate by 93%, and achieving a language-action matching degree of 95%. Fourth, this invention decouples action block prediction from action execution through an asynchronous inference mechanism, maintains an action queue on the robot side and triggers new block requests according to a threshold, effectively avoiding idle waiting for inference, improving control frequency under the same hardware conditions, and is suitable for edge computing and low-cost robot deployment scenarios.

[0018] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0019] Figure 1 This is a VLA model framework diagram according to one technical solution of the present invention; Figure 2 This is a network structure diagram of the VLA model described in another technical solution of the present invention; Figure 3 This is the architecture of the AR-VLA fusion robot virtual-real collaborative training system described in another technical solution of the present invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0021] It should be understood that terms such as “having,” “comprising,” and “including” as used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

[0022] like Figure 1 , 2 As shown, this invention provides a robot virtual-real collaborative training method based on AR-VLA fusion, comprising: Training data pairs (V) of human demonstrations performed by operators during task execution were collected using AR devices. human ,L,A human And convert it into robot-executable data pairs (V) that are aligned with the robot's viewpoint and action space. robot ,L,A robot ); Based on the robot executable data pair (V) robot ,L,A robotTrain the VLA model and perform domain adaptation on the trained VLA model to fit the real-world operating scenario; The VLA model, which has completed domain adaptive training, is deployed to a real robot to generate robot motion outputs based on real visual and task language command inputs.

[0023] The above technical solution addresses the problems of existing technologies that rely on robots to collect data, resulting in high costs, small scale, and a significant gap between simulation and real-world scenarios. It primarily collects human demonstration data from the operator's AR perspective, generating robot-executable data through viewpoint and action transfer, thus enabling VLA model training without the need for a robot's presence. Specifically, this technical solution uses AR devices to collect multimodal aligned training data pairs (V... human ,L,A human ), which are then converted into robot-executable data pairs (V) through viewpoint and motion transfer. robot ,L,A robot The VLA model was trained and deployed to a real robot after domain adaptation. This enabled large-scale data collection without the robot's presence, significantly reducing data costs and hardware dependence. By leveraging domain practicality, the gap between virtual and real environments was narrowed, improving the success rate and generalization ability of real-world tasks, and effectively embedding safety constraints.

[0024] The aforementioned AR-VLA fusion-based robot virtual-real collaborative training method can be applied to fields such as industrial assembly, medical surgery, and hazardous environment inspection. For example, in industrial assembly scenarios, virtual bearings, virtual wrenches, and safety zones are overlaid in the AR scene. Operators wearing AR devices perform assembly demonstrations, and the collected data is used to train robots to complete precision bearing assembly tasks, requiring a tightening torque of 0.8 N·m and a torque error ≤0.1 N·m. In medical surgery scenarios, virtual gallbladders, virtual arteries, and hazardous areas are overlaid in the AR scene. Expert physicians wearing AR devices perform resection demonstrations in a simulated surgical environment, and the collected data is used to train laparoscopic robots to complete gallbladder resection tasks, avoiding contact with arteries. In nuclear power plant inspection scenarios, virtual leak points, virtual radiation zones, and high-risk warnings are overlaid in the AR scene. Inspection personnel wearing AR devices perform inspection and labeling demonstrations in a simulated plant, and the collected data is used to train robots to complete pipeline leak detection tasks, avoiding radiation zones.

[0025] In one of the technical solutions, the robot virtual-real collaborative training method based on AR-VLA fusion specifically includes: S1. Utilize AR spatial registration technology to construct an editable virtual-real fusion scene for robot task execution and establish a unified world coordinate system; S2. In the virtual-real fusion scenario, the operator wears an AR device to perform tasks from the operator's perspective, and simultaneously collects and produces multimodal aligned training data pairs (V) from the operator's perspective.human ,L,A human Based on the unified world coordinate system, and through viewpoint transfer and motion transfer mechanisms, the multimodal aligned training data pairs (V) of the operator's viewpoint are... human ,L,A human ) is transformed into robot-executable data pairs (V) aligned with the robot's vision and robot motion space. robot ,L,A robot This enables large-scale data collection without the need for robots to be present. S3, Based on robot executable data pairs (V robot ,L,A robot Joint representation training is performed on the VLA model, enabling the VLA model to learn the mapping relationship from robot-perspective images and task language instructions to robot actions; S4. Perform domain adaptive training on the VLA model to reduce the feature distribution difference between the synthetic visual data generated by AR acquisition and transfer and the real visual data acquired by the robot in the real operation scenario. S5. Deploy the VLA model after domain adaptive training to the robot in a real-world work scenario to generate robot motion output based on real visual and language input. Among them, V human ,L,A human These are respectively operator-view images, task language instructions, and operator action data; V robot ,L,A robot These are robot vision, task language commands, and robot motion data.

[0026] This application further discloses a specific implementation method for VLA model training without the need for a robot's presence. This method primarily collects human demonstration data from the operator's AR perspective, generates robot-executable data through perspective and action transfer, and details five specific implementation steps to form a complete virtual-real collaborative training closed loop. Step S1 utilizes AR spatial registration technology to construct an editable virtual-real fusion scene. Visual marker codes are deployed at the work site, and the three-dimensional coordinates of each visual marker code in a unified world coordinate system are measured in advance. The AR device worn by the operator acquires scene images in real time through its built-in camera, detects the marker codes and obtains the two-dimensional pixel coordinates, and calculates the real-time six-DOF pose of the AR device using a pose algorithm. Based on the real-time six-DOF pose, virtual elements (such as virtual training targets, task operation constraint boundaries, and safe areas) are precisely anchored at designated positions in the world coordinate system, providing a consistent virtual-real spatial reference for subsequent data acquisition. Wherein, the visual identifier code adopts at least one of AprilTag, ArUco or QR code; the pose calculation adopts PnP algorithm or EPnP algorithm; the virtual training target 3D model includes at least one interactive CAD model of industrial parts, surgical instruments, and inspection equipment; the operation constraint boundary 3D model includes at least one of safe area box, danger area bounding box, and joint motion limiting surface.

[0027] Step S2 involves data acquisition and migration within the editable virtual-real fusion scene. The operator performs tasks from a first-person perspective, and the AR device simultaneously acquires an image V from the operator's perspective. human And detect and track hand movements from the field of view to generate operator motion data A human Simultaneously, it binds the task language instruction L, which is then timestamped to form (V). human , L, A human Based on a unified world coordinate system, the V coordinate system is transformed through a perspective shift mechanism. human Mapped to robot's perspective image V robot A through action transfer mechanism human Mapped to robot motion data A robot Ultimately, the robot's executable data pair (V) is obtained. robot , L, A robot The above process allows for large-scale data collection without the need for a robot to be present. Step S3 is based on (V robot , L, A robot Step S4 involves joint representation training of the VLA model to learn the mapping relationship from robot-perspective images and language commands to robot actions. Step S5 uses adversarial domain adaptive training to reduce the feature distribution differences between synthetic and real visual data. Step S6 deploys the trained model to a real robot, generating action outputs based on real visual and language inputs, completing the entire process from data acquisition to model deployment.

[0028] The above technical solution establishes a unified world coordinate system through spatial registration technology, providing a precise spatial benchmark for data collection and migration, and ensuring the spatiotemporal consistency between human demonstration data and robot perspective; through perspective and action transfer mechanisms, it enables large-scale data collection without the need for robot presence, significantly reducing data acquisition costs and hardware dependence; combined with VLA joint training and domain adaptation, it effectively narrows the gap between virtual and real, improves the model's generalization ability and task success rate in real scenarios, and naturally embeds safety constraints.

[0029] In one technical solution, step S1, the construction of an editable virtual-real fusion scene, is based on arranging visual marker codes, calculating the real-time six-degree-of-freedom pose of the AR device through the PnP algorithm, and anchoring virtual elements at specified positions in a unified world coordinate system. Specifically, this includes: Visual markers are deployed in the work environment, and the three-dimensional coordinates of each marker are measured in advance in a unified world coordinate system. The AR device worn by the operator captures images of the work scene in real time through the built-in camera, detects visual marker codes in the images, and obtains their corresponding two-dimensional pixel coordinates; Based on the two-dimensional pixel coordinates and the pre-measured three-dimensional coordinates, the real-time six-degree-of-freedom pose of the AR device in a unified world coordinate system is calculated using the PnP algorithm. Based on the real-time six-degree-of-freedom pose, virtual elements are anchored at a specified position in a unified world coordinate system to construct an editable virtual-real fusion scene.

[0030] This application further optimizes the method for constructing editable virtual-real fusion scenes. By deploying visual marker codes and pre-measuring their 3D coordinates, combined with the PnP algorithm, it calculates the six-DOF pose of the AR device in real time, providing a high-precision spatial reference for the entire training method. First, it ensures the precise anchoring of virtual elements to the real environment, with translation errors ≤1~2cm and posture errors ≤1~2°, meeting the requirements of high-precision scenarios such as precision assembly and medical surgery. Second, it achieves real-time tracking of the AR device's pose based on a unified world coordinate system, providing a reliable spatial mapping relationship for subsequent viewpoint and motion transfer. Third, it can construct editable virtual-real fusion scenes without relying on the presence of a robot, significantly reducing the cost and complexity of scene construction, supporting large-scale, low-cost data acquisition, and solving the problem of dependence on robot hardware in traditional methods from the source.

[0031] In one of the technical solutions, step S2 specifically includes: S21. In the editable virtual-real fusion scene constructed in step S1, the operator wears an AR device to perform tasks from the operator's perspective, and the AR device's sensors acquire images V from the operator's perspective in real time. humanIt records the real-time pose of the AR device in a unified world coordinate system, and simultaneously detects and tracks the operator's hand movements from the AR field of view, generating operator motion data A. human Simultaneously acquire task language commands L, which are either verbally entered by the operator or selected from a preset task library, and are bound to the currently executed task during acquisition; simultaneously acquire operator-perspective images V. human Task language instructions (L), operator action data (A) human Perform timestamp alignment to form multimodal aligned training data pairs (V) from the operator's perspective. human , L, A human ); S22. Based on the aforementioned unified world coordinate system, V is transferred through a viewpoint migration mechanism. human Mapped to a robot-view image Vr aligned with the robot's viewpoint obot A through action transfer mechanism human Mapped to robot motion data A aligned with the robot's motion space robot To obtain the robot's executable data pair (V) robot , L, A robot ).

[0032] The above technical solution, based on the high-precision editable virtual-real fusion scene constructed in step S1, realizes the complete acquisition and automatic migration of human demonstration data, and achieves this by acquiring operator-perspective images V in real time. human Hand movements A human Furthermore, it is aligned with the timestamp of the task instruction L to form standardized multimodal training data with spatiotemporal consistency, enabling the acquisition of high-quality demonstration samples without manual annotation; based on a unified world coordinate system, human demonstrations are precisely mapped to robot-perspective images V through viewpoint and motion transfer mechanisms. robot And robot action A robot This enables a seamless transformation from human demonstration to robot-executable data; and the entire process does not require the robot's presence. Large-scale data collection can be completed solely by operators wearing AR devices, fundamentally breaking through the dependence of traditional methods on robot hardware, significantly reducing the threshold and cost of data acquisition, and providing high-quality input data for subsequent VLA model training.

[0033] Preferably, step S2 also includes view normalization and task annotation enhancement steps: S23. The collected training data pairs of human demonstration multimodal alignment (V) human , L, A human ) and the robot-executable data obtained through migration (V robot , L, A robot)Viewpoint normalization is performed, mapping viewpoint images from different sources to a unified viewpoint type number in the dataset, including top view / main view, wrist / head-mounted view, and side view. During training, multi-viewpoint images are input in a fixed order. S24. Automatically enhance the task language instruction L. When the original language annotation L is ambiguous or missing, use a pre-trained visual-language model to automatically complete or rewrite the representative image frame and the original L to generate a concise single-sentence task description that starts with an action word, which is used for the joint training of the VLA model. And optional robot-side data acquisition and mapping learning steps: S25. When it is necessary to learn the relationship between perspective and action mapping or to fine-tune the model, introduce synchronous acquisition on the robot side. The humanoid robot acquires the same real environment through the RGB-D camera mounted on its head at the same or adjacent working positions, and obtains the image sequence and robot body pose from the robot's first perspective. S26. Using the same set of visual marker codes deployed in the scene, the PnP algorithm is used to calculate the real-time six-degree-of-freedom pose of the robot's head camera in a unified world coordinate system. S27. Align the human AR perspective with the robot head camera perspective in a unified world coordinate system, establish the spatiotemporal correspondence between human hand operations and robot observations, and form a transfer pair of human demonstration actions and robot observations to supervise the learning of perspective transfer and action transfer models. Among them, the precision operation scenario requires spatial calibration accuracy with translation error ≤1~2cm and posture error ≤1~2°, and the synchronization refresh rate of human AR and robot head camera data ≥30Hz; the coarse operation scenario can be relaxed to translation error ≤3~5cm and posture error ≤3~5°.

[0034] In the above technical solution, the data format collected and stored in S2 specifically includes: using "task ID + timestamp" as a unique index, prioritizing the storage of multimodal training data pairs (V) from the operator's perspective. human , L, A human ) and its metadata, the metadata including AR device pose, safety status, and whether it is a violation mark; optionally, it stores robot executable data pairs (V) obtained after viewpoint transfer and motion transfer. robot , L, A robot The data is either transferred or a feature representation used for VLA model training and domain adaptation; the data is aggregated into sequence samples according to a fixed time window of 100ms.

[0035] In one of the technical solutions, step S3 specifically includes: S31. Construct a VLA model, which includes a visual encoder, a language encoder, a multimodal fusion module, and an action decoder; S32, Transfer robot executable data pairs (V) robot , L, A robot Robot vision image V robot The visual encoder is input to extract features and obtain a visual feature vector; the task language instruction L is input to the language encoder to extract features and obtain a language feature vector; the visual feature vector and the language feature vector are input to the multimodal fusion module for joint representation learning to obtain multimodal joint features. S33. Based on the aforementioned multimodal joint features, predict robot action A using an action decoder. robot and with A robot As a supervisory signal, the VLA model is trained end-to-end using a joint loss function; Wherein, the joint loss function L total for: ; L VL The visual-language alignment loss is used to maximize the similarity between visual feature vectors and linguistic feature vectors. Its calculation method is as follows: ; L LA The language-action conditional loss is used to ensure that actions are generated from language instructions. It adopts the form of stream matching loss, and its calculation method is as follows: ; L AF The motion feasibility loss is used to constrain motions to conform to robot dynamics and joint limits. Its calculation method is as follows: ; L S Safety constraint loss, used to penalize actions that violate safety rules, is calculated as follows: ; Among them, v i Let l be the visual feature vector of the i-th sample; i and l j Let be the language feature vectors of the i-th and j-th samples, respectively; sim(·) is the cosine similarity function; T is the temperature parameter, with a value of 0.07; N is the batch sample size; σ t A represents the joint representation of multiple modes under current observations; t For the actual action block at time t; For noise-generating action blocks, ; The noise is Gaussian; τ is the noise scheduling parameter. u is the vector field predicted by the action decoder; o is the target vector field; t For current observation; q t Let q be the robot joint angle at time t; lim The robot's joint motion is limited; c is the tolerance, with a value of 0.01 rad; M is the length of the motion sequence; d t Let t be the distance from the robot's end effector to the danger zone at time t; k is the safety threshold, with a value of 0.05m; λ1, λ2, λ3 and λ4 are the weighting coefficients of each loss term, with values ​​of 0.3, 0.4, 0.2 and 0.1 respectively.

[0036] The above technical solution further constructs and trains a VLA model, specifically by constructing a VLA model architecture that includes a visual encoder, a language encoder, a multimodal fusion module, and an action decoder. The visual encoder can employ a lightweight convolutional neural network to process the robot's view image V... robot The code is encoded as a high-dimensional visual feature vector; the language encoder uses a pre-trained language model to encode the task language instructions L into language feature vectors. The robot's executable data pairs (V...) are then processed... robot , L, A robot Input the model for training. Robot's perspective image V robot Visual features are extracted by a visual encoder, and linguistic features are extracted by a language encoder for the task language command L. Both types of features are input into a multimodal fusion module, where joint representation learning is performed through a cross-attention mechanism to obtain a joint feature representation that integrates visual and linguistic information. The action decoder then predicts the robot's action A based on these joint multimodal features. robot The model is trained end-to-end using real motion data as supervision signals and a joint loss function. This joint loss function consists of four parts: visual-linguistic alignment loss to ensure consistency between visual and linguistic features in the semantic space; linguistic-motion conditional loss to ensure that the generated actions strictly follow the constraints of linguistic instructions; action feasibility loss to constrain predicted actions to conform to the robot's own dynamics and joint limits; and safety constraint loss to penalize actions that may touch dangerous areas, directly embedding safety rules into the model training process. Through joint optimization of multiple loss terms, the model simultaneously learns visual understanding, linguistic understanding, action generation, and safety constraints during training, ultimately obtaining a VLA model capable of generating safe, feasible, and accurate actions based on robot-perspective images and linguistic instructions.

[0037] like Figure 2In the above technical solution, the VLA model constructed in step S3 can adopt a lightweight architecture. The visual encoder uses ResNet-50 and outputs a 768-dimensional visual feature vector; the language encoder uses BERT and outputs a 768-dimensional language feature vector after projection alignment; the multimodal fusion module uses a cross-attention mechanism and a 2-layer Transformer structure to fuse visual features, language features, and robot body state features into a joint representation; the action decoder uses a causal Transformer to generate action block A based on the multimodal joint representation. t The action block contains a batch of future low-level actions for high-frequency control. In resource-constrained environments, a flow matching training method, a lightweight visual backbone network SigLIP, or truncation of the Transformer layer number are used to adapt to edge deployment requirements. The robot body state includes joint angles, end-effector pose, and torque information, which are normalized numerically and projected onto a linear layer to form a state token. This token is used as a prefix and input into the multimodal fusion module along with the visual token and the language token.

[0038] In the above technical solution, L LA An autoregressive form can also be used, and its specific calculation method is as follows: ;where a t The current step action is the robot action that the model needs to predict and output at time t; M is the length of the action sequence; l is the language instruction; a ＜t This represents the historical action sequence, which includes all actions generated before time t, and is used to ensure the continuity of the action sequence. This represents the conditional probability, where the model is given a language instruction l and a previously executed historical action a. ＜t Under the given conditions, predict the current step as a. t The probability value.

[0039] The above technical solution achieves joint representation and end-to-end training of vision, language, and action. Vision-language alignment improves the accuracy of instruction understanding; language-action conditional loss ensures that action generation strictly follows semantic constraints; action feasibility loss guarantees that the output action conforms to the robot's physical limitations; and safety constraint loss embeds safety rules into the model, significantly reducing deployment risks. This four-fold loss synergistic optimization increases the model's success rate in real-world tasks to over 85%, reduces the safety incident rate by 93%, and achieves a language-action matching degree of 95%.

[0040] In one of the technical solutions, step S4 specifically includes: S41. Data sources for domain-adaptive training include: Source Domain Data X sSynthetic visual data generated by AR acquisition and transfer, wherein the synthetic visual data comes from the robot's viewpoint image V obtained through viewpoint transfer in step S2. robot Or its visual feature vector; Target domain data X r Real visual data collected by the robot in real-world work scenarios; S42. Construct a domain adaptive network, wherein the domain adaptive network includes: The feature mapping network G shares the visual encoder of the VLA model in step S3 as a shared encoder, and a mapping head is linked after the shared encoder. The mapping head is a 3-layer fully connected network with a dimension change of 512→256→128. Each layer is followed by a ReLU activation function and a BatchNorm normalization layer. Domain discriminator D is a 4-layer fully connected network with dimensions changing from 128 to 128 to 64 to 1. Each layer is followed by a LeakyReLU activation function and a Dropout layer. The output layer uses a Sigmoid activation function to distinguish between the source domain and the target domain of the input features. A gradient inversion layer GRL is added after the feature mapping network G and before the domain discriminator D. The gradient inversion layer GRL implements the identity mapping during forward propagation and multiplies the gradient by -1 during backward propagation so that the shared encoder learns domain-invariant features. S43. Optimize the domain adaptive loss function L through adversarial domain adaptation training. DA and the VLA joint loss function L from step S3. total Joint optimization aligns the feature distribution of synthetic visual data with that of real visual data; The domain adaptive loss function L DA for: ; The joint optimization objective function is L joint : ; Where, x s The source domain data sample is the synthetic visual data. x r The target domain data sample is the real visual data. G(·) is a feature mapping network that maps input data to the feature space; D(·) is a domain discriminator that outputs the probability that the input feature belongs to the source domain; E represents expectation; λ DA The domain adaptive loss weight coefficient has a value of 0.1 to 0.5.

[0041] This application further refines the specific implementation of domain adaptive training and defines the domain adaptive loss function L. DA and joint optimization objective function L joint This approach fundamentally solves the problem of feature distribution discrepancies between AR synthetic visual data and real-world operational scenario data, providing a key technological guarantee for the successful migration of VLA models from the training environment to the real deployment environment. Firstly, by clearly defining the source domain data (AR synthetic visual data) and the target domain data (real-world visual data), a clear optimization objective is provided for domain-adaptive training, enabling the model to specifically narrow the gap between virtual and real data, solving the core problem of traditional methods where simulation data is difficult to directly apply to real-world scenarios. Secondly, through the visual encoder of the VLA model and the construction of a dedicated feature mapping network, seamless integration of the domain-adaptive module with the original VLA model is achieved. The shared encoder design avoids redundant training and parameters, maintaining the lightweight nature of the model structure; the three-layer fully connected mapping head further refines domain-invariant features, improving the generalization ability of feature representation. Furthermore, the introduction of a gradient reversal layer and an adversarial training mechanism allows the visual encoder to simultaneously pursue two objectives during training: minimizing the VLA task loss to ensure action prediction accuracy, and maximizing the classification error of the domain discriminator to eliminate domain-specific features. This adversarial game forces the encoder to learn robust feature representations that are both effective for the task and unaffected by domain labels, fundamentally improving the model's cross-domain generalization ability. Finally, through the synergistic optimization of domain adaptive loss and VLA joint loss, the model effectively reduces the feature distribution difference between synthetic and real data while maintaining the original task performance. Experiments show that this method can increase the success rate of real-world tasks from 50% to over 85% compared to traditional methods, significantly narrowing the gap between virtual and real data and removing technical obstacles for the practical application of VLA models in professional fields such as industry and medicine.

[0042] In one technical solution, step S5 is followed by an asynchronous inference step S6: after the VLA model is deployed, an asynchronous inference mechanism is used to decouple action block prediction from action execution, thereby improving the robot's control frequency and responsiveness; the asynchronous inference mechanism specifically includes: S61. Maintain an action queue on the robot side, the action queue storing action blocks A output by the VLA model. t =(a t ,a t+1 ,…,a t+n ), where n is the action block size, a t The underlying action instruction at time t; S62. The robot takes action a from the head of the action queue according to a fixed control cycle. t And execute; S63. Monitor the number of remaining actions in the action queue in real time. When the proportion of the number of remaining actions to the action block size n is lower than the preset threshold g, the robot collects the current observation o. t+1 It is then sent to the policy side where the VLA model is located, requesting the generation of a new action block, while continuing to execute the remaining actions in the current queue; S64, The strategy side receives observation o t+1 Then, the VLA model is run for forward inference to obtain new action blocks. And send the new action block back to the robot; S65. The robot aggregates the new action block with the remaining overlapping time step actions in the current queue and updates the action queue. The preset threshold g has a value range of 0.5 to 0.8.

[0043] This application decouples action block prediction from action execution through an asynchronous inference mechanism, significantly improving the real-time response performance of the robot after deployment. By maintaining an action queue on the robot and pre-storing action blocks for multiple time steps, the robot can execute actions continuously without waiting for each model inference, effectively eliminating execution idle time caused by inference delays. Furthermore, setting a threshold for the proportion of remaining actions triggers new block requests, allowing perception and prediction to be performed in the background in advance, achieving parallel prediction and execution pipelines. Under the same hardware conditions, this can increase the effective control frequency by 30% to 50%. Moreover, the strategy can be deployed on a remote server, with the robot only needing to execute actions, significantly reducing onboard computing power requirements, making it suitable for edge computing and low-cost robot scenarios. Finally, action aggregation through overlapping time steps ensures smooth queue updates, avoiding abrupt action switching and guaranteeing the continuity and safety of robot movement.

[0044] Furthermore, before sending the inference request, the present invention performs a certain analysis on the current observation o. t+1 A similarity judgment is made with the previous frame observation. If the distance in the joint space or image space is less than a preset threshold e, the current inference request is skipped to reduce redundant calculations. When the action queue is about to be exhausted, the latest observation is used to request a new action block to ensure safety. The similarity judgment threshold e is set according to the specific robot and sensor accuracy.

[0045] Table 1 compares the performance of the AR-VLA fusion-based robot virtual-real collaborative training method and the pure simulation / imitation learning training method provided by this invention.

[0046] Table 1 Performance Comparison Table As shown in Table 1, this invention demonstrates a comprehensive performance breakthrough compared to pure simulation / imitation learning. In terms of data acquisition, the cost per data instance has plummeted from 50,000 yuan to 5,000 yuan, a reduction of 90%, directly attributable to a large-scale data collection mode based primarily on the operator's AR perspective, eliminating the need for a robot's presence. Regarding data quality, long-tail scene coverage has jumped from 30% to 85%, an improvement of 183%, and language-action matching accuracy has increased from 70% to 95%, fully demonstrating that the perspective and action transfer mechanism can effectively preserve the richness of human demonstrations and transform them into precise actions that the robot can execute. In terms of deployment effectiveness, the success rate in real-world scenarios has increased from 50% to 85%, a 70% improvement, while the training safety incident rate has decreased from 15% to 1%, a 93% reduction. This is attributed to the domain-adaptive training effectively narrowing the gap between virtual and real environments, while the safety constraint loss in the joint loss function embeds safety rules into the model training process. In summary, this invention achieves comprehensive optimization of data collection costs, data quality, deployment success rate, and security through an AR-VLA fusion architecture, forming a complete technical closed loop from low-cost data collection to highly reliable real-world deployment.

[0047] This invention further claims a robot virtual-real collaborative training system based on AR-VLA fusion, comprising: AR devices, worn by operators, are used to perform tasks from the operator's perspective in a virtual-real fusion scene, simultaneously acquiring and producing images (V) from the operator's viewpoint. human Task language instructions (L) and operator action data (A) human Human demonstration multimodal alignment training data pairs (V human , L, A human ); The spatial registration module is used to construct an editable virtual-real fusion scene for robot task execution and to establish a unified world coordinate system; The migration module, connected to the spatial registration module, is used to migrate the V coordinate system based on the unified world coordinate system through a view migration mechanism. human Mapped to a robot-view image V aligned with the robot's viewpoint robot A through action transfer mechanism human Mapped to robot motion data A aligned with the robot's motion space robot To obtain the robot's executable data pair (V) robot , L, A robot This enables large-scale data collection without the need for robots to be present. The VLA model training module, connected to the transfer module, is used to train the robot based on the robot executable data pair (VLA). robot , L, A robotJoint representation training is performed on the VLA model, enabling the VLA model to learn the mapping relationship from robot-perspective images and task language instructions to robot actions. The domain adaptive training module, connected to the VLA model training module, is used to perform domain adaptive training on the trained VLA model to reduce the feature distribution differences between the synthetic visual data generated by AR acquisition and transfer and the real visual data acquired by the robot in the real operation scenario; the deployment module is used to deploy the VLA model after domain adaptive training to the robot in the real operation scenario, so that the robot can generate robot action output based on real visual and language input.

[0048] In the above technical solution, the spatial registration module includes: visual marker codes, which are deployed in the work scene, and the three-dimensional coordinates of each marker code in a unified world coordinate system are measured in advance; a pose calculation submodule, which is used to acquire images of the work scene in real time through the built-in camera of the AR device, detect the visual marker codes in the images and obtain their corresponding two-dimensional pixel coordinates, and calculate the real-time six-degree-of-freedom pose of the AR device in a unified world coordinate system using the PnP algorithm based on the two-dimensional pixel coordinates and the pre-measured three-dimensional coordinates; and a virtual element anchoring submodule, which is used to anchor virtual elements at a specified position in the unified world coordinate system based on the real-time six-degree-of-freedom pose, thereby constructing an editable virtual-real fusion scene. The migration module includes: a viewpoint migration submodule, which is used to transfer the operator's viewpoint image V based on the unified world coordinate system. human Mapped to a robot-view image V aligned with the robot's viewpoint robot The motion transfer submodule is used to transfer the operator's motion data A based on the unified world coordinate system. human Mapped to robot motion data A aligned with the robot's motion space robot .

[0049] In the above technical solution, the VLA model training module includes: a visual encoder, used to process the robot's view image V... robot The system is divided into three modules: a visual feature vector and a language encoder, which encodes the task language instruction L into a language feature vector; a multimodal fusion module, which performs joint representation learning on the visual and language feature vectors to obtain multimodal joint features; and an action decoder, which predicts the robot action A based on the multimodal joint features. robot and with A robot As a monitoring signal, it is used through the joint loss function L totalThe VLA model is trained end-to-end. The domain adaptive training module includes: a feature mapping network G, which shares the visual encoder of the VLA model as a shared encoder, and a mapping head is connected after the shared encoder; a domain discriminator D, used to distinguish whether the input features come from synthetic visual data or real visual data; and a gradient inversion layer GRL, placed after the feature mapping network G and before the domain discriminator D, used to invert gradients during backpropagation, enabling the shared encoder to learn domain-invariant features. The domain adaptive training module optimizes the domain adaptive loss function L through adversarial domain adaptation training. DA and the joint loss function L with VLA total Joint optimization aligns the feature distribution of synthetic visual data with the feature distribution of real visual data. The invention further claims a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned AR-VLA fusion-based robot virtual-real collaborative training method.

[0050] The VR-VLA fusion-based robot virtual-real collaborative training system also includes an asynchronous inference module, deployed on the robot, which decouples action block prediction from action execution after the VLA model is deployed. The asynchronous inference module includes an action queue for storing action block A output by the VLA model. t =(a t ,a t+1 ,…,a t+n ), where n is the size of the action block; the execution submodule is used to retrieve action a from the head of the action queue at fixed control cycles. t And execute; the monitoring submodule is used to monitor the number of remaining actions in the action queue in real time. When the proportion of the number of remaining actions to the action block size n is lower than the preset threshold g, the current observation o is triggered to be collected. t+1 It sends a request for a new action block to the policy side where the VLA model resides, while continuing to execute the remaining actions in the current queue; the aggregation submodule is used to receive the new action block returned by the policy side. It then aggregates the actions with the remaining overlapping time steps in the current queue and updates the action queue.

[0051] The VR-VLA-based robot virtual-real collaborative training system also includes a robot-side acquisition module. This module is used to acquire images of the same real environment through an RGB-D camera mounted on the humanoid robot's head when it is necessary to learn the mapping relationship between viewpoint and action or to fine-tune the model. This obtains the image sequence and robot pose from the robot's first-person perspective. Using the same set of visual markers deployed in the scene, the PnP algorithm is used to calculate the real-time six-DOF pose of the robot's head camera in a unified world coordinate system. This aligns the human AR viewpoint with the robot's head camera viewpoint in the unified world coordinate system, establishing a spatiotemporal correspondence between human hand operations and robot observations. This forms a transfer pair of human demonstration actions and robot observations, which is used to supervise the learning of viewpoint and action transfer models.

[0052] The present invention further claims a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the described AR-VLA fusion-based robot virtual-real collaborative training method.

[0053] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A robot virtual-real collaborative training method based on AR-VLA fusion, characterized in that, include: Training data pairs (V) of human demonstrations performed by operators during task execution were collected using AR devices. human ,L,A human And convert it into robot-executable data pairs (V) that are aligned with the robot's viewpoint and action space. robot ,L,A robot ); Based on the robot executable data pair (V) robot ,L,A robot Train the VLA model and perform domain adaptation on the trained VLA model to fit the real-world operating scenario; The VLA model, which has completed domain adaptive training, is deployed to a real robot to generate robot motion outputs based on real visual and task language command inputs.

2. The robot virtual-real collaborative training method based on AR-VLA fusion as described in claim 1, characterized in that, Specifically, it includes: S1. Utilize AR spatial registration technology to construct an editable virtual-real fusion scene for robot task execution and establish a unified world coordinate system; S2. In the virtual-real fusion scenario, the operator wears an AR device to perform tasks from the operator's perspective, and simultaneously collects and produces multimodal aligned training data pairs (V) from the operator's perspective. human ,L,A human Based on the unified world coordinate system, and through viewpoint transfer and motion transfer mechanisms, the multimodal aligned training data pairs (V) of the operator's viewpoint are... human ,L,A human ) is transformed into robot-executable data pairs (V) aligned with the robot's vision and robot motion space. robot ,L,A robot This enables large-scale data collection without the need for robots to be present. S3, Based on robot executable data pairs (V robot ,L,A robot Joint representation training is performed on the VLA model, enabling the VLA model to learn the mapping relationship from robot-perspective images and task language instructions to robot actions; S4. Perform domain adaptive training on the VLA model to reduce the feature distribution difference between the synthetic visual data generated by AR acquisition and transfer and the real visual data acquired by the robot in the real operation scenario. S5. Deploy the VLA model after domain adaptive training to the robot in a real-world work scenario to generate robot motion output based on real visual and language input. Among them, V human ,L,A human These are respectively operator-view images, task language instructions, and operator action data; V robot ,L,A robot These are robot vision, task language commands, and robot motion data.

3. The robot virtual-real collaborative training method based on AR-VLA fusion as described in claim 2, characterized in that, Step S1, the construction of an editable virtual-real fusion scene, is based on arranging visual marker codes and calculating the real-time six-DOF pose of the AR device using the PnP algorithm. This anchors virtual elements to specified positions within a unified world coordinate system. Specifically, this includes: Visual markers are deployed in the work environment, and the three-dimensional coordinates of each marker are measured in advance in a unified world coordinate system. The AR device worn by the operator captures images of the work scene in real time through the built-in camera, detects visual marker codes in the images, and obtains their corresponding two-dimensional pixel coordinates; Based on the two-dimensional pixel coordinates and the pre-measured three-dimensional coordinates, the real-time six-degree-of-freedom pose of the AR device in a unified world coordinate system is calculated using the PnP algorithm. Based on the real-time six-degree-of-freedom pose, virtual elements are anchored at a specified position in a unified world coordinate system to construct an editable virtual-real fusion scene.

4. The robot virtual-real collaborative training method based on AR-VLA fusion as described in claim 3, characterized in that, Step S2 specifically includes: S21. In the editable virtual-real fusion scene constructed in step S1, the operator wears an AR device to perform tasks from the operator's perspective, and the AR device's sensors acquire images V from the operator's perspective in real time. human It records the real-time pose of the AR device in a unified world coordinate system, and simultaneously detects and tracks the operator's hand movements from the AR field of view, generating operator motion data A. human Simultaneously acquire task language commands L, which are either verbally entered by the operator or selected from a preset task library, and are bound to the currently executed task during acquisition; simultaneously acquire operator-perspective images V. human Task language instructions (L), operator action data (A) human Perform timestamp alignment to form multimodal aligned training data pairs (V) from the operator's perspective. human , L, A human ); S22. Based on the aforementioned unified world coordinate system, V is transferred through a viewpoint migration mechanism. human Mapped to a robot-view image Vr aligned with the robot's viewpoint obot A through action transfer mechanism human Mapped to robot motion data A aligned with the robot's motion space robot To obtain the robot's executable data pair (V) robot , L, A robot ).

5. The robot virtual-real collaborative training method based on AR-VLA fusion as described in claim 4, characterized in that, Step S3 specifically includes: S31. Construct a VLA model, which includes a visual encoder, a language encoder, a multimodal fusion module, and an action decoder; S32, Transfer robot executable data pairs (V) robot , L, A robot Robot vision image V robot The visual encoder is input to extract features and obtain a visual feature vector; the task language instruction L is input to the language encoder to extract features and obtain a language feature vector; the visual feature vector and the language feature vector are input to the multimodal fusion module for joint representation learning to obtain multimodal joint features. S33. Based on the aforementioned multimodal joint features, predict robot action A using an action decoder. robot and with A robot As a supervisory signal, the VLA model is trained end-to-end using a joint loss function; Wherein, the joint loss function L total for: ； L VL The visual-language alignment loss is used to maximize the similarity between visual feature vectors and linguistic feature vectors. Its calculation method is as follows: ； L LA The language-action conditional loss is used to ensure that actions are generated from language instructions. It adopts the form of stream matching loss, and its calculation method is as follows: ； L AF The motion feasibility loss is used to constrain motions to conform to robot dynamics and joint limits. Its calculation method is as follows: ； L S Safety constraint loss, used to penalize actions that violate safety rules, is calculated as follows: ； Among them, v i Let be the visual feature vector of the i-th sample; l i and l j Let be the language feature vectors of the i-th and j-th samples, respectively; sim(·) is the cosine similarity function; T is the temperature parameter, with a value of 0.07; N is the batch sample size; o t For current observation, it refers to the raw sensory data acquired by the robot at time t; σ t This is the joint representation of the multimodalities under the current observations, i.e., o t The multimodal joint representation obtained after processing by the visual encoder, speech encoder, and multimodal fusion module; E represents the expectation symbol; A t For the actual action block at time t; For noise-generating action blocks, ; It is Gaussian noise; τ is the noise scheduling parameter; The vector field predicted by the action decoder; u is the target vector field; q t Let be the robot joint angle at time t; q lim Limit the movement angle of the robot's joints; c represents the tolerance, with a value of 0.01 rad. M is the length of the action sequence; d t Let t be the distance from the robot's end effector to the danger zone at time t; k is the safety threshold, with a value of 0.05m; λ1, λ2, λ3 and λ4 are the weighting coefficients of each loss term, with values ​​of 0.3, 0.4, 0.2 and 0.1 respectively.

6. The robot virtual-real collaborative training method based on AR-VLA fusion as described in claim 5, characterized in that, Step S4 specifically includes: S41. Data sources for domain-adaptive training include: Source Domain Data X s Synthetic visual data generated by AR acquisition and transfer, wherein the synthetic visual data comes from the robot view image Vrobot or its visual feature vector obtained by view transfer in step S2; Target domain data X r Real visual data collected by the robot in real-world work scenarios; S42. Construct a domain adaptive network, wherein the domain adaptive network includes: The feature mapping network G shares the visual encoder of the VLA model in step S3 as a shared encoder, and a mapping head is linked after the shared encoder. The mapping head is a 3-layer fully connected network with a dimension change of 512→256→128. Each layer is followed by a ReLU activation function and a BatchNorm normalization layer. Domain discriminator D is a 4-layer fully connected network with dimensions changing from 128 to 128 to 64 to 1. Each layer is followed by a LeakyReLU activation function and a Dropout layer. The output layer uses a Sigmoid activation function to distinguish between the source domain and the target domain of the input features. A gradient inversion layer GRL is added after the feature mapping network G and before the domain discriminator D. The gradient inversion layer GRL implements the identity mapping during forward propagation and multiplies the gradient by -1 during backward propagation so that the shared encoder learns domain-invariant features. S43. Optimize the domain adaptive loss function L through adversarial domain adaptation training. DA and the VLA joint loss function L from step S3. total Joint optimization aligns the feature distribution of synthetic visual data with that of real visual data; The domain adaptive loss function L DA for: ； The joint optimization objective function is L joint : ； Where, x s The source domain data sample is the synthetic visual data. x r The target domain data sample is the real visual data. G(·) is a feature mapping network that maps input data to the feature space; D(·) is a domain discriminator that outputs the probability that the input feature belongs to the source domain; E represents expectation; λ DA The domain adaptive loss weight coefficient has a value of 0.1 to 0.

5.

7. The robot virtual-real collaborative training method based on AR-VLA fusion as described in claim 6, characterized in that, Step S5 is followed by asynchronous inference step S6: After the VLA model is deployed, an asynchronous inference mechanism is used to decouple action block prediction from action execution, thereby improving the robot's control frequency and responsiveness; the asynchronous inference mechanism specifically includes: S61. Maintain an action queue on the robot side, the action queue storing action blocks A output by the VLA model. t =(a t ,a t+1 ,…,a t+n ), where n is the action block size, a t The underlying action instruction at time t; S62. The robot takes action a from the head of the action queue according to a fixed control cycle. t And execute; S63. Monitor the number of remaining actions in the action queue in real time. When the proportion of the number of remaining actions to the action block size n is lower than the preset threshold g, the robot collects the current observation o. t+1 It is then sent to the policy side where the VLA model is located, requesting the generation of a new action block, while continuing to execute the remaining actions in the current queue; S64, The strategy side receives observation o t+1 Then, the VLA model is run for forward inference to obtain new action blocks. And send the new action block back to the robot; S65. The robot aggregates the new action block with the remaining overlapping time step actions in the current queue and updates the action queue. The preset threshold g has a value range of 0.5 to 0.

8.

8. A robot virtual-real collaborative training system based on AR-VLA fusion, characterized in that, include: AR devices, worn by operators, are used to perform tasks from the operator's perspective in a virtual-real fusion scene, simultaneously acquiring and producing images (V) from the operator's viewpoint. human Task language instructions (L) and operator action data (A) human Human demonstration multimodal alignment training data pairs (V human , L, A human ); The spatial registration module is used to construct an editable virtual-real fusion scene for robot task execution and to establish a unified world coordinate system; The migration module, connected to the spatial registration module, is used to migrate the V coordinate system based on the unified world coordinate system through a view migration mechanism. human Mapped to a robot-view image V aligned with the robot's viewpoint robot A through action transfer mechanism human Mapped to robot motion data A aligned with the robot's motion space robot To obtain the robot's executable data pair (V) robot , L,A robot This enables large-scale data collection without the need for robots to be present. The VLA model training module, connected to the transfer module, is used to train the robot based on the robot executable data pair (VLA). robot ,L, A robot Joint representation training is performed on the VLA model, enabling the VLA model to learn the mapping relationship from robot-perspective images and task language instructions to robot actions. The domain adaptive training module is connected to the VLA model training module and is used to perform domain adaptive training on the trained VLA model to reduce the feature distribution difference between the synthetic visual data generated by AR acquisition and transfer and the real visual data acquired by the robot in the real operation scenario. The deployment module is used to deploy the VLA model, which has completed domain adaptive training, to the robot in a real-world work scenario, enabling the robot to generate robot action outputs based on real visual and verbal inputs.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot virtual-real collaborative training method based on AR-VLA fusion as described in any one of claims 1 to 7.

Images