Auxiliary teleoperation method based on vision-language-action model

Abstract

The invention belongs to the technical field of teleoperation and robot control, and particularly relates to an auxiliary teleoperation method based on a vision-language-action model. Based on a few-sample strong generalization auxiliary teleoperation framework, the key technology of the method is divided into two core stages, namely a data preprocessing stage and a strategy learning and reasoning stage, and the method comprises the following steps: S1, injecting random noise into a supervised track, and constructing intention disturbance distribution; s2, extracting a key frame of a supervised track, and constructing an intention representation of geometric perception; and S3, encoding the processed trajectory as potential embedding, and providing conditions for a vision-language-action model controller. According to the method, the visual information, the language instruction and the action strategy are fused, rapid adaptation of the teleoperation task is achieved, good generalization ability is achieved among different operators, cross-operator migration and robust control are supported, and accurate cross-operator intention recognition and strategy execution are achieved.

Description

Technical Field

[0001] This invention relates to the field of teleoperation and robot control technology, specifically to an assisted teleoperation method based on a vision-language-motion model. Background Technology

[0002] With the continuous development of embodied intelligence-related technologies, teleoperation systems have become a fundamental system driving the development of embodied intelligence, and are widely used in fields such as industrial manufacturing, remote services, and hazardous environment operations.

[0003] Teleoperation systems can leverage human multimodal perception and expert decision-making priors to collect demonstration data for training and optimizing general robot strategies. They improve task efficiency by employing a shared control model where humans provide high-level guidance and the robot executes low-level actions. However, the inherent mismatch between human and robot dynamics (especially under suboptimal operating conditions) can cause joints to approach their limits or become trapped in unusual configurations. Furthermore, individual differences in operator habits and professional backgrounds lead to highly heterogeneous trajectory distributions, posing a significant challenge to the stability of intent recognition.

[0004] To improve the stability and efficiency of intent inference, existing teleoperation schemes can be broadly categorized into fully teleoperated systems under closed-loop control, coordinated teleoperated systems based on fixed policies, and supervised control teleoperated systems where the operator acts only as a supervisor. However, these methods place a heavy burden on operators, relying heavily on training with large amounts of scenario data and struggling to generalize intent recognition across different operators and skill levels. More critically, while expanding the intent-policy library or employing traditional reinforcement learning on the teaching trajectory data of expert-operated robots can improve the efficiency, quality, and generalization ability of teleoperation to some extent, this significantly increases data collection and annotation costs, substantially increasing model computation and the supervisor's burden, without improving cross-operator generalization ability. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0006] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution: An assisted teleoperation method based on a vision-language-action model, and a few-shot strong generalization assisted teleoperation framework based on a vision-language-action model, are proposed. The key technologies are divided into two core stages: data preprocessing and policy learning and inference. The method includes the following steps: S1: Inject random noise into the supervised trajectory to construct the intended perturbation distribution: Let the supervised trajectory be... ,in The total time steps of the trajectory. For the joint angles at each time step, a trajectory-level perturbation kernel is used. Construct the intended perturbation distribution; S2: Extract keyframes from the supervised trajectory and construct a geometry-aware intent representation: given the trajectory ,in The total time steps of the trajectory. For each time step The corresponding trajectory geometric features. This is achieved through the directed Hausdorff distance. Define the geometric error of the trajectory and solve for the constraint. Minimum set of keyframes η is the preset error threshold; S3: The processed trajectory is encoded into a latent embedding to provide conditions for the visual-language-action model controller: including the visual-language-action model robustly aligning natural language instructions with visual observations to generate a consistent semantic context representation; the intent expert fusing trajectory guidance and semantic context to infer latent intents and concatenating them to form a unified multimodal context; and the action expert converting the multimodal context into action token outputs through conditional flow matching. Both the visual-language model and the action expert inherit the initial weights from the pre-trained visual-language-action model. The decoder-style intent expert adopts the same implementation as the Gemma backbone network.

[0007] As a preferred embodiment of the vision-language-action model-based assisted teleoperation method described in this invention, the specific steps in S1 for injecting random noise into the supervision trajectory to construct the intention perturbation distribution are as follows: Through trajectory-level perturbation kernel qψ(ξ) |ξ*) Supervision trajectory Construct the intentional perturbation distribution, defined as:

[0008] ; in : ; It is a single time step action command The dimension of the vector. This represents the total time steps of the trajectory. It is the first Time step corresponding The perturbation parameter block is specifically represented as follows: ; in It is the first Time step, number The perturbation variance of the dimensional action is used to control the noise intensity, and the diagonal matrix formally guarantees that the perturbations are independent between different time steps.

[0009] As a preferred embodiment of the vision-language-action model-based assisted teleoperation method described in this invention, the specific steps in S2 for extracting keyframes of the trajectory and constructing a geometrically perceived intent representation are as follows: Given trajectory For the geometric features of the trajectory, we define linear interpolation of the trajectory. ,in For the set of keyframe indices, satisfying ; Obtained through linear interpolation In the time step for: ; ; in, , For the first The time step of each keyframe, the current time step satisfy: The geometric error of the trajectory is defined as: ; in, Original trajectory points To reconstruct the trajectory The geometric distance is calculated as follows: ; The goal of keyframe extraction is to find the frame that satisfies the condition. The smallest set And its geometric error does not exceed a preset threshold. : .

[0010] As a preferred embodiment of the vision-language-action model-based assisted teleoperation method described in this invention, the specific steps of the motion expert in S3 for fine-tuning the assisted teleoperation system are as follows: Motion Expert: Converting Multimodal Contexts into Action Blocks To the state of proprioception Sum-flow matching noise term Mapped into the action embedding space to form queries; these queries drive a cross-modal attention decoder that focuses on the multimodal context as keys and values, including language. Observation and intention This architecture ensures that the generated actions strictly adhere to the semantic context and specified intent constraints;

[0011] During the training phase, a hybrid training strategy is adopted: to leverage the pre-trained visual-language model and the basic prior knowledge of action experts, low-rank adaptation is applied for training, and the low-rank decomposition formula for weight update is: ; in, The weight update amount introduced for low-rank adaptation, with dimension . , It is a low-rank dimension, much smaller than α is the scaling factor. The down-projection matrix has dimensions of . , The up-projection matrix has dimensions of . ; The forward propagation calculation formula is modified as follows: ; Right now The input features of the layer The pre-trained weight matrix is ​​inherited from the pre-trained visual-language model and action expert, with dimensions and... Consistent; Intent experts are trained with all parameters to ensure alignment with the context of the vision-language model and the strategy. Conditional flow matching is used to model the continuous transition from Gaussian prior to expert action distribution. Specifically, Stream matching training is used, through the target vector field The action gradually converges from the noise to the expert action, with the loss function being: ; in, Represents the conditional context, including language instructions. Visual observation Proprioception and the inferred operator's intention .

[0012] Compared with existing technologies, the beneficial effects of this invention are as follows: By integrating visual information, language commands, and action strategies, this invention achieves rapid adaptation to teleoperation tasks and possesses good generalization ability among different operators, thereby supporting cross-operator migration and robust control. Unlike pure inverse kinematics programming in fully teleoperation, and the disconnect between the fixed auxiliary logic of coordinated teleoperation and the human-computer interaction of supervised teleoperation, this invention innovatively injects random noise into the supervised trajectory to simulate diverse operator behaviors. Combined with an intent keyframe extraction mechanism, the processed trajectory is encoded into a latent embedding, providing conditions for a vision-language-action model controller. It not only inherits the core advantages of traditional teleoperation, such as overcoming spatial environment limitations, stable and controllable intent execution, and a certain degree of generalization ability to the working environment, but also effectively solves the problems of strong data dependence, heavy supervisor burden, and difficulty in generalizing cross-operator intent recognition, achieving accurate cross-operator intent recognition and strategy execution. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart of a few-sample strong generalization assisted teleoperation framework for an assisted teleoperation method based on a vision-language-action model according to the present invention. Figure 2 This is a graph showing the relationship between noise intensity and task success rate in an assisted teleoperation method based on a vision-language-action model according to the present invention. Figure 3 Error budgeting for an assisted teleoperation method based on a vision-language-action model according to the present invention. Relationship between task success rate and other metrics. Detailed Implementation

[0014] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0015] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0017] This invention provides an assisted teleoperation method based on a vision-language-action model (VLA). The VLA-based, low-sample, strong-generalization assisted teleoperation framework (adaptor) simulates diverse operator behaviors by injecting random noise into the supervised trajectory. It employs an intent keyframe extraction mechanism for sampling and encodes the processed trajectory as a latent embedding, providing conditions for the VLA controller. This not only inherits the core advantages of traditional teleoperation—overcoming spatial limitations and ensuring stable and controllable intent execution—but also effectively solves the problems of strong data dependence, heavy supervisor burden, and difficulty in intent recognition generalization, achieving accurate cross-operator intent recognition and policy execution.

[0018] The known conditions of this invention (i.e., input variables): multi-view RGB input Language instructions and trajectory data Variable to be determined: Motion control command. Explanation of known conditions and variable to be determined: Multi-view RGB input. In this article, it refers to image information data processed by ViT, and language commands. Refers to the operator's verbal description of the task being performed, and trajectory data. This is a demonstration trajectory for the operator. For example... Figure 1 As shown, its key technologies are divided into two core stages: data preprocessing and policy learning and inference, including the following steps: Step 1: At each time step t, the pre-trained visual-language model processes multi-view RGB input. The SigLIP encoder first extracts high-level perceptual features from the visual stream; then, these visual tokens and language tokens are integrated in a shared Transformer embedding space to achieve deep cross-modal fusion. This architecture explicitly captures the semantic correspondence between the task objective and the scene context, generating a consistent semantic context representation, including language instructions. and the semantic features of the image obtained after processing. .

[0019] Step Two: Intent experts are used to fuse trajectory data with semantic context to infer the operator's potential intent. To improve robustness against incomplete or interrupted demonstrations that may be encountered during inference, a time truncation strategy is used during training, randomly pruning the end portion of the expert trajectory. In the data preprocessing stage before training, noise perturbation is added to the expert trajectory to simulate the trajectory of a real operator, and then keyframes are extracted as teleoperation trajectory data using directed Haudos distance optimization. The teleoperation trajectory is first projected onto the language embedding space through a linear layer, and then concatenated with the visual-language model features to form a unified context, including: language... Observation and intention Simultaneously, deep semantic alignment and intent recognition are achieved through bidirectional attention. The specific steps for injecting random noise into the supervised trajectory to construct the intent perturbation distribution are as follows: Through trajectory-level perturbation kernel qψ(ξ) |ξ*) Supervision trajectory Construct the intended perturbation distribution. Defined as:

[0020] ; in : ; It is a single time step action command The dimension of the vector. This represents the total time step of the trajectory.

[0021] It is the first Time step corresponding The perturbation parameter block is specifically represented as follows: ; in It is the first Time step, number The perturbation variance of the dimension action is used to control the noise intensity.

[0022] Intent instances with varying perturbation amplitudes are extracted from this distribution, and behavior cloning is performed through supervised action regression to approximate expert policies, thereby indirectly recovering potential intentions from perturbation trajectories. Compared to traditional teleoperation, this technique significantly improves the model's robustness to intent drift and out-of-distribution scenarios, i.e., it significantly enhances the model's cross-operator generalization ability.

[0023] To evaluate whether injecting noise into the intent representation can improve trajectory diversity and cross-operator generalization, we conducted this ablation experiment on two tasks of the AlohaSim simulator. The results are as follows: Figure 2As shown in the figure, the horizontal axis represents noise intensity (0 indicates no noise, i.e., directly taking the presenter's trajectory as the intent), and the vertical axis represents the task success rate. The experimental results show a clear inverted U-shaped trend: when the noise intensity is close to 0, the model's generalization ability is insufficient, and the success rate is the lowest; conversely, excessive noise will also reduce model performance. The task success rate is highest when the noise intensity is at a moderate level.

[0024] The specific steps for extracting intent keyframes from the trajectory and constructing a geometry-aware intent representation are as follows: Given trajectory For the geometric features of the trajectory, we define linear interpolation of the trajectory. ,in For the set of keyframe indices, satisfying .

[0025] Obtained through linear interpolation In the time step for: ; ; in, , For the first The time step of each keyframe, the current time step satisfy: The geometric error of the trajectory is defined as: ; in, Original trajectory points To reconstruct the trajectory The geometric distance is calculated as follows: ; The goal of keyframe extraction is to find the frame that satisfies the condition. The smallest set And the geometric error does not exceed the preset trajectory reconstruction error. : ; By solving the above optimization problem, keyframes can be extracted, the geometric sensitivity required for high-frequency control details can be preserved, and redundant kinematic information can be filtered out. This avoids unnecessary computational overhead caused by retaining calculations at each time step. Compared with traditional teleoperation systems, this design significantly reduces the computational pressure on the model and greatly improves inference speed.

[0026] Figure 3Experimental results show the success rate of two tasks in the AlohaSim simulator as a function of trajectory reconstruction error. The horizontal axis represents trajectory reconstruction error, and the vertical axis represents task success rate. The experimental results for both tasks exhibit a typical inverted U-shaped relationship: when the error budget η is too large, it fails to fully capture the user's potential intent; conversely, when η is too small, the task success rate also decreases. Overall, selecting a moderate value of η can achieve a balance between trajectory representation accuracy and model generalization ability, reaching the highest success rate.

[0027] Step 3: The motion expert converts the multimodal context into action blocks. To understand the state of proprioception Sum-flow matching noise term Mapping to the action embedding space to form a query. Multimodal context (language) Observation and intention The query is split into key and value. These queries drive a cross-modal attention decoder, which calculates Q and K similarities as weights and then weights and fuses them into V as the output. This architecture ensures that the generated actions strictly adhere to the semantic context and the specified intent constraints.

[0028] During the training phase, a hybrid training strategy is employed. To leverage the pre-trained visual-language model and the prior knowledge of action experts, low-rank adaptation is applied for training. The low-rank decomposition formula for weight updates is: ; in, The pre-trained weight matrix is ​​inherited from the pre-trained visual-language model and action expert, with dimensions of [missing information]. , It is a low-rank dimension, much smaller than α is the scaling factor. The down-projection matrix has dimensions of . , The up-projection matrix has dimensions of . .

[0029] The forward propagation calculation formula is modified as follows: ; Right now Layer input features The intent expert is trained with all parameters to ensure alignment with the context of the visual-language model. Based on these integrated representations, the policy... Conditional flow matching is used to model the continuous transition from Gaussian priors to expert action distributions. Specifically, Training through stream matching, using target vector fields The action gradually converges from the noise to the expert action. The loss function is defined as: ; in, It represents the conditional context, including verbal instructions, visual observations, proprioception, and inferred operator intent. Compared to traditional teleoperation frameworks, this design significantly improves the quality of generated action instructions while ensuring efficient instruction generation.

[0030] Quantitative assessment of six operational tasks Table I presents the average success rate and teleoperation time for six tasks across three platforms, including the adapter. The comparison objects are: a novel few-shot strong generalization assisted teleoperation framework based on a vision-language-action model proposed in this invention (the adapter), HAJL (Coordinated Teleoperation), and full teleoperation. The "Improvement Rate" column represents the relative improvement percentage (%). The best results are highlighted in bold.

[0031]

[0032] Table I The adapter proposed in this invention achieved the highest task success rate and the shortest human active teleoperation time in all six evaluation tasks under the test system, outperforming existing teleoperation methods such as full teleoperation and HAJL. This result fully demonstrates the technical superiority of the proposed few-sample strong generalization assisted teleoperation framework (adaptor) based on a vision-language-action model: relying on the multimodal fusion mechanism of the vision-language-action model and the construction logic of intent perturbation distribution, it significantly improves the generation efficiency and quality of action control commands. At the same time, through the collaborative design of keyframe extraction and multimodal embedding, it ensures high accuracy of intent inference. It can not only quickly adapt to different operation requirements with few-sample expert trajectory input, but also effectively avoid the bias problem of existing teleoperation methods at the intent understanding level.

[0033] Cross-operator generalization analysis Table II shows the average success rate (%) of the three subjects in the table above under different practice times for the operators. SD (standard deviation) represents the statistical stability of performance under the three proficiency levels.

[0034]

[0035] Table II Table II presents the task success rate evaluation results of the three systems under different operator proficiency levels (practice durations of 30 minutes, 60 minutes, and 120 minutes). As shown in the table, the adapter proposed in this invention achieved the highest task success rate and the lowest performance fluctuation standard deviation in the cross-operator generalization evaluation scenario with different operator practice durations, comprehensively outperforming existing teleoperation methods such as full teleoperation and HAJL. The results fully demonstrate the superiority of the few-sample strong generalization assisted teleoperation framework (adaptor) based on the vision-language-action model proposed in this invention: it can significantly weaken the constraint of operator proficiency on system performance, greatly improve cross-operator generalization ability, and at the same time ensure the high stability of system operation under different user groups. Based on the aforementioned experiments, the novel few-sample strong generalization assisted teleoperation framework (adapter) based on the vision-language-action model proposed in this invention significantly outperforms existing teleoperation schemes in terms of task success rate, action generation efficiency, and cross-operator intent recognition generalization ability.

[0036] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. An assisted teleoperation method based on a vision-language-action model, characterized in that, The few-shot strong generalization assisted teleoperation framework based on a vision-language-action model has two core technologies: data preprocessing and policy learning and inference. The framework includes the following steps: S1: Inject random noise into the supervised trajectory to construct the intended perturbation distribution: Let the supervised trajectory be... ,in The total time steps of the trajectory. For the joint angles at each moment, the trajectory-level perturbation kernel is used. Construct the intended perturbation distribution; S2: Extract keyframes from the supervised trajectory and construct a geometry-aware intent representation: given the trajectory ,in The total time steps of the trajectory. For each time step The corresponding trajectory geometric features are obtained through the directed Hausdorff distance. Define the geometric error of the trajectory and solve for the constraint. Minimum set of keyframes η is the preset error threshold; S3: The processed trajectory is encoded into a latent embedding to provide conditions for the visual-language-action model controller: including the visual-language-action model robustly aligning natural language instructions with visual observations to generate a consistent semantic context representation; the intent expert fusing trajectory guidance and semantic context to infer latent intents and concatenating them to form a unified multimodal context; and the action expert converting the multimodal context into action token outputs through conditional flow matching. Both the visual-language model and the action expert inherit the initial weights from the pre-trained visual-language-action model. The decoder-style intent expert adopts the same implementation as the Gemma backbone network.

2. The assisted teleoperation method based on a vision-language-action model according to claim 1, characterized in that, In step S1, the specific steps for injecting random noise into the supervised trajectory to construct the intended perturbation distribution are as follows: Through trajectory-level perturbation kernel qψ(ξ) |ξ*) Supervision trajectory Construct the intentional perturbation distribution, defined as: ； in : ； It is a single time step action command The dimension of the vector. This represents the total time steps of the trajectory. It is the first Time step corresponding The perturbation parameter block is specifically represented as follows: ； in It is the first Time step, number The perturbation variance of the dimensional action is used to control the noise intensity, and the diagonal matrix formally guarantees that the perturbations are independent between different time steps.

3. The assisted teleoperation method based on a vision-language-action model according to claim 1, characterized in that, In step S2, the specific steps for extracting keyframes of the trajectory and constructing a geometrically-aware intent representation are as follows: Given trajectory For the geometric features of the trajectory, we define linear interpolation of the trajectory. ,in For the set of keyframe indices, satisfying ; Obtained through linear interpolation In the time step for: ； ； in, , For the first The time step of each keyframe, the current time step satisfy: The geometric error of the trajectory is defined as: ； in, Original trajectory points To reconstruct the trajectory The geometric distance is calculated as follows: ； The goal of keyframe extraction is to find the frame that satisfies the condition. Minimum set And its geometric error does not exceed a preset threshold. : 。 4. The assisted teleoperation method based on a vision-language-action model according to claim 1, characterized in that, The specific steps of the motion expert in S3 for fine-tuning the auxiliary teleoperation system are as follows: Motion Expert: Converting Multimodal Contexts into Action Blocks To the state of proprioception Sum-flow matching noise term Mapping to the action embedding space to form a query; These queries drive a cross-modal attention decoder that focuses on the multimodal context as keys and values, including language. Observation and intention This architecture ensures that the generated actions strictly adhere to the semantic context and specified intent constraints; During the training phase, a hybrid training strategy is adopted: to leverage the pre-trained visual-language model and the basic prior knowledge of action experts, low-rank adaptation is applied for training, and the low-rank decomposition formula for weight update is: ； in, The weight update amount introduced for low-rank adaptation, with dimension . , It is a low-rank dimension, much smaller than α is the scaling factor. The down-projection matrix has dimensions of . , The up-projection matrix has dimensions of . ; The forward propagation calculation formula is modified as follows: ； Right now The input features of the layer The pre-trained weight matrix is ​​inherited from the pre-trained visual-language model and action expert, with dimensions and... Consistent; Intent experts are trained with all parameters to ensure alignment with the context of the vision-language model and the strategy. Conditional flow matching is used to model the continuous transition from Gaussian prior to expert action distribution. Specifically, Stream matching training relies on the target vector field The action gradually converges from the noise to the expert action, with the loss function being: ； in, It represents the conditional context, including linguistic instructions, visual observations, proprioception, and inferred operator intent.

Images