A vision-language-action movement operation method and system based on multi-view geometric vision coding

Abstract

This invention discloses a vision-language-action mobile operation method and system based on multi-view geometric visual encoding, belonging to the field of robot perception control and embodied intelligence. This invention introduces a multi-view geometric visual encoder on the basis of the semantic visual encoder of the base vision-language-action model, combining residual projection and low-rank adaptation. Utilizing mobile operation training data containing multi-view image observations, language commands, and expert action sequences, the parameters of the residual projection module, as well as the attention layers in the vision-language model and action experts, are efficiently fine-tuned. This allows the fused features to stably participate in action generation, thereby improving the model's understanding of target positions, operation areas, and three-dimensional spatial relationships, reducing training costs, and increasing the success rate of continuous mobile operation tasks such as navigation, approach, grasping, and deployment. It is suitable for mobile operation tasks requiring spatial reasoning and real-world robot deployment scenarios.

Description

Technical Field

[0001] This invention belongs to the field of robot perception and control and embodied intelligence. More specifically, it relates to a vision-language-action movement operation method and system based on multi-view geometric visual encoding. The method enhances the spatial reasoning ability of the vision-language-action model through multi-view geometric visual encoding and is used for movement operations of continuous tasks such as navigation, target approach, grasping, carrying and dropping. Background Technology

[0002] Vision-Language-Action (VLA) models are a class of end-to-end policy models that unify visual observation, verbal commands, and robot control. In existing technologies, these models typically receive single-view or limited-view RGB images and natural language task descriptions, and directly output control quantities for the chassis, robotic arm, or end effector. While these methods perform well in object recognition, semantic understanding, and linguistic conditional control, they still have significant limitations in mobile operation scenarios requiring sophisticated spatial reasoning.

[0003] In mobile operation scenarios, robots often need to navigate to and approach a target before performing operations such as grasping, carrying, and placing. These tasks require models not only to identify "what the target is," but also to understand "where the target is located," "the relative relationships between the robot and the target, obstacles, and the target area," and "whether it is currently reachable, graspable, or placeable." However, existing VLA models rely heavily on semantic visual encoders that are pre-trained for image-text alignment. Their feature spaces are biased towards categorical semantics and text-descriptive semantics, lacking explicit modeling of depth relationships, relative pose relationships, occlusion relationships, and cross-view geometric consistency.

[0004] Especially on real robotic platforms where head-mounted and wrist-mounted cameras work together, different perspectives contain complementary spatial geometric information. The head-mounted camera is beneficial for providing global scene layout and target region information, while the wrist-mounted camera is beneficial for providing fine-grained visual information about local operating areas, grasping contact areas, and the end effector relative to the target. If images from different perspectives are input independently into a single semantic encoder without establishing correspondences between multiple perspectives, the model will struggle to form a unified, stable, and generalizable 3D spatial representation.

[0005] Furthermore, directly integrating new geometric vision branches into a pre-trained vision-language-action model often leads to significant feature distribution shifts. The attention patterns of the pre-trained vision-language model are already adapted to the statistical distribution of existing semantic visual features. If newly introduced geometric features are injected without proper adaptation, the alignment between visual and textual terms can be disrupted, resulting in training instability, attention disorder, fluctuating action output, and slow convergence. Ultimately, this leads to a low success rate in continuous motion tasks. Using full parameter fine-tuning significantly increases training costs and is not conducive to rapid adaptation and deployment on real robotic platforms.

[0006] It is evident that existing technologies suffer from technical problems such as insufficient modeling of 3D spatial structures by vision-language-action models, difficulty in stably injecting heterogeneous geometric features into pre-trained models, high training costs, and low success rates in continuous mobile operation tasks. Summary of the Invention

[0007] To address the above-mentioned deficiencies or improvement needs of existing technologies, this invention provides a vision-language-action mobile operation method and system based on multi-view geometric vision encoding, thereby solving the technical problems of insufficient modeling of three-dimensional spatial structures by vision-language-action models, difficulty in stably injecting heterogeneous geometric features into pre-trained models, high training costs, and low success rate of continuous mobile operation tasks in existing technologies.

[0008] To achieve the above objectives, according to a first aspect of the present invention, a method for training a vision-language-action model based on multi-view geometric visual encoding is provided, comprising: Acquire mobile operation training data, including multi-view image observations of the robot during mobile operations, language commands, and expert action sequences; The vision-language-action model is trained using mobile action training data, including a semantic visual encoder, a multi-view geometric visual encoder, a residual projection module, a vision-language model, an action expert, and an action generation head; During training, multi-view image observations are input into the semantic visual encoder and the multi-view geometric visual encoder respectively to obtain semantic visual features and geometric visual features. After feature mapping of the geometric visual features through the residual projection module, they are fused with the semantic visual features to obtain fused visual features. The fused visual features and language instructions are input into the vision-language model to obtain task condition representation. The task condition representation is input into the action expert to predict the velocity field. The action generation head converts the predicted velocity field into a predicted action sequence. The error between the predicted action sequence and the expert action sequence is used as the loss function for backpropagation to update the parameters. During training, the parameters of the semantic visual encoder and the multi-view geometric visual encoder are frozen. Low-rank adaptation parameters are introduced into the attention layers of the visual-language model and the action expert. Backpropagation is used to update the parameters of the residual projection module and the action generation head, as well as the low-rank adaptation parameters in the visual-language model and the action expert. Training continues until convergence, resulting in a well-trained visual-language-action model.

[0009] Furthermore, the multi-view geometric visual encoder includes a visual backbone network based on a self-supervised visual representation model and a cross-view Transformer decoder. The visual backbone network based on the self-supervised visual representation model is used to encode the images from each viewpoint in the multi-view image observation, and extract image block-level visual features that preserve spatial location information. The cross-view Transformer decoder adopts an alternating approach of intra-view attention and cross-view global attention to perform correlation modeling of image block-level visual features under different views, so as to output geometric visual features that represent the target depth relationship, relative pose relationship, scene topology and geometric constraints of the operation area.

[0010] Furthermore, the semantic visual encoder is a visual Transformer encoder based on image-text alignment pre-training, including an image block embedding layer, a position encoding layer, a multi-layer self-attention encoding block, and a semantic projection layer. The image block embedding layer divides the image in the multi-view image observation into multiple image blocks and maps each image block to a fixed-dimensional visual word. The position encoding layer obtains two-dimensional spatial position information from the multi-view image observation and injects two-dimensional spatial position information into each visual word. The multi-layer self-attention encoding block uses the visual words with injected two-dimensional spatial position information to model the relationship between each image block in the global scope to obtain semantic visual features. The semantic projection layer maps the semantic visual features to an embedding space compatible with language instructions to obtain semantic visual features aligned with language instructions. The semantic visual encoder and the multi-view geometric visual encoder work in parallel to form a semantic-geometric dual encoder architecture.

[0011] Furthermore, the residual projection module includes a first linear mapping layer, a nonlinear activation layer, a second linear mapping layer, and a residual bypass connection. The second linear mapping layer is initialized with zero or near-zero initialization, so that the residual projection module approximates an identity mapping in the early stage of training. The first linear mapping layer maps geometric visual features to a high-dimensional latent space, resulting in enhanced geometric visual features. The nonlinear activation layer introduces nonlinearity into the enhanced geometric visual features, resulting in nonlinear geometric visual features. The second linear mapping layer maps the nonlinear geometric visual features back to the embedding dimension required by the visual-language model. The residual bypass connection fuses the semantic visual features with the geometric visual features in the embedding dimension required by the visual-language model, resulting in fused visual features.

[0012] Furthermore, the vision-language model is a multi-layer Transformer conditional representation network, which combines visual features and language instructions into a joint sequence. The joint sequence is jointly encoded through a multi-layer multi-head self-attention and feedforward network, so that the language instructions modulate the attention of the fused visual features, and the fused visual features support the understanding of the language instructions in reverse, thus obtaining a task conditional representation that represents the task objective, spatial relationship and operational intention. The motion expert is a lightweight Transformer motion modeling network that uses an attention mechanism to conditionally model the task condition representation in order to predict the velocity field. The visual-language model and motion experts achieve cross-modal information interaction through a joint attention mechanism.

[0013] Furthermore, the motion generation head adopts a continuous motion generation structure based on flow matching, including a motion input projection layer, a time step encoding layer, and a motion output projection layer. The motion input projection layer is used to linearly interpolate the expert motion sequence with random Gaussian noise according to the sampling time step to obtain intermediate motion states, and maps the intermediate motion states to the latent space that the motion expert can process. The time step encoding layer is used to generate time conditions corresponding to the denoising progress. The motion expert is used to receive the task condition representation, the time conditions output by the time step encoding layer, and the intermediate motion states mapped to the latent space output by the motion input projection layer, and predicts the velocity field under the constraints of the task condition representation. The motion output projection layer is used to map the output of the motion expert into control quantities that the robot can execute, thereby predicting the motion sequence.

[0014] According to a second aspect of the present invention, a vision-language-action movement operation method based on multi-view geometric visual encoding is provided, comprising: The robot acquires current multi-view image observations and language commands, inputs a visual-language-action model trained by a multi-view geometric visual encoding method, and outputs a robot action sequence to control the robot to complete the movement operation task.

[0015] According to a third aspect of the present invention, a vision-language-action mobile operating system based on multi-view geometric visual encoding is provided, the system being deployed on a mobile operation robot platform, the mobile operation robot platform comprising: a mobile chassis, a robotic arm, and an end effector; The system includes: The multi-sensor acquisition module uses a head camera, a left wrist camera, and a right wrist camera to collaboratively acquire multi-view images and form a multi-view image observation. The instruction input module is used to acquire language instructions; The training module uses a vision-language-action model training method based on multi-view geometric vision encoding to train the vision-language-action model; The control processing unit is used to process the current multi-view image observation of the robot collected by the multi-sensor acquisition module and the command input module and the visual-language-action model trained by the language command input training module. The action generation head generates control commands for the chassis, robotic arm and end effector that match the robot configuration by starting from random Gaussian noise and iterating through multiple steps of Euler integration, which serve as the robot action sequence. The actuator control module controls the mobile chassis, robotic arm, and end effector to complete the mobile operation task based on the robot motion sequence output by the control processing unit.

[0016] According to a fourth aspect of the present invention, an electronic device is provided, including a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, is used to implement a visual-language-action model training method based on multi-view geometric visual encoding or a visual-language-action movement operation method based on multi-view geometric visual encoding.

[0017] According to a fifth aspect of the present invention, a computer product is provided, which, when running, enables the computer to execute a visual-language-action model training method based on multi-view geometric visual encoding or a visual-language-action movement operation method based on multi-view geometric visual encoding.

[0018] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: (1) This invention introduces a multi-view geometric visual encoder on the basis of the semantic visual encoder of the base vision-language-action model, so that the model no longer relies solely on semantic visual information, but can extract target depth relationships, relative pose relationships, occlusion relationships and scene topology from multi-view images, thereby improving the ability to understand three-dimensional spatial relationships. After feature mapping of geometric visual features through the residual projection module, they are fused with semantic visual features. The fused visual features and language instructions are input into the vision-language model to form a unified, stable and generalizable task condition representation, so that heterogeneous geometric features are stably injected into the pre-trained model to improve the success rate of continuous mobile operation tasks such as navigation, approach, grasping and dropping. During training, the parameters of the semantic visual encoder and the multi-view geometric visual encoder are frozen, and only the parameters of the residual projection module and the motion generation head, as well as some parameters in the visual-language model and motion expert, are updated. This preserves the multi-view geometric understanding ability obtained from pre-training and reduces training costs. By introducing LoRA low-rank adaptation parameters into the visual-language model and motion expert, the geometric priors can be quickly adapted at a lower training cost while preserving the main parameters and original knowledge of the base model. This is more conducive to the engineering deployment and task expansion of real robot platforms.

[0019] (2) The multi-view geometric visual encoder of the present invention includes a visual backbone network based on a self-supervised visual representation model and a cross-view Transformer decoder. This encoder differs from the semantic visual encoder oriented towards image-text alignment. Its focus is on recovering the structural consistency and spatial correspondence between viewpoints from multi-view images. The visual backbone network based on the self-supervised visual representation model is more sensitive to local structure, surface texture changes, and spatial layout, and is more suitable for undertaking geometric prior extraction tasks. The cross-view Transformer decoder adopts an alternating attention structure: some layers perform intra-view attention to refine the local geometric structure within a single viewpoint; other layers perform cross-view global attention to make the lexical units in one viewpoint pay attention to the corresponding regions in other views, thereby establishing spatial geometric consistency between multiple viewpoints.

[0020] (3) In this invention, the semantic visual encoder preferably adopts a visual Transformer structure based on image-text alignment pre-training. This encoder is mainly used to extract category semantics, object semantics, and scene context information that are highly related to the language description from the input image. This encoder is retained in the vision-language-action model, which can ensure that after the introduction of the geometric vision branch, the original vision-language-action model still maintains its ability to understand natural language task instructions and recognize target semantic objects.

[0021] (4) Considering that the features output by the semantic visual encoder are mainly distributed in the image-text alignment semantic space, while the features output by the multi-view geometric visual encoder are mainly distributed in the self-supervised geometric space, and there are significant differences between the two in terms of statistical distribution, feature scale, and attention adaptation, this invention sets up a residual projection module to perform stable mapping of geometric features. By using zero initialization or near-zero initialization for the second linear mapping layer, geometric features can be injected into the pre-trained vision-language-action model in a more stable way, avoiding the problems of distribution shift, attention disorder, and training instability caused by direct fusion.

[0022] (5) The visual-language model of this invention uses multi-layer multi-head self-attention and feedforward networks for joint encoding, which can simultaneously absorb semantic and geometric information in a unified multimodal context. After joint encoding, the model obtains a more complete cross-modal conditional representation of the current task that can be stably read by subsequent action experts. The action expert is a lightweight Transformer action modeling network. Through the attention mechanism, the action lexical can continuously pay attention to key information such as the target object, grasping area, placement area, and reachable path, thereby improving the consistency between action generation and scene constraints. At the same time, the visual-language model and the action expert achieve cross-modal information interaction through the joint attention mechanism.

[0023] (6) This invention performs unified continuous modeling of actions by using a motion generation head based on flow matching. The motion input projection layer maps intermediate motion states to a latent space that can be processed by motion experts. The time step encoding layer generates time conditions corresponding to the denoising progress, which helps motion experts predict the velocity field under the constraints of task condition representation. The motion output projection layer maps the output of motion experts to control quantities that can be executed by the robot. This can reduce the error accumulation caused by segmented rule control and improve the smoothness of actions and the success rate of tasks in continuous tasks such as navigation, target approach, grasping, handling and delivery.

[0024] (7) The training samples of this invention simultaneously include multi-view observations, language semantics, and action results. Under supervision, the model learns the correspondence between "multi-view spatial structure - task semantics - action control". The semantic visual encoder is responsible for answering "what is in the scene and what the task requires", the multi-view geometric visual encoder is responsible for answering "where is the target relative to the robot, is it reachable, and how to approach it more stably", the residual projection module is responsible for stably fusing the two types of information, and the visual-language model, action expert, and action generation head are responsible for converting the above information into action decisions, thereby forming a complete spatial understanding and action generation closed loop. Thus, the visual-language-action model trained using this invention can improve the success rate of mobile operations in continuous tasks such as navigation, target approach, grasping, carrying, and delivery.

[0025] (8) The system of the present invention can be deployed on a mobile operation robot platform. It can collect multi-view images through multiple cameras, obtain language commands through the command input module, and input the multi-view image observations and language commands into the trained model through the control processing unit. The action generation head generates control commands for the chassis, robotic arm and end effector that match the robot configuration by starting from random Gaussian noise and iterating through multiple Euler integrals. These commands serve as the robot action sequence. Subsequently, the execution control module drives the robot to complete navigation, target approach, grasping, transporting and delivery actions according to the action sequence. This achieves closed-loop decision-making from multi-view perception to action execution, and completes continuous mobile operation tasks in real-world scenarios. Attached Figure Description

[0026] Figure 1 This embodiment of the invention provides a structural diagram of a vision-language-action model based on multi-view geometric vision encoding. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] A method for training a vision-language-action model based on multi-view geometric visual encoding includes: Acquire mobile operation training data, including multi-view image observations of the robot during mobile operations, language commands, and expert action sequences; Use mobile operation training data to train a vision-language-action model, such as Figure 1 As shown, the vision-language-action model includes a semantic visual encoder, a multi-view geometric visual encoder, a residual projection module, a vision-language model, an action expert, and an action generation head; During training, multi-view image observations are input into the semantic visual encoder and the multi-view geometric visual encoder respectively to obtain semantic visual features and geometric visual features. After feature mapping of the geometric visual features through the residual projection module, they are fused with the semantic visual features to obtain fused visual features. The fused visual features and language instructions are input into the vision-language model to obtain task condition representation. The task condition representation is input into the action expert to predict the velocity field. The action generation head converts the predicted velocity field into a predicted action sequence. The error between the predicted action sequence and the expert action sequence is used as the loss function for backpropagation to update the parameters. During training, the parameters of the semantic visual encoder and the multi-view geometric visual encoder are frozen. Low-rank adaptation parameters are introduced into the attention layers of the visual-language model and the action expert. Backpropagation is used to update the parameters of the residual projection module and the action generation head, as well as the low-rank adaptation parameters in the visual-language model and the action expert. Training continues until convergence, resulting in a well-trained visual-language-action model.

[0029] The reason why this invention can improve spatial understanding is that the training samples simultaneously contain multi-view observations, language semantics, and action results. Under supervision, the model learns the correspondence between "multi-view spatial structure - task semantics - action control". The semantic visual encoder is responsible for answering "what is in the scene and what the task needs to operate", the multi-view geometric visual encoder is responsible for answering "where is the target relative to the robot, is it reachable, and how to approach it more stably", the residual projection module is responsible for stably fusing the two types of information, and the vision-language model and action expert are responsible for transforming the above information into action decisions, thereby forming a complete closed loop of spatial understanding and action generation.

[0030] Example 1 A vision-language-action motion manipulation method based on multi-view geometric vision encoding includes: Step S11: Obtain multi-view image observations, language commands, and expert action sequences of the robot during the mobile operation task.

[0031] Step S12: Input the multi-view image observations into the semantic visual encoder to extract semantic visual features related to the target category, task object, target region and scene context.

[0032] Step S13: Input the multi-view image observation into the multi-view geometric vision encoder to extract geometric vision features related to target depth, relative pose, occlusion relationship, platform boundary and operable area.

[0033] Step S14: The geometric visual features are mapped to a feature space compatible with the vision-language model through the residual projection module, and then fused with the semantic visual features to form fused visual features.

[0034] Step S15: Input the text lexical units corresponding to the fused visual features and language instructions into the visual-language model to obtain a task condition representation that represents the task objective, spatial relationship and operational intention.

[0035] Step S16: Input the task condition representation into the action expert to predict the velocity field. The action generation head converts the velocity field into an action sequence and updates the trainable parameters based on the loss between the prediction result and the expert action sequence.

[0036] Step S17: During the task execution phase, acquire current multi-view image observations and language commands, and use the trained model to generate control sequences for the chassis, robotic arm, and end effector to control the robot to complete navigation, target approach, grasping, handling, or delivery tasks.

[0037] In Example 1, multi-view image observation preferably includes RGB images acquired simultaneously by at least two cameras (head camera, left wrist camera, and right wrist camera) at the same time, or image sequences acquired by the same camera at consecutive times. Step S11 also acquires the robot body state, preferably including one or more of chassis pose, chassis velocity, manipulator joint angle, manipulator joint velocity, and end effector opening / closing state. The expert action sequence preferably includes one or more of chassis linear velocity, chassis angular velocity, manipulator joint position increment, manipulator joint velocity, and end effector opening / closing control quantity. Here, language instructions are used to describe the operation object, operation target, and task type; the robot body state reflects the current chassis and manipulator configuration; and the expert action sequence serves as a supervisory signal to guide the model in learning the mapping relationship from observation to action.

[0038] Training data can be collected through teleoperation teaching, whereby an operator remotely controls the robot to perform operations such as navigation, target approach, grasping, transporting, and delivery using a handle, master-slave control device, virtual reality equipment, or graphical interface, simultaneously recording multi-view images, language task labels, robot status, and control commands during the execution process. Training data can also be automatically generated through simulation environments. For example, target objects, obstacles, and target areas can be randomly configured in a simulation scene, and then expert strategies or rule controllers can execute tasks and automatically record the corresponding observations and action labels.

[0039] In a preferred embodiment, multi-view image observation includes a global scene image acquired by the head camera and local manipulation images acquired by the left and right wrist cameras. The head camera is preferably used to characterize the global relationships between the robot and the target area, obstacles, and scene layout; the wrist cameras are preferably used to characterize the local relationships between the end effector and the target object, the grasping contact area, and the placement area. Through this combination, the model can simultaneously obtain long-range navigation semantics and short-range manipulation geometry.

[0040] In a preferred embodiment, the robot's body state can be recorded as:

[0041] The expert action sequence can be denoted as:

[0042] In the formula, Indicates the chassis position. Indicates chassis speed. Indicates the joint angle of the robotic arm. Indicates the joint speed of the robotic arm. g Indicates the open / closed state of the end effector; Indicates chassis control volume. Indicates the control quantity of the robotic arm. This represents the gripper control quantity. The first expression can be understood as "the robot's current state," and the second expression as "the action the robot will perform next."

[0043] To ensure the effectiveness of data supervision, it is preferable to align multi-view images, robot states, and action sequences using a unified timestamp, and to bind language commands at the task level or time period level. For continuous tasks, a sliding time window approach can be used to combine the current observation and several future actions into training samples, so that the action generation head can learn the temporal continuity between short-term action blocks.

[0044] In Example 1, the semantic visual encoder preferably employs a visual Transformer structure based on image-text alignment pre-training, such as a SigLIP-type visual encoder. This encoder is primarily used to extract category semantics, object semantics, and scene context information that are highly relevant to the language description from the input image.

[0045] The semantic visual encoder preferably includes an image patch embedding layer, a positional encoding layer, a multi-layer self-attention encoding block, and a semantic projection layer. The image patch embedding layer is used to divide the input image into several local image patches and map each image patch to a fixed-dimensional visual word; the positional encoding layer is used to inject two-dimensional spatial positional information into each visual word; the multi-layer self-attention encoding block is used to model the relationship between each image patch globally; and the semantic projection layer is used to map the final visual representation to an embedding space compatible with language words (language instructions).

[0046] Specifically, the input image is first adjusted to a preset resolution and normalized. Then, the image is divided into multiple fixed-size patches, such as 14×14 or 16×16 squares. Each patch is then mapped to a visual word, and its corresponding positional encoding is superimposed. Subsequently, each visual word undergoes alternating processing through multiple layers of multi-head self-attention and feedforward networks to progressively extract high-level semantic features. Finally, the encoder outputs a sequence of visual words aligned with the semantics of the task language.

[0047] For example, dividing a 224×224 image into 16×16 segments yields 14×14 image blocks. The model does not process the entire image at once, but first processes the visual units corresponding to these image blocks, and then integrates the relationships between these visual units to understand the whole image.

[0048] The semantic visual encoder focuses on identifying target object categories, object attributes, scene functional areas, and task-related areas. For example, in the task of "moving to the trash can and picking up the cup from the ground," the semantic visual encoder can identify semantic entities such as trash cans, cups, the ground, desktops, corridors, or workbenches, and enhance the response to areas related to "trash cans" and "cups" based on language instructions.

[0049] Because the semantic visual encoder is pre-trained on large-scale image and text data, its output features naturally have a strong alignment with text semantics. Retaining this encoder in the system ensures that, even after introducing the geometric vision branch, the original vision-language-action model maintains its ability to understand natural language task instructions and recognize target semantic objects.

[0050] In Example 1, the multi-view geometric visual encoder preferably adopts a Pi3-type multi-view geometric coding structure, which generally consists of a self-supervised visual backbone network (a visual backbone network based on a self-supervised visual representation model) and a cross-view Transformer decoder. This encoder differs from semantic visual encoders oriented towards image-text alignment; its focus is on recovering the structural consistency and spatial correspondence between viewpoints from multi-view images.

[0051] The self-supervised visual backbone network preferably adopts a DINOv2-type network structure. For each input image, the self-supervised visual backbone network first performs image block segmentation and patch encoding, outputting image patch-level visual features that retain local texture, edge, contour, and position-sensitive information. Compared with image-text alignment models, self-supervised visual features are generally more sensitive to local structure, surface texture changes, and spatial layout, and are therefore more suitable for geometric prior extraction tasks.

[0052] After obtaining the image patch-level visual features from each viewpoint, the cross-view Transformer decoder jointly models the multi-view lexical units. Preferably, the cross-view Transformer decoder employs an alternating attention structure: some layers perform intra-view attention to refine the local geometry within a single viewpoint; other layers perform cross-view global attention to ensure that lexical units in one viewpoint focus on corresponding regions in other views, thereby establishing spatial geometric consistency among multiple views.

[0053] Furthermore, the cross-view Transformer decoder can employ a permutational equivariant design, meaning that when the order of multi-view inputs changes, the decoder output can maintain a stable transformation relationship consistent with the change in viewpoint arrangement, avoiding instability in geometric feature representation due to changes in camera input order. This structure helps improve the generalization ability of the method under different robot camera arrangements and different sensor activation combinations.

[0054] Through the above encoding process, the multi-view geometric vision encoder can output geometric visual features representing the target depth relationship, the relative pose between the target and the robot, the topological relationship between obstacles and the target area, and the geometric constraints of the platform boundary and the operable area. In other words, this encoder focuses on answering spatial geometric questions such as "Is the target behind the line of sight?", "What is the distance between the target and the obstacle?", and "Is the gripper currently aligned with the graspable area?".

[0055] The multi-view geometric vision encoder can recover a relatively stable 3D structure of a scene from multi-view images in both simulated and real-world scenarios, thus providing explicit geometric priors for subsequent action decisions.

[0056] In a preferred embodiment, the visual backbone network of the multi-view geometric visual encoder and the cross-view Transformer decoder are kept frozen or have most of their parameters frozen during fine-tuning, utilizing only the stable geometric representation capabilities obtained through large-scale pre-training. This helps to avoid damage to the geometric representation when there is insufficient data from real robots and reduces the overall training cost.

[0057] In Example 1, considering that the features output by the semantic visual encoder are mainly distributed in the image-text alignment semantic space, while the features output by the multi-view geometric visual encoder are mainly distributed in the self-supervised geometric space, and there are obvious differences between the two in statistical distribution, feature scale and attention adaptation, the present invention sets up a residual projection module to perform stable mapping of geometric features.

[0058] The residual projection module preferably includes a first linear mapping layer, a GELU activation layer, a second linear mapping layer, and a residual bypass connection. Its core calculation can be expressed as:

[0059] In the formula, x For input geometric features, z The output is the mapped features. GELU is used to first map the input features to a higher-dimensional latent space to enhance representation capabilities, and it is used to introduce nonlinearity. This step is used to map latent space features back to the embedding dimension required by the vision-language model. It can be understood as "processing first, then directly adding the original information back," thus balancing the ability to create new mappings with the preservation of original geometric information.

[0060] In a preferred embodiment, the features of the last layer or the last few layers of features from the multi-view geometric visual encoder can be concatenated, averaged, or weighted and then input into the residual projection module to more fully preserve multi-scale geometric information. The residual bypass connection allows the module to retain the main information of the input geometric features even when it has not yet learned an effective mapping in the early stages of training.

[0061] The second linear mapping layer is preferably initialized with zero or near-zero initialization. This setting ensures that training begins... Since the output is approximately zero, the module initially satisfies z≈x, meaning it starts from a state of "preserving the original geometric features as little as possible" and then gradually learns transformations suitable for the vision-language model. This strategy can significantly reduce distribution bias and gradient oscillation problems in the early stages of training.

[0062] In a preferred embodiment, the mapped geometric lexical units and semantic visual lexical units are concatenated along the sequence dimension, and then input together with the text lexical units into the visual-language model. In another embodiment, a gated addition method can also be used, that is, different weights are assigned to the geometric features according to the semantics of the task. A suitable fusion method can be selected according to the specific base model structure, but all of them do not depart from the concept of the present invention.

[0063] In Example 1, the vision-language model is preferably a multi-layer Transformer conditional representation network, used to jointly model fused visual features and language instructions. Its input includes the fused visual lexical sequence and the text lexical sequence encoded by the word embedding layer of the language model, and its output is a task conditional representation containing task objectives, spatial constraints, operation stages, and environmental context.

[0064] Specifically, the visual-language model first concatenates visual and textual lexical units along the sequence dimension, or inserts them into a unified context sequence according to a pre-defined template. Then, it performs joint encoding through multi-layer multi-head self-attention and a feedforward network, enabling textual semantics to modulate visual region attention, while visual geometry and semantic features can in turn support language conditional understanding. After joint encoding, the model obtains a more complete cross-modal conditional representation of the current task.

[0065] In a preferred embodiment, the visual-language model may employ an 18-layer Transformer structure, but the invention is not limited to a specific number of layers. The key is that the model needs to be able to simultaneously absorb semantic and geometric information within a multimodal unified context and output a task condition representation that can be stably read by subsequent action experts.

[0066] The motion expert is preferably a lightweight Transformer motion modeling network. Its input includes task condition representations, intermediate motion states, and time step encodings from a vision-language model; its output is an intermediate motion representation or a velocity field. Through a conditional attention mechanism, the motion expert enables action lexical units to continuously focus on key information such as the target object, grasping area, placement area, and reachable path, thereby improving the consistency between motion generation and scene constraints.

[0067] Temporal step encoding is used to characterize the current denoising stage of motion generation. Preferably, time step t is first mapped to a temporal conditional vector using sinusoidal position encoding or other temporal embedding methods, and then fed into the motion expert to modulate the attention layer, normalization layer, or feedforward layer. In this way, the motion expert can decide whether to focus more on the global task structure or local motion details based on the current denoising progress.

[0068] During the inference phase, it is preferable to first use a vision-language model to perform a prefix encoding on multi-view images and language instructions to obtain a cacheable task condition representation. Then, this task condition representation can be reused in each integration step of action generation, without having to repeatedly encode all visual and language inputs, thereby reducing inference latency and improving deployment efficiency.

[0069] In Example 1, the present invention preferably employs a parameter-efficient fine-tuning method to adapt the base vision-language-action model, rather than performing a full update of all parameters. The basic idea is to retain the large-scale pre-trained knowledge learned by the semantic visual encoder and the main body of the vision-language model, and to complete the injection of geometric priors and task adaptation through a small number of newly added parameters and a small number of trainable modules.

[0070] Specifically, the visual backbone network of the multi-view geometric visual encoder and the cross-view Transformer decoder are preferably frozen, the main parameters of the semantic visual encoder are frozen, the residual projection module is trained, and LoRA low-rank adaptation parameters are introduced into the attention layers of the visual-language model and the action expert. In one preferred embodiment, the action expert can be trained in its entirety; in another preferred embodiment, the action expert is trained only on a portion of the attention layers or feedforward layers.

[0071] The LoRA low-rank adaptation can be expressed as:

[0072] In the formula, Let A represent the original frozen weights, and let B represent the newly added trainable first and second low-rank matrices. r Represents the rank of a low-rank decomposition. αThis represents the scaling factor. LoRA can be understood as: instead of directly and significantly modifying the original model, it adds a set of small "correction factors" next to the original weights, achieving adaptation with fewer parameters.

[0073] Preferably, the LoRA parameter is applied to both the query projection layer and the numerical projection layer. The former determines which terms the model currently focuses on, while the latter determines what information the model extracts from these terms. When multi-view geometric features are introduced, these two types of projection layers have a crucial impact on whether the model can effectively read geometric information.

[0074] During training, a sample is first taken from the training data to obtain the corresponding multi-view image observations, language commands, robot states, and target action sequences. a Subsequently, random Gaussian noise was sampled. ε and time step t To make it easier to understand, we can define the intermediate action states. This can be viewed as a "transitional state between pure noise and real motion." Its definition is:

[0075] The corresponding real velocity field Defined as:

[0076] The model represents the task conditions. c Predicting the velocity field of intermediate motion states Accordingly, the training loss is denoted as Its expression is:

[0077] The mean squared error is preferred as the base loss. This loss can be understood as "making the adjustment direction predicted by the model as close as possible to the actual adjustment direction".

[0078] In a preferred embodiment, the optimizer can be Adam or AdamW, beta1 can be set to 0.9, beta2 can be set to 0.95, and the learning rate can be set to 1×10⁻⁶. -5 Alternatively, other values ​​matching the model size can be used, along with learning rate warm-up, cosine decay, and gradient clipping strategies for training. This invention does not strictly limit the specific hyperparameters, as long as stable fine-tuning can be achieved.

[0079] Through the above training process, the model can gradually learn how to read, absorb and utilize explicit spatial priors from the multi-view geometric visual encoder on the basis of its original semantic understanding capabilities without destroying the main structure of the original base VLA model, thereby improving its action decision-making ability in spatially related tasks.

[0080] In Example 1, the motion generation head preferably adopts a continuous motion generation structure based on flow matching, including a motion input projection layer, a time step encoding layer, and a motion output projection layer. The motion input projection layer is used to map intermediate motion states to a latent space that can be processed by a motion expert; the time step encoding layer is used to generate temporal conditions corresponding to the denoising progress; the motion expert is used to predict the velocity field under the constraints of task condition representation; and the motion output projection layer is used to map the motion expert output into control quantities that can be executed by the robot.

[0081] The inference phase preferably includes the following steps. Step S71: Based on the current multi-view image observations and language instructions, the task condition representation is obtained through a semantic visual encoder, a multi-view geometric visual encoder, a residual projection module, and a visual-language model. c .

[0082] Step S72, initialize the action state from random Gaussian noise, and denote the initial state as... Here It is not the final action, but the starting point of the action generation process.

[0083] Step S73, at the k-th discrete time step The current action state Time step and task condition representation c Input the motion expert and motion generation head to obtain the predicted velocity field. The velocity field here can be understood as "in which direction the current action still needs to be adjusted".

[0084] Step S74: Perform numerical integration update on the action state based on the predicted velocity field. The Euler integration method is preferred, and its discrete update formula can be written as:

[0085] In the formula, Indicates the integration step size. Indicates the first k The state of movement of the step, This represents the updated action state. By integrating stepwise from t=1 to t=0, the action state can be gradually evolved from initial noise into an effective action sequence that meets the task requirements.

[0086] In step S75, after integration, the final action sequence is obtained and parsed into chassis speed control, robotic arm joint control, and end effector control. Subsequently, the execution control module drives the robot to complete navigation, target approach, grasping, transport, and delivery actions according to the action sequence. For action block control scenarios, several future control steps can be output at once and then executed in an overlapping or rolling manner.

[0087] For example, in the task of "moving to the trash can, picking up a cup from the ground, and then dropping it," the vision-language model first understands the semantic and geometric relationships between the trash can, the cup, and surrounding obstacles based on images from the head and wrist cameras. The motion generation head then outputs continuous velocity control values ​​for the chassis to approach the trash can, and continues to output robotic arm extension, gripper closure, and target drop actions after the robot reaches the graspable position. Because the multi-view geometric vision encoder explicitly models the spatial relationship between the target and the robot, it can reduce problems such as proximity deviation, misalignment of the grasp, and unstable drop point.

[0088] In real-world mobile operation scenarios, this invention can continuously complete operations such as navigation, target approach, grasping, and deployment, thus demonstrating good real-world execution capabilities.

[0089] As can be seen from the above embodiments, the present invention does not simply add an extra visual branch in front of the original VLA model, but rather constructs a complete link around the stable injection of multi-view geometric priors: first, the semantic visual encoder retains the original task semantic understanding ability, then the multi-view geometric visual encoder provides explicit spatial structure information, then the residual projection module completes the distribution alignment, then the vision-language model forms a unified task condition representation, and finally the action expert and action generation head output a continuous action control sequence, thereby realizing closed-loop decision-making from multi-view perception to action execution.

[0090] The above description is merely a preferred embodiment of the present invention. Those skilled in the art can make various substitutions, improvements, or adjustments to the number of cameras, the selection of the visual encoder, the backbone structure of the geometric encoder, the feature fusion method, the low-rank adaptation position, the training hyperparameters, the definition of the motion space, and the robot platform configuration without departing from the concept of the present invention; all such substitutions, improvements, or adjustments should fall within the protection scope of the present invention.

[0091] Example 2 A vision-language-action mobile operating system based on multi-view geometric visual encoding, the system being deployed on a mobile operation robot platform, the mobile operation robot platform comprising: a mobile chassis, a robotic arm, and an end effector; The system includes: The multi-sensor acquisition module uses a head camera, a left wrist camera, and a right wrist camera to collaboratively acquire multi-view images and form a multi-view image observation. The instruction input module is used to acquire language instructions; The training module uses a visual-language-action model training method based on multi-view geometric visual encoding proposed in this invention to train the visual-language-action model. The control processing unit is used to process the current multi-view image observation of the robot collected by the multi-sensor acquisition module and the command input module and the visual-language-action model trained by the language command input training module. The action generation head generates control commands for the chassis, robotic arm and end effector that match the robot configuration by starting from random Gaussian noise and iterating through multiple steps of Euler integration, which serve as the robot action sequence. The actuator control module controls the mobile chassis, robotic arm, and end effector to complete mobile operation tasks based on the robot motion sequence output by the control processing unit. These mobile operation tasks include one or more of the following: autonomous navigation to a target area, identification of a target object, approaching the target, performing a grasping action, and placing the target object into a designated area.

[0092] The control and processing unit can be an onboard computing platform, an edge computing platform, or a remote inference server.

[0093] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a vision-language-action model based on multi-view geometric visual encoding, characterized in that, include: Acquire mobile operation training data, including multi-view image observations of the robot during mobile operations, language commands, and expert action sequences; The vision-language-action model is trained using mobile action training data, including a semantic visual encoder, a multi-view geometric visual encoder, a residual projection module, a vision-language model, an action expert, and an action generation head; During training, multi-view image observations are input into the semantic visual encoder and the multi-view geometric visual encoder respectively to obtain semantic visual features and geometric visual features. After feature mapping of the geometric visual features through the residual projection module, they are fused with the semantic visual features to obtain fused visual features. The fused visual features and language instructions are input into the vision-language model to obtain task condition representation. The task condition representation is input into the action expert to predict the velocity field. The action generation head converts the predicted velocity field into a predicted action sequence. The error between the predicted action sequence and the expert action sequence is used as the loss function for backpropagation to update the parameters. During training, the parameters of the semantic visual encoder and the multi-view geometric visual encoder are frozen. Low-rank adaptation parameters are introduced into the attention layers of the visual-language model and the action expert. Backpropagation is used to update the parameters of the residual projection module and the action generation head, as well as the low-rank adaptation parameters in the visual-language model and the action expert. Training continues until convergence, resulting in a well-trained visual-language-action model.

2. The method for training a vision-language-action model based on multi-view geometric visual encoding as described in claim 1, characterized in that, The multi-view geometric visual encoder includes a visual backbone network based on a self-supervised visual representation model and a cross-view Transformer decoder. The visual backbone network based on the self-supervised visual representation model is used to encode images from each viewpoint in multi-view image observation, extracting image block-level visual features that preserve spatial location information. The cross-view Transformer decoder uses an alternating approach of intra-view attention and cross-view global attention to perform correlation modeling of image block-level visual features under different views, so as to output geometric visual features that represent the target depth relationship, relative pose relationship, scene topology, and geometric constraints of the operation area.

3. A method for training a vision-language-action model based on multi-view geometric visual encoding as described in claim 1 or 2, characterized in that, The semantic visual encoder is a visual Transformer encoder based on image-text alignment pre-training, which includes an image block embedding layer, a position encoding layer, a multi-layer self-attention encoding block and a semantic projection layer. The image block embedding layer divides the image in multi-view image observation into multiple image blocks and maps each image block to a fixed-dimensional visual word. The location coding layer obtains two-dimensional spatial location information from multi-view image observations and injects two-dimensional spatial location information into each visual word; the multi-layer self-attention coding block uses the visual words with injected two-dimensional spatial location information to model the relationship between each image block in the global scope and obtain semantic visual features. The semantic projection layer maps semantic visual features to an embedding space compatible with language instructions, resulting in semantic visual features aligned with language instructions. The semantic visual encoder and the multi-view geometric visual encoder work in parallel to form a semantic-geometric dual encoder architecture.

4. A method for training a vision-language-action model based on multi-view geometric visual encoding as described in claim 1 or 2, characterized in that, The residual projection module includes a first linear mapping layer, a nonlinear activation layer, a second linear mapping layer, and a residual bypass connection. The second linear mapping layer is initialized with zero or near-zero initialization, so that the residual projection module approximates an identity mapping in the early stage of training. The first linear mapping layer maps geometric visual features to a high-dimensional latent space, resulting in enhanced geometric visual features. The nonlinear activation layer introduces nonlinearity into the enhanced geometric visual features, resulting in nonlinear geometric visual features. The second linear mapping layer maps the nonlinear geometric visual features back to the embedding dimension required by the visual-language model. The residual bypass connection fuses the semantic visual features with the geometric visual features in the embedding dimension required by the visual-language model, resulting in fused visual features.

5. A method for training a vision-language-action model based on multi-view geometric visual encoding as described in claim 1 or 2, characterized in that, The vision-language model is a multi-layer Transformer conditional representation network. It combines visual features and language instructions into a joint sequence, and encodes the joint sequence through a multi-layer multi-head self-attention and feedforward network. This enables the language instructions to modulate the attention of the fused visual features, and the fused visual features in turn support the understanding of the language instructions, thus obtaining a task conditional representation that represents the task objective, spatial relationship and operational intention. The motion expert is a lightweight Transformer motion modeling network that uses an attention mechanism to conditionally model the task condition representation in order to predict the velocity field. The visual-language model and motion experts achieve cross-modal information interaction through a joint attention mechanism.

6. A method for training a vision-language-action model based on multi-view geometric visual encoding as described in claim 1 or 2, characterized in that, The motion generation head employs a continuous motion generation structure based on flow matching, comprising a motion input projection layer, a time step encoding layer, and a motion output projection layer. The motion input projection layer is used to linearly interpolate the expert motion sequence with random Gaussian noise at sampling time steps to obtain intermediate motion states, and maps the intermediate motion states to a latent space that the motion expert can process. The time step encoding layer is used to generate time conditions corresponding to the denoising progress. The motion expert receives the task condition representation, the time conditions output by the time step encoding layer, and the intermediate motion states mapped to the latent space output by the motion input projection layer, and predicts the velocity field under the constraints of the task condition representation. The motion output projection layer is used to map the output of the motion expert into control quantities that the robot can execute, thereby predicting the motion sequence.

7. A vision-language-action movement operation method based on multi-view geometric visual encoding, characterized in that, include: The robot acquires current multi-view image observations and language commands, inputs the visual-language-action model trained by the visual-language-action model training method based on multi-view geometric visual encoding as described in any one of claims 1-6, and outputs a robot action sequence to control the robot to complete the movement operation task.

8. A vision-language-action mobile operating system based on multi-view geometric visual encoding, characterized in that, The system is deployed on a mobile operation robot platform, which includes: a mobile chassis, a robotic arm, and an end effector. The system includes: The multi-sensor acquisition module uses a head camera, a left wrist camera, and a right wrist camera to collaboratively acquire multi-view images and form a multi-view image observation. The instruction input module is used to acquire language instructions; The training module trains a visual-language-action model using a visual-language-action model training method based on multi-view geometric visual encoding as described in any one of claims 1-6. The control processing unit is used to process the current multi-view image observation of the robot collected by the multi-sensor acquisition module and the command input module and the visual-language-action model trained by the language command input training module. The action generation head generates control commands for the chassis, robotic arm and end effector that match the robot configuration by starting from random Gaussian noise and iterating through multiple steps of Euler integration, which serve as the robot action sequence. The actuator control module controls the mobile chassis, robotic arm, and end effector to complete the mobile operation task based on the robot motion sequence output by the control processing unit.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, is used to implement the visual-language-action model training method based on multi-view geometric visual encoding as described in any one of claims 1-6 or the visual-language-action movement operation method based on multi-view geometric visual encoding as described in claim 7.

10. A computer product, characterized in that, When the product is running, it enables the computer to execute the visual-language-action model training method based on multi-view geometric visual encoding as described in any one of claims 1-6, or the visual-language-action movement operation method based on multi-view geometric visual encoding as described in claim 7.

Images