A robot control method and system based on visual language action agent

Abstract

The application discloses a robot control method and system based on a visual language action intelligent agent, and belongs to the technical field of intelligent robot control. The method comprises the following steps: acquiring environment visual observation and natural language task instructions, and extracting and fusing a multi-modal representation vector through the first K layers of a cloud native multi-modal visual language backbone network; concatenating the vector and a robot ontology perception vector to construct a joint conditional feature; iteratively generating a continuous control action sequence and driving the robot to execute by a flow matching driven pure cross attention diffusion transformer with the joint conditional feature as a condition; and performing dynamic sub-task decomposition and re-planning by a thinking chain reasoning mechanism during execution. Through the lightweight design at the architecture level, the flow matching diffusion strategy and the two-stage training paradigm, the application realizes the extreme compression of model parameters and the timing consistency of action generation while maintaining the multi-modal semantic understanding capability, and takes into account the intelligent level and real-time control performance.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot control technology, specifically a robot control method and system based on visual language action intelligent agents. Background Technology

[0002] As robot applications shift from structured industrial production lines to unstructured home environments, the core technological challenge facing these systems lies in how to reliably translate high-level, ambiguous human natural language commands into low-level, continuous, and physically feasible sequences of robot control actions. Existing technological solutions suffer from the following four constraints at the fundamental principle level.

[0003] First, the tightly coupled architecture of perception, decision-making, and execution leads to conflicts in the representation space. Traditional end-to-end models require a single neural network to simultaneously handle multiple functions, including scene semantic understanding, task logical reasoning, and precise motion control. From the perspective of representation learning, semantic understanding requires the feature space to have high invariance to categories and relationships, while motion control requires the feature space to have high sensitivity to pose, force, and velocity. These two requirements for feature distribution are fundamentally conflicting. When the model is forced to fit two tasks with significantly different distributions on the same embedding manifold, the training signals interfere with each other, causing the model to experience representation collapse and performance avalanche when faced with environmental perturbations or task variations outside the training distribution.

[0004] Secondly, modal adaptation of pre-trained visual language models to the robot's action space relies on a large number of trainable parameters. To adapt frozen or fine-tuned visual language models to action generation tasks, existing solutions generally insert cross-modal projection matrices, multilayer perceptron adapters, or reparameterization modules between the two. From an information theory perspective, such structures essentially align two feature manifolds that were originally trained independently. The number of parameters required increases superlinearly with the geodesic distance between the manifolds, directly leading to bloated models, difficult edge deployment, and high inference latency, making it difficult to meet the dual constraints of hardware cost and real-time response in home environments.

[0005] Third, the autoregressive paradigm commonly used in action sequence generation violates the principle of temporal consistency in continuous physical motion. Existing action generation models based on Transformer decoders mostly employ a time-step autoregressive prediction strategy, meaning that the action component at each moment is generated only based on the hidden states of previous moments. However, the smoothness of physical motion trajectories requires that the position, velocity, and even acceleration of adjacent control cycles satisfy higher-order continuity constraints. Because autoregressive models lack explicit probabilistic modeling of the global temporal structure, the trajectories they generate exhibit high-frequency jitter at the second derivative level. This necessitates post-processing smoothing filters for actual execution, but the filtering operation introduces phase delay, further impairing the execution accuracy of dynamic tasks.

[0006] Fourth, there is a mismatch between the communication latency of a pure cloud-based inference architecture and the real-time requirements of local control loops. In a typical home wireless network environment, the round-trip latency for uploading all visual data to the cloud for inference and returning control commands is typically in the hundreds of milliseconds, while the effective control cycle for precise robotic arm operations generally requires less than 50 milliseconds. This latency mismatch causes a sharp drop in the system's success rate in tasks requiring rapid visual servoing or dynamic grasping, constituting a fundamental contradiction between cloud intelligence and local execution.

[0007] Therefore, the industry urgently needs a technical solution that can systematically resolve the above contradictions from three levels: model architecture, training paradigm and system deployment. This solution should maintain the depth of cross-modal semantic understanding while achieving extreme compression of model parameters, temporal consistency generation of action trajectories, and low-latency closed-loop control of cloud-edge collaboration. Summary of the Invention

[0008] The purpose of this invention is to provide a robot control method and system based on visual language action intelligence, in order to solve the problems of model bloat and real-time response constraints, low cross-modal adaptation efficiency, insufficient sequential continuity of generated actions, and cloud-end feedback delay mismatch in the prior art.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A robot control method based on visual language action agents includes the following steps: Step S1: Obtain the visual observation sequence and natural language task instructions of the robot's working environment, and upload the visual observation sequence and natural language task instructions to the cloud; the native multimodal large language model deployed in the cloud performs task reasoning and decomposition based on the visual observation sequence and natural language task instructions, generates a sequence of structured sub-tasks containing multiple temporal dependencies, and sends the sub-task sequence to the robot through a standardized data interface. Step S2: In the current execution cycle, the robot receives a structured subtask to be executed from the cloud; it inputs the real-time visual observations of the current environment and the language description of the structured subtask into a lightweight visual language action model deployed locally on the robot; using the first K layers of the native multimodal visual language backbone network in this model, it performs cross-modal fusion encoding on the visual observations and the language description of the subtask, and extracts the fused multimodal representation vector output by the Kth layer; where K is a truncated layer index that is less than the total depth of the backbone network and is determined based on the cross-modal semantic alignment strength. Step S3: Concatenate the fused multimodal representation vector with the ontology perception vector representing the robot's current kinematic and dynamic state along the feature dimension to construct joint conditional features; Step S4: Using joint conditional features as conditions, a continuous control action sequence covering a future preset time window is iteratively generated through a flow-matched driven cross-modulation diffusion converter, wherein the diffusion converter continuously injects conditional information into the denoising evolution process of the action sequence using a pure cross-attention mechanism. Step S5: Output the continuous control action sequence to the robot actuator to drive the robot to complete the physical operation corresponding to the natural language task instruction; Step S6: During the execution of physical operations, the multimodal representation vector is fused with real-time visual feedback and task execution status updates, and the unfinished natural language task instructions are dynamically decomposed or replanned through the thought chain reasoning mechanism.

[0010] According to the above technical solution, in step S1, when the native multimodal large language model deployed in the cloud performs task reasoning and decomposition, the truncated layer index K of its internal backbone network is determined in the following way: calculate the mutual information or cross-modal attention alignment metric between the visual features and language features output by each intermediate layer of the backbone network, and select the layer index that has reached a local maximum value and has not yet entered the deep layer of pure language logic reasoning as K.

[0011] According to the above technical solution, in step S2, before inputting the current environmental visual observation into the robot's local lightweight visual language action model, the following steps are also included: performing a pixel-level inverse rearrangement downsampling operation on each image frame of the current visual observation to compress the spatial dimension of the visual token sequence to the preset compression ratio before input. The compression ratio is determined by the square of the scaling factor of the inverse rearrangement operation.

[0012] According to the above technical solution, in step S4, the cross-modulation diffusion converter optimizes its internal velocity field prediction network by minimizing the flow matching loss function during the training phase. The flow matching loss function is defined as the expected mean square error between the predicted velocity field and the target flow field on the noise disturbance action state distribution, and its mathematical expression is:

[0013] in, This represents a sequence of real actions sampled from the robot's teaching trajectory. This represents the fused multimodal representation vector. Represents the ontology perception vector. Indicates the interpolated weights Intermediate action states are generated by mixing real action sequences with Gaussian noise. Indicates by parameters Parameterized velocity field This represents the target flow direction vector that guides the intermediate action state to the actual action sequence.

[0014] According to the above technical solution, the intermediate action state Generated based on the following linear interpolation:

[0015] in, This represents an independent and identically distributed noise vector sampled from a standard multivariate Gaussian distribution. Interpolation weights, sampled and truncated from a specified probability distribution, are used to control the degree of noise perturbation in the action sequence.

[0016] According to the above technical solution, the stacking method of the pure cross-attention mechanism in step S4 is as follows: in each transformation layer of the diffusion transformer, the action sequence embedding after noise perturbation is used as the query vector, the linear projection of the joint conditional features is used as the key vector and value vector, the cross-attention output is calculated and the action sequence representation is updated through residual connection and feedforward network, and there is no self-attention sub-layer between layers.

[0017] According to the above technical solution, the native multimodal visual language backbone network, the integration module for cascading and fusing multimodal representation vectors and ontology perception vectors, and the diffusion transformer adopt a two-stage progressive training paradigm for parameter optimization: In the first stage, all learnable parameters of the backbone network are frozen, and only the parameters of the ensemble module and the diffusion transformer are updated. The training objective is to minimize the flow matching loss function in order to establish the alignment between the action generation space and the multimodal semantic space. In the second stage, all learnable parameters of the backbone network are unfrozen, and joint fine-tuning is performed on the parameters of the backbone network, integration modules, and diffusion transformers. The training objective is to minimize the flow matching loss function in order to adapt to downstream operation tasks while preserving semantic priors.

[0018] According to the above technical solution, the thought chain reasoning mechanism in step S6 includes: constructing a contextual prompt sequence containing task objectives, environmental states, and execution history based on the returned visual feedback and the current fused multimodal representation vector; inputting the contextual prompt sequence into the cloud-based multimodal large language model to guide it to output thought chain text that conforms to the preset reasoning pattern. The thought chain text sequentially contains the following parsable fields: subtask semantic description, spatial constraints, action primitive type, and failure recovery strategy identifier; based on the failure recovery strategy identifier, the cloud-based multimodal large language model automatically triggers task replanning and updates the natural language task instructions sent to step S2 when an execution anomaly is detected.

[0019] According to the above technical solution, in step S5, the continuous control action sequence is generated by the cross-modulation diffusion converter from the initial state of pure noise through iterative denoising based on the flow matching mechanism. The continuous control action sequence includes synchronous control instructions for the six-dimensional pose variables of the robot's dual-arm end effectors in Cartesian space and the velocity variables of the mobile chassis in the plane. In each denoising iteration, the diffusion converter only uses joint conditional features as cross-attention modulation signals to maintain the temporal consistency of the generated trajectory.

[0020] A robot control system based on a visual language action agent includes: The cloud-based inference and planning unit is configured to: receive environmental image sequences collected by a visual sensor and natural language task instructions input by the user; dynamically decompose the natural language task instructions into a set of structured subtasks with temporal dependencies by running a multimodal large language model deployed in the cloud and invoking a thought chain inference mechanism; and distribute description information of the structured subtask set through a standardized data interface. The local execution control unit, deployed on the robot body or its associated edge computing device, includes: The visual compression encoder performs pixel-level inverse rearrangement downsampling on the input image to reduce the visual token space dimension input to subsequent networks; The lightweight multimodal fusion encoder uses the first K layers of the native multimodal visual language backbone network to perform cross-modal fusion encoding of visual features and subtask language descriptions, and outputs the fused multimodal representation vector of the Kth layer, where K is a truncated layer index that is less than the total depth of the backbone network and is determined based on the cross-modal semantic alignment metric. The state integrator cascades the multimodal representation vectors and the robot's ontological perception vectors along the feature dimension to construct joint conditional features; The action generator, based on joint conditional features, iteratively generates a continuous sequence of control actions covering a preset future time window through a pure cross-attention diffusion transformer driven by flow matching. The actuator converts the continuous control action sequence into robot joint-level or end-effector pose-level control commands, and transmits the execution status and visual observations back to the cloud-based inference and planning unit in real time to form a closed-loop feedback.

[0021] Compared with the prior art, the present invention has the following beneficial effects: In terms of model lightweighting and real-time performance, by combining deep truncation of the native multimodal backbone network, spatial compression of visual tokens, and pure cross-attention diffusion transformer, the total number of model parameters is controlled to below one billion, the inference frequency is increased to above 15 Hz, and the graphics processor memory usage is compressed to the range that consumer-grade hardware can bear, providing a feasible path for the low-cost commercial deployment of home service robots.

[0022] Regarding the temporal consistency of motion generation, a flow matching diffusion strategy is adopted to replace the autoregressive model, explicitly modeling the global temporal probability distribution of the motion sequence. The generated trajectory maintains high-order continuity in terms of position, velocity, and acceleration, and can directly drive the robot hardware without the need for post-processing smoothing filters, thus eliminating the phase delay introduced by filtering.

[0023] In terms of semantic preservation and rapid task adaptation, the two-stage progressive training paradigm enables the model to converge with only a very small number of manually taught trajectories when introducing new operational tasks. Moreover, the training process does not cause catastrophic forgetting or semantic drift of the cross-modal semantic space accumulated in the pre-training stage, thus ensuring the robust generalization ability of the model under non-distributed out-of-test conditions.

[0024] In terms of cloud-edge collaboration and closed-loop fault tolerance, computationally intensive high-level semantic reasoning and task planning are offloaded to the cloud, while latency-sensitive action generation and closed-loop control are retained at the local edge. Collaborative work is achieved through standardized interfaces and real-time feedback mechanisms, which not only ensures the level of intelligence required for complex tasks, but also meets the stringent real-time requirements of fine-grained operations.

[0025] In terms of dynamic replanning and anomaly recovery, the task decomposition and failure recovery mechanism based on the thinking chain enables the system to perform precise local replanning when the execution of sub-tasks fails, without having to perform a costly global restart of the entire task process. This significantly improves the robot's task success rate and user experience in unstructured and dynamically changing home environments. Attached Figure Description

[0026] Figure 1 This is a flowchart of the robot control method of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example 1 The technical solution provided by this invention is divided into a cloud-based inference planning unit and a robot-local execution control unit. The cloud-based inference planning unit receives environmental visual observations and user natural language commands. By invoking the native multimodal visual language backbone network and thought chain inference mechanism, it dynamically decomposes high-level task commands into structured sub-task sequences and sends them to the local machine using a standardized data interaction protocol. The local execution control unit embeds a lightweight, architecture-designed visual language action model. After receiving sub-task language descriptions and real-time visual observations, this model iteratively generates a continuous sequence of control actions covering a preset future time window using a flow-matched, pure cross-attention diffusion transformer. This ultimately drives the robot to perform physical operations and transmits the execution status and visual feedback back to the cloud in real time to form a closed-loop fault-tolerant control circuit.

[0029] like Figure 1 As shown, the robot control method based on visual language action agent provided by the present invention includes the following steps.

[0030] Step S1: Obtain the visual observation sequence and natural language task instructions of the robot's working environment, and upload the visual observation sequence and natural language task instructions to the cloud; the native multimodal large language model deployed in the cloud performs task reasoning and decomposition based on the visual observation sequence and natural language task instructions, generates a sequence of structured sub-tasks containing multiple temporal dependencies, and sends the sub-task sequence to the robot through a standardized data interface. Step S2: In the current execution cycle, the robot receives a structured subtask to be executed from the cloud; it inputs the real-time visual observations of the current environment and the language description of the structured subtask into a lightweight visual language action model deployed locally on the robot; using the first K layers of the native multimodal visual language backbone network in the model, it performs cross-modal fusion encoding on the visual observations and the language description of the subtask, and extracts the fused multimodal representation vector output by the Kth layer; where K is a truncated layer index that is less than the total depth of the backbone network and is determined based on the cross-modal semantic alignment strength.

[0031] In a preferred embodiment of the invention, before inputting the visual observation sequence into the backbone network, a pixel-level inverse rearrangement downsampling operation is performed on each image frame in the sequence to compress the spatial dimension of the visual token sequence to a preset compression ratio before input. This compression ratio is determined by the square of the scaling factor of the inverse rearrangement operation. This spatial compression operation significantly reduces the computational complexity of self-attention when subsequent Transformer layers process long sequences without losing information.

[0032] Step S3: Concatenate and stitch the fused multimodal representation vector with the ontology perception vector representing the robot's current kinematic and dynamic state along the feature dimension to construct a joint conditional feature. The ontology perception vector may include, but is not limited to, the current angles of each joint of the robot, joint angular velocities, the six-dimensional pose of the end effector, and the odometry readings of the mobile chassis.

[0033] Step S4: Using joint conditional features as generation conditions, a continuous control action sequence covering a preset future time window is iteratively generated through a first-class matching-driven cross-modulation diffusion transformer. This diffusion transformer employs a unique pure cross-attention layer stack structure. In each step of the denoising evolution, the noise-perturbed action sequence is embedded as a query vector, and the linear projection of the joint conditional features is used as the key vector and value vector. This continuously injects multimodal contextual conditions into the refinement process of the action sequence, and the entire transformer stack does not contain any self-attention sublayers.

[0034] Step S5: Output the continuous control motion sequence to the robot actuators to drive the robot to complete the physical operations corresponding to the natural language task instructions. This motion sequence may include synchronized control instructions for the six-dimensional pose variables of the robot's dual-arm end effectors in Cartesian space and the velocity variables of the mobile chassis in the plane.

[0035] Step S6: During the execution of physical operations, the multimodal representation vector is fused with real-time visual feedback and task execution status updates. The unfinished natural language task instructions are dynamically decomposed or replanned through the cloud-based thinking chain reasoning mechanism to cope with abnormal situations such as external interference or execution failure.

[0036] In the above method, the determination of the truncated layer index K in step S2 is not arbitrary, but based on a quantitative assessment of the cross-modal alignment strength of each intermediate layer of the backbone network. In a specific implementation of the present invention, the system can pre-calculate the mutual information value or cross-modal attention alignment metric between the visual tokens and language tokens output by each intermediate layer of the backbone network, and select the layer index where the alignment metric reaches a local maximum and has not yet entered the deep layer of pure language logic reasoning as K. Experiments show that for a typical native multimodal Transformer backbone network, the features near layer 14 reach the peak in visual-language cross-modal semantic coupling strength, and from layer 15 onwards, the attention distribution begins to shift towards the self-aggregation pattern within the language tokens. Therefore, in the preferred embodiment of the present invention, K is set to 14, so that the truncated network retains the strongest cross-modal alignment capability while avoiding the computational redundancy caused by the deep network's bias towards pure language reasoning.

[0037] The cross-modulation diffusion converter used in step S4 optimizes its internal velocity field prediction network during the training phase by minimizing the first-flow matching loss function. This flow matching loss function is defined as the expected mean square error between the predicted velocity field and the target flow field on the noise disturbance action state distribution, and its mathematical expression is:

[0038] in, This represents the sequence of real actions sampled from the robot's teaching trajectory, which covers the future from the current moment. Expert teaching action values ​​for each control cycle; This represents the fused multimodal representation vector extracted in step S2, which encodes the joint semantics of the current environmental visual observation and the subtask language description; This represents the robot's body perception vector, providing a kinematic and dynamic state reference at the current moment; Indicates the interpolated weights Real action sequence The intermediate noise action state generated after linear mixing with Gaussian noise is used to construct a continuous diffusion path from pure noise to real data. Indicates by parameters A parameterized learnable velocity field, given the current noisy action state. and conditions Under the premise of this, predict the direction and rate at which the state should evolve into the actual action; Indicates the intermediate action state Guided to real action sequences The target flow direction vector is usually defined as... Or its normalized form; The symbol represents the mathematical expectation, and the subscript indicates that the expectation is a conditional distribution of the actual actions. Conditional distribution of noise action Please obtain.

[0039] The above intermediate action state It is generated based on the following linear interpolation formula:

[0040] in, This represents an independent and identically distributed noise vector sampled from a standard multivariate Gaussian distribution, whose dimension is the same as that of the actual action sequence. Consistent; Interpolation weights are used to sample from a specified probability distribution (such as a Beta distribution) and truncate it to that interval, controlling the degree of noise perturbation. During training, different values ​​are randomly sampled for each training sample. The value enables the model to learn the velocity field along the entire continuous transition path from a completely noisy state to a noiseless, realistic action state.

[0041] This invention further specifies the stacking method of the pure cross-attention mechanism: In each transformation layer of the diffusion transformer, the noise-perturbed action sequence embedding vector is first used as the query vector, and the joint conditional features are mapped to key vectors and value vectors through two independent linear projection matrices respectively; then, the scaled dot product attention weight of the query vector and the key vector is calculated, and the value vector is weighted and summed using this weight to obtain the cross-attention output; this output is fed into the feedforward neural network after residual connection and layer normalization, and after residual connection and layer normalization again, an updated action sequence representation is formed and passed to the next transformation layer. The entire stacking process does not contain any self-attention sublayers, which allows the model to maintain generation quality while reducing the floating-point computation of each layer by about 30%, and avoids excessive constraints of self-attention on the temporal relationships within the action sequence, providing greater freedom for the flow matching mechanism to shape a globally smooth trajectory.

[0042] To enable the system to converge quickly when introducing new tasks while maintaining the rich multimodal semantic knowledge accumulated during the pre-training phase, this invention designs a two-stage progressive training paradigm for visual language action models.

[0043] In the first stage, the action expert alignment stage, the training system completely freezes all learnable parameters of the native multimodal visual language backbone network, updating only the parameters of the ensemble module used for feature concatenation and the action experts in the diffusion transformer. The training objective is to minimize the aforementioned flow matching loss function. Because the backbone network parameters are frozen, the randomly initialized action expert weights gradually learn to establish a mapping relationship with the fixed multimodal embedding space during gradient descent. Gradient noise will not backpropagate to the backbone network, thus strictly protecting the pre-trained semantic space from being destroyed. Stable perception-action alignment can be established with only a very small number of task teaching trajectories.

[0044] In the second stage, the full-parameter adaptation stage, the training system unfreezes all learnable parameters of the backbone network and performs joint fine-tuning on the parameters of the backbone network, ensemble modules, and diffusion transformers. The training objective remains the same: minimizing the flow matching loss function. Since the action experts have already established a stable initial alignment with the multimodal semantic space in the first stage, the fine-tuning process of the backbone network is extremely smooth. Only minor parameter updates are made to adapt to the specific visual distribution and language expression habits of the target task. This allows the system to improve the success rate of downstream tasks while fully maintaining its original cross-modal semantic understanding and generalization capabilities.

[0045] In step S6, the implementation of the thought chain reasoning mechanism relies on the collaborative operation of the cloud-based multimodal large language model and local real-time feedback. Specifically, the cloud model constructs a contextual prompt sequence based on the real-time visual feedback and the current fused multimodal representation vector, which includes a description of the task objective, spatial relationships of environmental objects, the history of executed subtasks, and reasons for failure. This contextual prompt sequence is input into the large language model, and the model is guided to output structured thought chain text through carefully designed prompt word engineering. This thought chain text sequentially includes the following fields that can be parsed by the program: subtask semantic description field, spatial constraint condition field, action primitive type field to be invoked field, and failure recovery strategy identifier field. When the execution driver detects an abnormality in the current subtask execution (such as capture failure or path obstruction), the cloud model automatically triggers the task replanning process based on the failure recovery strategy identifier, correcting and re-issuing only the incomplete part of the subtask sequence without restarting the entire task process, greatly improving the robustness and execution efficiency of the system in unstructured environments.

[0046] Furthermore, in step S5, the continuous control action sequence is generated stepwise by a cross-modulation diffusion converter from a pure noise initial state according to a flow matching mechanism, through a fixed-step iterative denoising process. This continuous control action sequence includes synchronized control commands for the six-dimensional pose variables (three-dimensional position and three-dimensional attitude) of the robot's dual-arm end effectors in Cartesian space and the velocity variables (linear velocity and angular velocity) of the mobile chassis in the plane. In each denoising iteration, the diffusion converter only uses joint conditional features as the cross-attention modulation signal, without introducing additional self-attention calculations, thus achieving extremely low inference latency while ensuring the consistency of the generated trajectory's timing. The length of the aforementioned future preset time window... It can be dynamically adjusted according to the forward-looking needs of the task, and is set to 8 to 16 control cycles in typical implementations.

[0047] In addition to the methods and steps described above, this invention also provides a robot control system based on a visual language action intelligent agent. The system consists of two main parts: a cloud-based inference and planning unit and a local execution control unit. These two parts are connected via a wireless network and continuously exchange data to form a closed loop.

[0048] The cloud-based inference and planning unit is configured on a GPU-accelerated cloud server to receive environmental image sequences uploaded by the robot and natural language task instructions input by the user. This unit runs a large-scale multimodal language model deployed in the cloud and dynamically decomposes high-level user instructions into a set of structured subtasks with clear temporal dependencies through a built-in thought chain inference mechanism. Each subtask includes semantic descriptions, target object identifiers, and spatial constraints, which are encoded in JSON format and sent to the local execution control unit via a standardized data interface (such as a RESTful API).

[0049] The local execution control unit is deployed on the robot body or its associated embedded edge computing device (such as the NVIDIA Jetson series platform). Its internal functional modules include, in sequence, a visual compression encoder, a lightweight multimodal fusion encoder, a state integrator, a motion generator, and an execution driver.

[0050] A visual compression encoder is used to perform pixel-level inverse reordering downsampling on the input image, thereby significantly reducing the visual token space dimension input to subsequent networks. For an image with a resolution of... The input image, after being segmented and embedded, first forms a spatial dimension of A visual token grid; then scaled by a factor of 100. The reverse rearrangement operation compresses the spatial dimension to its original size. The number of channels is expanded to the original number. This reduces the length of the Transformer sequence by a factor of two, without losing spatial information.

[0051] The lightweight multimodal fusion encoder employs the first K layers of a native multimodal visual-language backbone network. It performs cross-modal fusion encoding on the compressed visual token sequence and the subtask language descriptions delivered from the cloud, outputting a fused multimodal representation vector at the Kth layer. This backbone network is a native multimodal Transformer model jointly pre-trained on massive image-text pairing data. The number of parameters in its visual encoding branch can be compressed to the order of 300 million using knowledge distillation techniques, ensuring the overall model's lightweight nature.

[0052] The state integrator simply concatenates the aforementioned fused multimodal representation vectors with the real-time acquired robot ontology perception vectors along the feature dimension to construct joint conditional features. This concatenation operation avoids introducing additional learnable parameters or complex cross-modal projection matrices, achieving seamless fusion of high-level semantics and low-level physical states with extremely low computational cost.

[0053] The action generator uses joint conditional features as conditional variables and iteratively generates a continuous sequence of control actions covering a preset future time window $H$ through a flow-matched driven pure cross-attention diffusion transformer. During the inference phase, the diffusion transformer performs a fixed number of denoising iterations (e.g., ten steps). At each step, the joint conditional features are used as the key value source for cross-attention to refine the current noisy action sequence, ultimately outputting a temporally smooth control action sequence.

[0054] The actuator is responsible for converting the continuous control motion sequence output by the motion generator into joint-level or end-effector pose-level commands executable by the robot hardware. For dual-arm end-effector pose commands, the actuator calculates the target angles of each joint using a built-in inverse kinematics solver; for chassis speed commands, it converts them into left and right wheel speed commands using a low-level proportional-integral-derivative controller. Simultaneously, within each control cycle, the actuator packages and transmits the current execution state (including actual joint angles, end-effector pose feedback, and chassis odometer increments) along with the latest acquired visual observations back to the cloud-based inference and planning unit, completing closed-loop feedback.

[0055] Example 2 This embodiment is a further refinement of Embodiment 1. This embodiment describes the overall architecture and core workflow of the system of the present invention.

[0056] The system is physically deployed in two parts: a cloud-based inference and planning unit and a local execution and control unit. The cloud-based inference and planning unit is hosted on a cloud server instance equipped with a high-performance GPU and maintains a connection with the robot in the home environment via the internet. The robot is equipped with a head-mounted RGB camera to collect environmental visual information and a microphone array to receive user voice commands (voice is converted into text by a local automatic speech recognition module). Simultaneously, the robot body reads data from the encoders and inertial measurement units of each joint in real time via the controller local area network bus as its own perception information.

[0057] When the user issues the natural language command to place the blue water glass on the dining table next to the kitchen sink, the cloud-based reasoning and planning unit first receives an image of the current dining area uploaded by the robot. A multimodal large language model deployed in the cloud performs scene analysis on the image, identifying object instances such as the blue water glass, dining table, and plate, along with their spatial coordinates. Then, it initiates a thought chain reasoning process based on the task command text. Guided by prompts, the model outputs a structured thought chain: First, navigate the moving chassis to a preset operation point near the dining table; second, coordinate the movement of both arms to grasp the blue water glass; third, navigate to a preset placement point next to the kitchen sink; fourth, release the water glass and return it to its original position. Each subtask is encapsulated as a JSON description, including fields such as action type, target object identifier, and target pose constraints, and is sequentially sent to the robot's local execution control unit via 4G / 5G or Wi-Fi networks.

[0058] The local execution control unit is deployed on an NVIDIA Jetson AGX Orin edge computing module within the robot's torso. The visual compression encoder first divides the 640×480 resolution RGB image captured by the head camera into 14×14 pixel image blocks, generating a visual token grid with a spatial dimension of 46×34. Then, through a pixel-level inverse rearrangement operation with a scaling factor of 2, the token spatial dimension is compressed to 23×17, while the number of channels is quadrupled. The lightweight multimodal fusion encoder adopts the first 14 layers of a native multimodal visual-language backbone network (the backbone network has a total depth of 24 layers). It receives the compressed visual token sequence and the language description text of the current subtask, and outputs the fused multimodal representation vector generated by the 14th layer after forward propagation computation.

[0059] Meanwhile, the state integrator reads the current angle values ​​of the seven joints of the two arms, the readings of the six-dimensional force sensor at the end effector, and the cumulative displacement and heading angle from the mobile chassis odometer from the robot's underlying drive, constructing a body perception vector with tens of dimensions. The state integrator directly concatenates the multimodal representation vector and the body perception vector along the feature dimension to form a joint conditional feature.

[0060] The action generator internally consists of eight layers of pure cross-attention layers. At the start of inference, the action generator samples a 16×14 tensor from a standard Gaussian distribution as the initial noisy action sequence, where 16 represents the next 16 control cycles (each cycle is 50 milliseconds), and 14 represents the control dimensions (6-dimensional end-effector pose increments for both arms + 2-dimensional speed commands from the chassis). Joint conditional features are linearly projected to generate a key matrix and a value matrix, while the noisy action sequence is embedded to generate a query matrix. A diffusion transformer performs 10 denoising iterations. In each iteration, the query matrix and key matrix are used to calculate attention weights and are weighted and fused with the value matrix. After updating via residual connections and a feedforward network, the result is input to the next denoising step. After 10 iterations, a smooth control action sequence covering the next 0.8 seconds is obtained.

[0061] The execution driver inputs the end-effector pose commands for each time step in the motion sequence into the inverse kinematics solver to calculate the corresponding joint angle sequence, which is then sent to the joint motor driver via position servo control mode. The chassis speed command is input into the PID controller and converted into speed commands for the differential drive wheels. Simultaneously with command execution, the execution driver packages the current head camera image, actual joint angles, and task progress status at a frequency of 16.4Hz and transmits them back to the cloud inference and planning unit via the User Datagram Protocol (UDP).

[0062] The cloud-based inference planning unit continuously monitors and transmits data. If the fingertip pressure sensor detects an empty grasp during the execution of a grasping subtask, and the system determines that the grasp has failed, the cloud-based thinking chain module automatically generates a recovery strategy based on the current image and the failure status. This strategy could include adjusting the grasping posture and retrying, or re-identifying the position of the cup. After updating the subtask sequence, the strategy is immediately deployed to achieve seamless fault recovery.

[0063] Example 3 Based on Embodiment 2, this embodiment further discloses the quantization selection process of the truncated layer index K in the lightweight multimodal fusion encoder and the specific parameter configuration of the model.

[0064] This embodiment employs a native multimodal visual-language backbone network, a joint encoder based on the Transformer architecture, comprising 24 encoding layers. Each encoding layer consists of alternating self-attention sublayers and cross-attention sublayers (used when fusing visual and linguistic features). To determine the optimal truncation layer index K, this embodiment constructs a validation set containing one hundred images of a home scene and their corresponding descriptive text. For each image and its text description, it is input into the complete 24-layer backbone network, and the average cross-attention weight between the visual token and the corresponding linguistic token in the output of each encoding layer is recorded.

[0065] Analysis revealed that the visual-linguistic cross-attention weights output from layer 14 reached a local maximum, and the attention map exhibited a clear alignment pattern with task-related objects in its spatial distribution. Starting from layer 15, the distribution of attention weights gradually shifted towards a self-aggregating pattern within the linguistic tokens, and the attention intensity of visual tokens to linguistic tokens began to decrease, indicating that the network was entering a deeper processing stage focused on abstract linguistic logical reasoning. Based on these observations, selecting K=14 as the truncation layer index effectively reduced the significant computational overhead of layers 15 to 24 while preserving the strongest cross-modal semantic coupling representation capability.

[0066] To further verify the universality of the cutoff layer selection, this embodiment compares the task success rates corresponding to cutoff layers from 10 to 20 on three typical household task tasks (tabletop object grasping, refrigerator door opening, and drawer pushing / pulling). The experimental results are presented as mean ± standard deviation of success rate: layer 10: 62.3% ± 5.1%; layer 12: 71.8% ± 4.2%; layer 14: 85.6% ± 3.3%; layer 16: 82.1% ± 3.8%; layer 18: 76.4% ± 4.5%; layer 20: 68.9% ± 5.6%. These results quantitatively confirm the optimality of layer 14 as the cutoff layer for cross-modal task tasks.

[0067] In terms of parameter scale, the visual coding branch after knowledge distillation has 307 million parameters, the truncated language fusion branch (first 14 layers) has 422 million parameters, and adding the parameters of the integration module and diffusion transformer, the total number of model parameters is approximately 770 million. On an NVIDIA RTX 3070 consumer-grade GPU, the model inference frequency is stable at 16.4 Hz, and the peak memory usage is 2.3 GB.

[0068] Example 4 Based on Example 2, this embodiment details the specific implementation details of the two-stage progressive training paradigm and the ablation experiment results.

[0069] During the training data acquisition phase, for each task to be learned, a human operator controlled the robot to complete fifty to one hundred successful demonstrations via a teleoperation device. Simultaneously, the RGB image sequence (sampled at 30Hz and then downsampled to 5Hz), task description text, robot body perception state sequence (sampled at 125Hz), and action command sequence for each demonstration were recorded. All data were time-aligned to form a supervised learning sample set.

[0070] In the first phase, the action expert alignment phase, the training script completely freezes all parameters of the lightweight multimodal fusion encoder (the first 14 layers of the backbone network). The trainable parts include: the linear projection layers in the state integrator (if any), and all parameters of the diffusion transformer (including the query, key, and value projection matrices and feedforward network weights in the 8-layer cross-attention module). The optimizer used is AdamW, with a learning rate set to 1×10⁻⁶. -4 The weight decay coefficient is 1×10 -2 Each training batch contains 32 randomly sampled trajectory segments. Training continues until the flow matching loss function does not decrease for five consecutive epochs on the validation set, typically requiring about fifty training epochs. At the end of this phase, the diffusion transformer has learned to map conditional signals in the multimodal semantic space to physically plausible action trajectories.

[0071] In the first stage, the optimization objective can be formally expressed as:

[0072] in, This represents the parameters of the motion expert module, i.e., the diffusion converter; Indicates the parameters of the integrated module; This indicates that the parameters of the visual language backbone network are frozen and do not participate in gradient updates at this stage. For the aforementioned flow matching loss function, This represents the optimal parameters for the motion expert module. This represents the optimal parameters of the ensemble module. The training objective of this stage is to establish an alignment mapping between the randomly initialized action expert weights and the fixed multimodal embedding space, ensuring that gradient noise does not backpropagate to the backbone network, thereby strictly protecting the pre-trained semantic space.

[0073] In the second phase, the full parameter adaptation phase, the training script unfreezes all parameters of the lightweight multimodal fusion encoder. The optimizer uses a relatively small learning rate of 1×10⁻⁶ for the backbone network. -5 Maintain a 1×10 for the integrated module and diffusion converter section. -4 The learning rate was set. Other hyperparameters remained consistent with those in the first stage. This stage typically converges in just twenty training epochs. By monitoring the update norm of the backbone network parameters during training, it was found that the parameter changes were effectively limited to a small neighborhood of the initial values, indicating that only slight adaptive adjustments occurred in the semantic space without structural forgetting.

[0074] The second-stage optimization objective is expanded to:

[0075] in, This represents the complete set of learnable parameters for the entire model. This represents the initial values ​​of the parameters after the first phase of training is completed. This represents the final optimal set of parameters for the model. This represents the optimization operation, which involves finding the parameters that minimize the loss function. In this stage, the backbone network is unfrozen, and global joint fine-tuning is performed. The training objective remains minimizing the flow matching loss function.

[0076] To verify the effectiveness of the two-stage training paradigm, this embodiment designed an ablation comparison experiment, comparing the two-stage model of this invention with a baseline model that directly performs end-to-end full-parameter joint training (single-stage). In unseen test scenarios (including changes in ambient lighting, background object interference, and random shifts in the target object's position), the two-stage model achieved a task success rate of 87.2%, while the single-stage model's task success rate was only 63.5%. Further examination of the attention heatmap of the model's intermediate layers using visualization tools revealed that the two-stage model's attention consistently and stably focused on the task-related object region, while the single-stage model's attention exhibited significant diffusion and drift in the interference environment. This comparison strongly demonstrates the significant advantages of the two-stage training paradigm in terms of semantic preservation and generalization robustness.

[0077] Example 5 This embodiment further describes the mathematical expression of the end-to-end control mapping relationship of the system of the present invention.

[0078] The end-to-end mapping relationship from multimodal perceptual input to low-level control action output in the visual language action model of this invention can be summarized by the following formula:

[0079] in, Indicates at time The vector of consecutive actions executed. Indicates from RGB image observation from multiple perspectives Indicates instructions for natural language tasks. Indicates the robot's perception state. This represents the set of learnable parameters for the entire model. This formula summarizes the lightweight mapping mechanism from multimodal sensor inputs to low-level physical control actions.

[0080] Specifically, the lightweight technical features of this invention in architectural design are reflected in the following three aspects: First, a native multimodal joint learning backbone network is employed to eliminate parameter redundancy in post-hoc alignment. Unlike traditional VLA models that use "post-hoc alignment" (i.e., forcibly transforming a large plain text model into a multimodal model), this invention uses a native multimodal visual language model (VLM) as the backbone network. This native design achieves joint learning of vision and language during the pre-training stage, enabling natural alignment across modal semantic spaces. This tightly coupled architecture avoids the large cross-modal projection matrices or complex multilayer perceptron (MLP) networks introduced in traditional methods to bridge the modal gap, thus achieving higher-density multimodal understanding with a smaller parameter base.

[0081] Second, a spatial compression mechanism for visual tokens based on pixel-unshuffle is introduced. At the visual perception end, addressing the quadratic computational complexity of sequence length faced by traditional Transformer architectures when processing high-dimensional images, this invention specifically modifies the visual encoding branch. The visual encoder first extracts key visual representations from a large-scale teacher model using knowledge distillation (compressing parameters to approximately 300 million); more importantly, after extracting image patch features, this invention innovatively introduces a pixel-unshuffle downsampling operation. This operation densely reassembles the visual token sequence in the spatial dimension, significantly reducing the total number of visual tokens input to the subsequent fusion network to one-quarter of the original. This structural optimization, while preserving sufficient fine-grained spatial information, greatly reduces the memory usage and computational latency of the self-attention mechanism when processing long sequences.

[0082] Third, a deep-level compression and feature extraction strategy based on 14th-layer feature truncation is implemented. At the language decoding and cross-modal fusion stage, this invention employs a patch-level image embedding strategy to embed the compressed visual feature sequence into the natural language instruction sequence. The formula for generating the fused multimodal representation is as follows:

[0083] in, This represents a fused multimodal representation vector that jointly encodes visual and linguistic information. In computation... In this invention, the redundant and deep complete Transformer decoder structure is abandoned, and only the features of the first 14 layers of the backbone network are retained and extracted as... The underlying technical principle lies in the fact that the features of deep networks are often highly abstract and biased towards pure linguistic logical reasoning, while the features of intermediate layers (such as layer 14) exhibit the strongest coupling in visual-linguistic cross-modal alignment, and their semantic representation level is highly compatible with the physical motion control requirements of the underlying embodied intelligence. By truncating redundant network branches after layer 14, this invention not only further reduces the significant computational overhead of model forward propagation, but also directly extracts the intermediate features most suitable for downstream ActionExpert calls, achieving extreme lightweighting of the VLA model from both the physical network depth and the physical meaning of features.

[0084] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0085] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot control method based on a visual language action agent, characterized in that: Includes the following steps: Step S1: Obtain the visual observation sequence and natural language task instructions of the robot's working environment, and upload the visual observation sequence and natural language task instructions to the cloud; the native multimodal large language model deployed in the cloud performs task reasoning and decomposition based on the visual observation sequence and natural language task instructions, generates a sequence of structured sub-tasks containing multiple temporal dependencies, and sends the sub-task sequence to the robot through a standardized data interface. Step S2: In the current execution cycle, the robot receives a structured subtask to be executed from the cloud; it inputs the real-time visual observations of the current environment and the language description of the structured subtask into a lightweight visual-language action model deployed locally on the robot; using the first K layers of the native multimodal visual-language backbone network in this model, it performs cross-modal fusion encoding on the visual observations and the language description of the subtask, and extracts the fused multimodal representation vector output by the Kth layer; where K is a truncated layer index that is less than the total depth of the backbone network and is determined based on the cross-modal semantic alignment strength; the truncated layer index K is determined as follows: calculate the mutual information or cross-modal attention alignment metric between the visual features and language features output by each intermediate layer of the backbone network, and select the layer index that has reached a local maximum value and has not yet entered the deep layer of pure language logic reasoning as K; Step S3: Concatenate the fused multimodal representation vector with the ontology perception vector representing the robot's current kinematic and dynamic state along the feature dimension to construct joint conditional features; Step S4: Using joint conditional features as conditions, a continuous control action sequence covering a preset future time window is iteratively generated through a flow-matched driven cross-modulation diffusion transformer. The diffusion transformer continuously injects conditional information into the denoising evolution process of the action sequence using a pure cross-attention mechanism. The stacking method of the pure cross-attention mechanism is as follows: In each transformation layer of the diffusion transformer, the noise-perturbed action sequence embedding is used as the query vector, and the linear projection of the joint conditional features is used as the key vector and value vector. The cross-attention output is calculated and the action sequence representation is updated through residual connections and feedforward networks. There are no self-attention sub-layers between layers. Step S5: Output the continuous control action sequence to the robot actuator to drive the robot to complete the physical operation corresponding to the natural language task instruction; Step S6: During the execution of physical operations, the fused multimodal representation vector is updated using real-time visual feedback and task execution status. The incomplete natural language task instructions are dynamically decomposed or replanned using a thought chain reasoning mechanism. This includes: constructing a contextual prompt sequence containing task objectives, environmental states, and execution history based on the returned visual feedback and the current fused multimodal representation vector; inputting the contextual prompt sequence into the cloud-based multimodal large language model to guide it to output thought chain text conforming to a preset reasoning pattern. This thought chain text sequentially contains the following parsable fields: subtask semantic description, spatial constraints, action primitive type, and failure recovery strategy identifier; based on the failure recovery strategy identifier, the cloud-based multimodal large language model automatically triggers task replanning and updates the natural language task instructions sent to Step S2 when an execution anomaly is detected.

2. The robot control method based on visual language action intelligent agent according to claim 1, characterized in that: In step S2, before inputting the current environmental visual observation into the robot's local lightweight visual language action model, the following steps are also included: performing a pixel-level inverse rearrangement downsampling operation on each image frame of the current visual observation to compress the spatial dimension of the visual token sequence to a preset compression ratio before input. The compression ratio is determined by the square of the scaling factor of the inverse rearrangement operation.

3. The robot control method based on visual language action intelligent agent according to claim 1, characterized in that: In step S4, the cross-modulated diffusion converter optimizes its internal velocity field prediction network during the training phase by minimizing the flow matching loss function. The flow matching loss function is defined as the expected mean square error between the predicted velocity field and the target flow field on the noise disturbance action state distribution, and its mathematical expression is: in, This represents a sequence of real actions sampled from the robot's teaching trajectory. This represents the fused multimodal representation vector. Represents the ontology perception vector. Indicates the interpolated weights Intermediate action states are generated by mixing real action sequences with Gaussian noise. Indicates by parameters Parameterized velocity field, This represents the target flow direction vector that guides the intermediate action state to the actual action sequence.

4. The robot control method based on visual language action intelligent agent according to claim 1, characterized in that: intermediate action state Generated based on the following linear interpolation: in, This represents an independent and identically distributed noise vector sampled from a standard multivariate Gaussian distribution. Interpolation weights, sampled and truncated from a specified probability distribution, are used to control the degree of noise perturbation in the action sequence.

5. The robot control method based on visual language action intelligent agent according to claim 1, characterized in that: The native multimodal visual language backbone network, the integration module for cascading and fusing multimodal representation vectors and ontology perception vectors, and the diffusion transformer employ a two-stage progressive training paradigm for parameter optimization: In the first stage, all learnable parameters of the backbone network are frozen, and only the parameters of the ensemble module and the diffusion transformer are updated. The training objective is to minimize the flow matching loss function in order to establish the alignment between the action generation space and the multimodal semantic space. In the second stage, all learnable parameters of the backbone network are unfrozen, and joint fine-tuning is performed on the parameters of the backbone network, integration modules, and diffusion transformers. The training objective is to minimize the flow matching loss function in order to adapt to downstream operation tasks while preserving semantic priors.

6. The robot control method based on visual language action intelligent agent according to claim 1, characterized in that: In step S5, the continuous control action sequence is generated by the cross-modulation diffusion converter from the initial state of pure noise through iterative denoising based on the flow matching mechanism. The continuous control action sequence includes synchronous control commands for the six-dimensional pose variables of the robot's dual-arm end effectors in Cartesian space and the velocity variables of the mobile chassis in the plane. In each denoising iteration, the diffusion converter only uses the joint conditional features as the cross-attention modulation signal to maintain the temporal consistency of the generated trajectory.

7. A robot control system based on a visual language action agent, used in the robot control method of claim 1, characterized in that: include: The cloud-based inference planning unit is configured to: receive environmental image sequences collected by a visual sensor and natural language task instructions input by the user; run a multimodal large language model deployed in the cloud and call the thought chain inference mechanism to dynamically decompose the natural language task instructions into a set of structured subtasks with temporal dependencies; and issue descriptive information of the set of structured subtasks through a standardized data interface. A local execution control unit, deployed on the robot body or its associated edge computing device, includes: The visual compression encoder performs pixel-level inverse rearrangement downsampling on the input image to reduce the visual token space dimension input to subsequent networks; The lightweight multimodal fusion encoder uses the first K layers of the native multimodal visual language backbone network to perform cross-modal fusion encoding of visual features and subtask language descriptions, and outputs the fused multimodal representation vector of the Kth layer, where K is a truncated layer index that is less than the total depth of the backbone network and is determined based on the cross-modal semantic alignment metric. The state integrator cascades the multimodal representation vectors and the robot's ontological perception vectors along the feature dimension to construct joint conditional features; The action generator, based on joint conditional features, iteratively generates a continuous sequence of control actions covering a preset future time window through a pure cross-attention diffusion transformer driven by flow matching. The actuator converts the continuous control action sequence into robot joint-level or end-effector pose-level control commands, and transmits the execution status and visual observations back to the cloud-based inference and planning unit in real time to form a closed-loop feedback.

Images