Dynamic look-ahead fusion robot control method based on VLA model

Abstract

The invention belongs to the technical field of robot control, and discloses a VLA model-based dynamic look-ahead fusion robot control method, which comprises the following steps of: acquiring initial observation information and uploading the initial observation information to a server, receiving an initial action sequence which is generated by calling a VLA model and according to the initial observation information by the server, and adding the initial action sequence to the tail of an execution queue; monitoring whether a prospective planning request condition is met or not, calling an instruction dequeuing operation of an execution queue, and obtaining and executing a control instruction to drive a robot; if the look-ahead planning request condition is met, asynchronously initiating a look-ahead request to a server based on the latest observation information, and receiving a look-ahead action sequence returned by the server; and based on the execution queue and the look-ahead action sequence, evaluating whether the look-ahead action sequence is accepted by adopting a cost function, and if so, fusing the execution queue and the look-ahead action sequence to update the execution queue. The method improves the task success rate and the system reliability of the existing method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot control, and particularly relates to a dynamic forward-looking fusion robot control method based on a VLA model. BACKGROUND

[0002] Under the impetus of the current artificial intelligence wave, robot technology is gradually breaking through the limitations of traditional industrial scenes and moving towards complex and variable open environments. In recent years, as an emerging end-to-end solution, the vision-language-action model (VLA) has shown great potential in directly mapping visual perception and language understanding into robot control instructions, providing new possibilities for building general robot systems.

[0003] However, in order to balance the computing power demand and hardware cost, the industry generally adopts a server-client architecture as the standard deployment paradigm when deploying VLA models in robot systems. Under this architecture, the computationally intensive VLA model is deployed on a cloud server, while the robot body is controlled by a resource-limited industrial computer, and the two communicate instructions and data through network protocols such as TCP / IP. This architecture balances the computing power demand and hardware cost, but also forms its unique "request-execute-stop" system architecture, i.e. after the industrial computer executes a sequence of actions, it must stop moving and wait for the latest observation data to be sent to the server, the server to complete the model inference and return a new sequence of actions before it can continue execution. This intermittent control mode causes a certain pause period in robot movement, severely disrupting the continuity of action execution, making it difficult for the VLA model to handle tasks that require high motion fluency.

[0004] Currently, the main solutions aimed at ensuring continuous action control include local trajectory filling, streaming action generation, and forward-looking fusion mechanisms. However, these solutions all have significant shortcomings. The "local trajectory filling" solution relies too much on an accurate local environment model, with poor environmental adaptability; the "streaming action generation" solution has high requirements for network stability and computing power, with significant practical limitations. In comparison, the "forward-looking fusion mechanism" hides delays by planning and sequence fusion in advance, which is more theoretically promising, but its practical application faces two key defects. First, the forward-looking opportunity is fixed and cannot be dynamically adjusted according to the urgency of the task, such as the distance to the target, leading to inefficient resource allocation and real-time imbalance; second, there is a lack of quality assessment of the server returned sequence, which may blindly fuse inferior action sequences, causing trajectory jitter and even task failure.

[0005] In summary, the prior art solution faces inherent limitations in continuity and adaptability in dynamic environments. Therefore, optimizing the "look-ahead fusion mechanism" to address the fixed look-ahead timing and the lack of quality evaluation has become a better way to improve the performance of continuous robot control. SUMMARY

[0006] To address the two technical problems of resource imbalance and real-time performance caused by fixed look-ahead timing in existing look-ahead fusion mechanisms, and the blind fusion of poor action sequences due to the lack of quality evaluation, the present application provides a dynamic look-ahead fusion robot control method based on a VLA model, specifically a method for implementing continuous action control of a robot under a server-client architecture. This method realizes precise matching of computing resources and task real-time requirements through two core mechanisms: dynamic look-ahead request and sequence-level policy fusion. It ensures action continuity while avoiding the execution of poor action sequences, thereby improving the task success rate and system reliability of existing methods.

[0007] To achieve the above objectives, the technical solution adopted by the present application is as follows:

[0008] A dynamic look-ahead fusion robot control method based on a VLA model, involving an industrial computer and a server, implemented on the industrial computer side, comprising the following steps:

[0009] Collecting initial observation information based on observation nodes and uploading it to the server, receiving the initial action sequence generated by the server calling the VLA model based on the initial observation information, and adding the initial action sequence to the end of the execution queue;

[0010] Monitoring whether the look-ahead planning request condition is met, and calling the instruction out-of-queue operation of the execution queue to obtain and execute the control instructions to drive the robot;

[0011] If the look-ahead planning request condition is met, initiate a look-ahead request asynchronously to the server based on the latest observation information, and receive the look-ahead action sequence returned by the server;

[0012] Based on the execution queue and the look-ahead action sequence, use a cost function to evaluate whether to accept the look-ahead action sequence. If not, discard the look-ahead action sequence and initiate a look-ahead request asynchronously to the server based on the latest observation information. Otherwise, fuse the execution queue and the look-ahead action sequence to update the execution queue;

[0013] Collect the latest observation information at a fixed control frequency, monitor whether the look-ahead planning request condition is met, and obtain and execute the control instructions to drive the robot until the robot control is completed.

[0014] The following also provides several optional modes, but not as an additional limitation of the above general scheme, just a further supplement or preferred, without technical or logical contradiction, each optional mode can be combined alone for the above general scheme, but also can be combined between multiple optional modes.

[0015] As preferred, the execution queue has three operations, respectively, instruction dequeuing operation, sequence re-planning operation and sequence expansion operation;

[0016] The instruction dequeuing operation is used to output the head instruction in the execution queue;

[0017] The sequence re-planning operation is used to replace the remaining sequence in the execution queue which has not been executed with an action sequence of equal length;

[0018] The sequence expansion operation is used to connect an action sequence to the end of the execution queue.

[0019] As preferred, the monitoring whether it meets the forward planning request condition includes:

[0020] Calculating a dynamic forward interval according to the latest observation information;

[0021] Judging whether the current action execution count is equal to or greater than the dynamic forward interval;

[0022] If the current action execution count is equal to or greater than the dynamic forward interval, it means that it meets the forward planning request condition, and the action execution count is reset to zero; otherwise, it means that it does not meet the forward planning request condition, and the action execution count is incremented by one after executing a control instruction.

[0023] As preferred, the dynamic forward interval is calculated before the execution of the control instruction, and the calculating a dynamic forward interval according to the latest observation information includes:

[0024] According to the latest observation information, calculating the distance between the spatial position of the robot end effector and the task target;

[0025] Normalizing the distance to obtain a normalized real-time distance;

[0026] Using an exponential mapping function to map the normalized real-time distance to a forward interval interval to obtain the final dynamic forward interval.

[0027] As preferred, the forward action sequence is generated by the server calling the VLA model and according to the latest observation information.

[0028] As preferred, the cost function is used to evaluate whether to accept the forward action sequence based on the execution queue and the forward action sequence, including:

[0029] taking a queue residual sequence in the execution queue which is not executed;

[0030] extracting a sequence from the lookahead action sequence which is equal in length to the queue residual sequence as an overlap action sequence;

[0031] inputting the queue residual sequence and the overlap action sequence into a cost function respectively to obtain a current cost and a lookahead cost;

[0032] if the difference between the lookahead cost and the current cost is greater than a cost threshold, it indicates that the lookahead action sequence is not accepted; otherwise, it indicates that the lookahead action sequence is accepted.

[0033] Preferably, the cost function calculation process is as follows:

[0034] taking a to-be-calculated action sequence, estimating an expected robot end pose corresponding to each control instruction in the to-be-calculated action sequence based on the control instruction through a forward kinematics model;

[0035] calculating the distance between each expected robot end pose and the spatial position of the task target, and accumulating all distances as a cost.

[0036] Preferably, the fusion of the execution queue and the lookahead action sequence to update the execution queue comprises:

[0037] taking a queue residual sequence in the execution queue which is not executed;

[0038] extracting a sequence from the lookahead action sequence which is equal in length to the queue residual sequence as an overlap action sequence;

[0039] performing weighted fusion on the queue residual sequence and the overlap action sequence to generate a fused sequence;

[0040] calling a sequence replanning operation of the execution queue to replace the queue residual sequence in the execution queue which is not executed with the fused sequence;

[0041] calling a sequence expansion operation of the execution queue to add a future expansion sequence in the lookahead action sequence other than the overlap action sequence to the end of the execution queue.

[0042] Preferably, the weighted fusion of the queue residual sequence and the overlap action sequence to generate the fused sequence comprises:

[0043] calculating a fusion weight according to the current cost and the lookahead cost as follows:

[0044]

[0045] wherein, is the fusion weight, and These are the forward-looking costs and the current costs, respectively. It is a smoothing factor;

[0046] By fusion weights The remaining sequence in the queue and the overlapping action sequence are linearly weighted to generate a fused sequence.

[0047] This invention provides a dynamic look-ahead fusion robot control method based on a VLA model, which hides motion planning delays within the execution process and dynamically adjusts the planning frequency according to the distance to the target, balancing efficiency and accuracy. Subsequently, through intelligent evaluation and fusion of new and old instructions, the smoothness of instruction switching is improved, and abrupt action changes are avoided, thereby enhancing the operational accuracy and stability of the server-client architecture system in real-world environments. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a robot control system based on a server-client architecture in an embodiment of the present invention;

[0049] Figure 2 A flowchart illustrating the dynamic look-ahead fusion robot control method based on the VLA model provided in an embodiment of the present invention;

[0050] Figure 3 This is a timing diagram illustrating the dynamic look-ahead request and action execution in the control method of the present invention;

[0051] Figure 4 This is a schematic diagram illustrating the adaptive variation of the dynamic look-ahead interval with end-effector distance in this invention.

[0052] Figure 5 This is a schematic diagram of sequence weighted fusion in this invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0055] Example 1:

[0056] In view of the defects of the prior art, the forward fusion mechanism is optimized in the embodiment to solve the problems of fixed forward opportunity and missing quality evaluation in the prior art. The embodiment provides a dynamic forward fusion robot control method based on a VLA model, as shown in Figure 1 , which relates to an industrial computer (industrial computer side), a server (cloud server side) and an execution layer. The execution layer is the robot body. The industrial computer and the server interact through a communication layer, such as a TCP / IP protocol.

[0057] As shown in Figure 2 , the dynamic forward fusion robot control method based on the VLA model provided by the embodiment is implemented on the industrial computer side and includes the following steps:

[0058] Step 1, collect initial observation information based on observation nodes and upload it to the server, receive the initial action sequence generated by the server calling the VLA model according to the initial observation information, and add the initial action sequence to the end of the execution queue.

[0059] During initialization and the first round of planning, the industrial computer uses its observation nodes to generate observation information . The observation information is sent to the server through the communication layer based on the TCP protocol. After receiving the observation information , the server calls the deployed VLA model (the structure, training process and hyperparameter setting of the VLA model refer to the literature Black K, Brown N, Driess D, et al. $\pi_0 $: A Vision-Language-Action Flow Model for General Robot Control[J]. arXiv preprint arXiv:2410.24164, 2024.) for planning. The VLA model is a diffusion model with observation information as input. Through a step-by-step denoising iterative process, the initial random action sequence is optimized to the final output initial action sequence . After planning is completed, the server sends the initial action sequence back to the industrial computer through the communication layer. The industrial computer then adds the initial action sequence to the end of the execution queue that manages action sequences, and completes initialization.

[0060] Among them, the observation information, the initial action sequence, the diffusion model and the execution queue satisfy the following definitions:

[0061] The observation information is the observation information at time step Data acquisition is the digital description of the robot's own state and its environment. Its mathematical expression is:

[0062] (1)

[0063] in, For time steps No. Zhang observation images (the robot has 1 to 3 images). The total number of observed images, It is a time step The sequence of language commands is the same when the robot performs the same task. It is a time step Vectors representing the joint positions of the robot.

[0064] Initial action sequence The sequence of actions to be performed by the robot, generated by the VLA model. Its mathematical expression is:

[0065] (2)

[0066] in, For time steps Specific action instructions, The length of the initial action sequence.

[0067] diffusion model A denoising process is defined, which denoises the observed information. Guided by this process, starting with a randomly initialized action sequence, noise is removed through multiple iterations, and the final output is the initial action sequence. Its mathematical expression is:

[0068] (3)

[0069] Execution queue It is the motion scheduling module of the industrial control computer, which is a time-stepping module. The evolving dynamic structure functions to output control commands sequentially. It also dynamically manages the action sequence. Its mathematical expression is:

[0070] (4)

[0071] (5)

[0072] in, It is the current time step Header instructions to be executed This represents the remaining sequence in the queue, i.e., all actions that have not yet been executed. a connection operation of sequences, the length of the queue remaining sequence.

[0073] the execution queue in this embodiment There are three operations defined:

[0074] (1) The instruction dequeue operation is used to output the head instruction in the execution queue to be executed, and update to . The mathematical expression is:

[0075] (6)

[0076] (7)

[0077] wherein, is the output control instruction, represents the assignment operation, is the instruction dequeue operation.

[0078] (2) The sequence re-planning operation is used to replace the queue remaining sequence in the execution queue that has not been executed with an action sequence of the same length. The mathematical expression is:

[0079] (8)

[0080] wherein, is the sequence re-planning operation, is the action sequence used for replacement.

[0081] (3) The sequence expansion operation is used to connect an action sequence of unlimited length to the end of the current queue . The mathematical expression is:

[0082] (9)

[0083] wherein, is the sequence expansion operation, is the action sequence used for expansion.

[0084] Step 2, monitor whether the forward planning request condition is met, and call the instruction dequeue operation of the execution queue to obtain and execute the control instruction to drive the robot. If the forward planning request condition is met, initiate a forward request to the server asynchronously based on the latest observation information, and receive the forward action sequence returned by the server.

[0085] ​In this embodiment, the industrial control computer cyclically calls the execution queue at a fixed control frequency. The instruction dequeue operation retrieves and executes control instructions. To drive the robot. During this continuous execution, the look-ahead controller is synchronously invoked to make decisions regarding the sending of look-ahead requests. For example... Figure 3 As shown, in each loop execution, the latest observation information is first obtained, which records the latest observation image and the latest robot joint position vector; then, it is monitored whether the look-ahead planning request conditions are met, and if the look-ahead planning request conditions are met, the look-ahead planning and current robot motion control are executed asynchronously.

[0086] The look-ahead action sequence is generated by the server calling the VLA model and based on the latest observation information. The generation logic is the same as the initial action sequence, and will not be elaborated here. Its mathematical expression is:

[0087] (10)

[0088] in, The VLA model is based on the time step during look-ahead requests. Control commands generated based on observation information Forward action sequence The length.

[0089] The look-ahead controller is a decision logic unit deployed on an industrial control computer. Its function is to dynamically determine and trigger the sending of look-ahead planning requests to the front-end server based on the robot's real-time status. Specifically, it involves counting the actions executed within the look-ahead controller. Equal to or greater than the dynamic look-ahead interval At that time, the industrial control computer based on the latest observed status An asynchronous lookahead request is sent to the server, and the action execution count is reset to zero. The lookahead request is non-blocking; the industrial control computer immediately resumes execution from the queue after receiving the lookahead request. The remaining control instructions are acquired and executed, thereby hiding communication and computation delays within the action execution time and achieving uninterrupted continuous control.

[0090] The specific process for monitoring whether the conditions for a forward-looking planning request are met is as follows:

[0091] (1) Calculate the dynamic look-ahead interval based on the latest observation information. In this embodiment, the dynamic look-ahead interval is calculated before the control command is executed. Specifically, based on the latest observation information, the distance between the robot end effector and the spatial position of the task target is calculated; the distance is normalized to obtain the normalized real-time distance; and an exponential mapping function is used to map the normalized real-time distance to the look-ahead interval interval to obtain the final dynamic look-ahead interval.

[0092] Real-time distance is the Euclidean distance between the robot end-effector (e.g. the robot gripper end-point) and the task goal center at time step . The calculation process is:

[0093] (11)

[0094] where and are the spatial positions of the robot end-effector and the task goal at the previous time step , respectively. denotes the Euclidean distance.

[0095] The dynamic lookahead interval is a variable that dynamically adjusts according to the real-time distance to determine the number of execution steps that should be spaced between two adjacent lookahead requests. Its design aims to implement a core adaptive strategy: when the robot is far from the goal, automatically reduce the planning frequency to save computational resources; when the robot approaches the goal, automatically increase the planning frequency to ensure operation precision and responsiveness. The calculation process is:

[0096] (12)

[0097] (13)

[0098] where is the normalized real-time distance, and are the preset fine operation distance and free operation distance, respectively, and are the minimum and maximum values of the lookahead interval range, respectively, is the gain coefficient to adjust the response curve.

[0099] Equation (12) calculates the normalized distance, which maps the real-time distance to the [0, 1] interval. Equation (13) uses an exponential mapping function to map the normalized real-time distance to the lookahead interval range, making the planning frequency more sensitive to changes when the real-time distance is close, thereby improving the delicacy and responsiveness of control within the fine operation distance.

[0100] As Figure 4As shown, the correspondence between the distance between the robot gripper and the target and the dynamic lookahead interval is shown: when the gripper is far from the target, more steps of action need to be planned in advance to ensure the continuity of subsequent motion; when the gripper is close to the target, only a small number of steps (the action is more precise and does not need to be planned too much in advance) need to be planned; when the distance exceeds 0.15m, 20 steps of lookahead coverage planning requirement is fixed to avoid excessive calculation overhead. The dynamic adjustment of the lookahead interval in this embodiment allows the robot to efficiently and accurately plan actions in advance at different distances, ensuring smooth motion and controlling the calculation cost.

[0101] (2) Determine whether the current action execution count is equal to or greater than the dynamic lookahead interval; if the current action execution count is equal to or greater than the dynamic lookahead interval, it indicates that the lookahead planning request condition is met, and the action execution count is reset to zero; otherwise, it indicates that the lookahead planning request condition is not met.

[0102] In this embodiment, the action execution count is an integer counter initialized to zero, used to record the number of control instructions that have been taken out of the execution queue and sent to the robot since the last lookahead planning. Its update rule is: when a control instruction is taken out of the execution queue and sent to the robot, the count is incremented . When the lookahead request is triggered , the count is reset: .

[0103] Step 3, based on the execution queue and the lookahead action sequence, use the cost function to evaluate whether to accept the lookahead action sequence, if not, discard the lookahead action sequence; otherwise, merge the execution queue and the lookahead action sequence to update the execution queue.

[0104] Step 3.1, lookahead sequence evaluation and processing. The industrial computer receives the lookahead action sequence returned by the lookahead controller in step 2 , and then calls the sequence decision maker to evaluate . The sequence decision maker makes a decision to accept or reject by evaluating the lookahead action sequence .

[0105] The sequence decision maker is a sequence evaluation and decision unit deployed on the industrial computer, which evaluates the lookahead action sequence and makes a decision to accept or reject based on the preset standard to ensure that the action sequence input into the execution queue remains high quality. Specifically, the steps are as follows:

[0106] Step 3.1.1, take the remaining sequence of the queue that has not been executed in the execution queue.

[0107] Step 3.1.2: Extract the sequence from the look-ahead action sequence that has the same length as the remaining sequence in the queue starting from the time the look-ahead action sequence is returned by the server as the overlapping action sequence.

[0108] Overlapping action sequences It is a look-ahead action sequence The corresponding time interval The subsequence of . Its mathematical expression is:

[0109] (14)

[0110] Among them, time step To receive the lookahead action sequence returned by the server At that moment, For overlapping action sequences Length, and the remaining sequence in the queue Equal in length, and .

[0111] Step 3.1.3: Input the remaining sequence in the queue and the overlapping action sequence into the cost function to obtain the current cost and the look-ahead cost.

[0112] Trajectory cost function It is a tool for quantitatively evaluating action sequences. This is a function of the planned motion trajectory quality. It comprehensively reflects the proximity of the entire trajectory to the target point through a scalar value, providing an objective and comparable quality indicator for the sequence decision-maker. The calculation process involves taking the action sequence to be calculated, estimating the expected robot end-effector pose corresponding to each control command in the sequence using a forward kinematics model, calculating the distance between each expected robot end-effector pose and the spatial position of the task target, and accumulating all distances as the cost. The formula is expressed as follows:

[0113] (15)

[0114] (16)

[0115] in, For length is The sequence of actions to calculate the price. It is a forward kinematics model, based on action commands. Calculate the expected pose of the robot's end effector. The spatial location of the mission objective. It represents Euclidean distance.

[0116] Current cost It is the remaining sequence of the queue. The trajectory cost. Its mathematical expression is:

[0117] (17)

[0118] Forward-looking costs It is a look-ahead action sequence Overlapping action sequences The trajectory cost. Its mathematical expression is:

[0119] (18)

[0120] Step 3.1.4: If the difference between the prospective cost and the current cost is greater than the cost threshold, then the prospective action sequence is not accepted; otherwise, the prospective action sequence is accepted.

[0121] This embodiment has a preset cost threshold. A threshold used to determine whether the quality of an action sequence is acceptable; when the evaluation result meets... At that time, determine the look-ahead action sequence The decision is rejection. In this case, the prospective action sequence is discarded. Simultaneously, immediately resend the lookahead replanning request to the server. If the evaluation result satisfies... Then determine the lookahead action sequence. The decision is accepted, and subsequent actions will be implemented to integrate the process.

[0122] Step 3.2, Fusion of the Look-Ahead Sequence. The industrial control computer receives the acceptance decision from the sequence decision-maker in step 3.1, and then calls the sequence updater to fuse the look-ahead action sequence. With execution queue Merge and update the execution queue .

[0123] Step 3.2.1: Process the remaining sequence in the queue. and overlapping action sequences Perform weighted fusion to generate a fusion sequence with a smooth transition. .

[0124] fusion function It is a tool for combining two action sequences of equal length. and The function for fusion. Through fusion weights. Linearly weighting the two sequences generates a smooth transition sequence. This is to achieve a smooth transition between old and new action commands. The calculation process is as follows:

[0125] (19)

[0126] (20)

[0127] in, and The forward cost and the current cost calculated in step 3.1.3 are compared separately. To integrate weights, This is a smoothing factor.

[0128] Formula (19) calculates the adaptive fusion weights. The quality assessment cost is mapped to the [0,1] interval. Formula (20) uses fusion weights. As parameters, a linear weighting function is used to calculate the action sequence. and By merging the weights into a single sequence, the generated action instructions tend to be better planned when there are significant differences in quality, and achieve a smooth transition when the quality is similar, thereby improving the smoothness and stability of control during the execution phase.

[0129] Fusion sequence The fusion function is called by the sequence updater. Generate, used to replace the remaining sequence in the queue. The transition sequence of actions. Based on fusion weights. Implement overlapping action sequences With the remaining sequence of the queue The fusion of . Its mathematical expression is:

[0130] (twenty one)

[0131] Step 3.2.2: Call the sequence replanning operation of the execution queue to replace the unexecuted remaining sequences in the execution queue with the fused sequences.

[0132] Step 3.2.3: Call the sequence expansion operation of the execution queue to add the future expansion sequence (excluding overlapping action sequences) in the look-ahead action sequence to the end of the execution queue.

[0133] Future extended sequence It is a look-ahead action sequence In the overlapping region The next time step after the end Beginning, up to the look-ahead action sequence end The part that is considered in the look-ahead action sequence, excluding the overlapping action sequence. Its mathematical expression is:

[0134] (twenty two)

[0135] in, The length of the overlapping action sequence. Forward action sequence The total length.

[0136] Execution queue The update process is executed by the sequence updater calling the execution queue. The sequence reprogramming and sequence expansion operations are completed, which is a two-stage process of replacement followed by expansion. Its mathematical expression is:

[0137] (twenty three)

[0138] The sequence replanning operation will execute the queue. The remaining sequence in the queue Replace with an equal-length fusion sequence The sequence expansion operation looks ahead to the action sequence. The middle represents a future expansion sequence of entirely new actions. Added to the updated execution queue in chronological order. end.

[0139] Step 4: Collect the latest observation information cyclically at a fixed control frequency, monitor whether the conditions for forward planning are met, and acquire and execute control commands to drive the robot until robot control is complete. In this embodiment, steps 2 to 4 are continuously iterated until the task is completed or the system receives a termination command.

[0140] Example 2:

[0141] This embodiment implements the method of the present invention on a specific robot system. The system includes:

[0142] Robot Body: The robot platform is Astrapro, equipped with two seven-DOF robotic arms. The robotic arms use Yiyou PH series harmonic servo motors, and the end effectors use Junduo RG series electric grippers. The sensing system is equipped with three Intel RealSense depth cameras, which are installed on the wrists of the left and right robotic arms and the head, respectively, providing RGB-D data with a resolution of 640×480.

[0143] Target object: A red cube (5cm on each side) located on the workbench, whose spatial position is identified by the robot's wrist camera.

[0144] Industrial control computer: It adopts the Super En Intelligent spc-5100 industrial wide-temperature embedded host, equipped with an Intel(R) Core(TM) i7-8665UE processor and 8GB of memory, running the ROS2 Humble operating system, which is responsible for the robot's underlying control and algorithm scheduling.

[0145] Server-side: Workstation equipped with a GeForce RTX 5090 graphics card, with 32GB of video memory used to support the VLA model. The operating system is Ubuntu 22.04, deployed through a virtual environment and its dependencies.

[0146] Network communication: two main types of observation messages and action messages are defined, connected through a gigabit Ethernet switch, and communicate using the TCP protocol. According to the content of transmission, two main message types are defined: observation messages sent by the client and action messages returned by the server. The observation message and the action message are defined as follows:

[0147] The observation message is used to request action planning from the server and is structured data in the form of key-value pairs, containing the following fields:

[0148] Joint_state: current robot joint position array, data type float32

[16] , arranged in the order [left arm joints 1-7, left gripper opening and closing, right arm joints 1-7, right gripper opening and closing].

[0149] Images: 3 arrays of size (640, 480, 3) from the head, left wrist, and right wrist cameras.

[0150] Language_command: text instructions describing the current task, such as "grab the red square".

[0151] The action message is used for the server's response to the planning request and is structured data in the form of key-value pairs, containing the following fields:

[0152] Action_sequence: planned future action sequence, data type float32[40, 16], first dimension 40 represents 40 future time steps, second dimension 16 represents action instructions at each time step, arranged in the order [left arm joints 1-7, left gripper opening and closing, right arm joints 1-7, right gripper opening and closing].

[0153] Inference_time_ms: inference time consumed by the server-side VLA model to complete this planning.

[0154] Specifically, the embodiment of the present application is based on a dynamic forward-looking fusion robot continuous action control method, which mainly includes the following steps:

[0155] Step S1, initialization and first round planning. The industrial computer collects observation information including joint angles, wrist camera images, and "grab the red square" language instructions and sends it to the server. The server VLA model generates a 40-step action sequence through the diffusion process , returns and loads it into the execution queue .

[0156] Step S2: Dynamic lookup and asynchronous requests. The industrial computer executes the queue at a frequency of 2Hz. The instructions in the code. The parameters of the look-ahead controller are set as follows: , , , , .like Figure 4 As shown, the dynamic look-ahead interval occurs when the robot's end effector moves away from the target. Increase to nearly 20 steps; when approaching the target... The process is quickly reduced to 5 steps, triggering more frequent lookahead requests.

[0157] Step S3: Look-ahead sequence evaluation and processing. In the sequence decision-maker, a preset cost threshold is used. The value is 0.1 when the look-ahead sequence is received. When using the cost function Calculate the current cost separately and the cost of looking ahead In one request of this embodiment, it was measured that... , The difference is less than the preset threshold. The decision was to accept it.

[0158] Step S4: Fusion of look-ahead sequences. In the sequence updater, a preset smoothing factor is used. It is 0.01. Current lookahead sequence. When evaluating using a sequence decision maker, the fusion weights are calculated according to formula (19). Subsequently, the overlapping action sequences... Remaining sequence and fusion weight As a fusion function The input is used to generate a fusion sequence. And replace the remaining sequence Finally, the future extended sequence will be... Append to the end of the queue. In this example of sequence fusion, the fusion weights are calculated. The values ​​are 0.537 and 0.526 respectively, and the fusion results are as follows: Figure 5 As shown, it can generate a fusion sequence with smooth transitions between joints.

[0159] Step S5: Repeat steps S2 to S4 until the gripper successfully grasps the target object.

[0160] The embodiment details a method of server asynchronous planning and industrial computer continuous control based on a VLA model. By constructing a dynamic forward-looking trigger mechanism and action execution fusion, the large model inference delay is completely hidden in the robot action execution time, providing a new solution for high real-time robot operation in real environment.

[0161] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0162] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as a limitation on the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A dynamic look-ahead fusion robot control method based on a VLA model, characterized in that, Involving industrial control computers and servers, the dynamic look-ahead fusion robot control method based on the VLA model, implemented on the industrial control computer side, includes the following steps: The initial observation information is collected by the observation node and uploaded to the server. The server calls the VLA model and generates an initial action sequence based on the initial observation information. The initial action sequence is added to the end of the execution queue. Monitor whether the conditions for forward planning are met, and call the instruction dequeue operation in the execution queue to obtain and execute control instructions to drive the robot; If the conditions for a forward planning request are met, a forward planning request is asynchronously initiated to the server based on the latest observation information, and the forward action sequence returned by the server is received. Based on the execution queue and the look-ahead action sequence, a cost function is used to evaluate whether to accept the look-ahead action sequence. If it is not accepted, the look-ahead action sequence is discarded, and a new look-ahead request is asynchronously sent to the server based on the latest observation information; otherwise, the execution queue and the look-ahead action sequence are merged to update the execution queue. The system continuously collects the latest observation information at a fixed control frequency, monitors whether it meets the conditions for forward planning, and acquires and executes control commands to drive the robot until robot control is completed.

2. The dynamic look-ahead fusion robot control method based on the VLA model according to claim 1, characterized in that, The execution queue has three operations: instruction dequeue operation, sequence replanning operation, and sequence expansion operation. The instruction dequeue operation is used to output the head instruction in the execution queue; The sequence replanning operation is used to replace the remaining unexecuted sequences in the execution queue with action sequences of equal length; The sequence expansion operation is used to append an action sequence to the end of the execution queue.

3. The dynamic look-ahead fusion robot control method based on the VLA model according to claim 1, characterized in that, Whether the monitoring meets the conditions for forward planning requests includes: Calculate the dynamic look-ahead interval based on the latest observation information; Determine whether the current action execution count is equal to or greater than the dynamic look-ahead interval; If the current action execution count is equal to or greater than the dynamic look-ahead interval, it indicates that the look-ahead planning request condition is met, and the action execution count is set to zero; otherwise, it indicates that the look-ahead planning request condition is not met, and the action execution count is incremented by one after executing a control instruction.

4. The dynamic look-ahead fusion robot control method based on the VLA model according to claim 3, characterized in that, The dynamic look-ahead interval is calculated before the control command is executed. The calculation of the dynamic look-ahead interval based on the latest observation information includes: Based on the latest observation information, calculate the spatial distance between the robot's end effector and the task target; The distance is normalized to obtain the normalized real-time distance; An exponential mapping function is used to map the normalized real-time distance to the look-ahead interval, thus obtaining the final dynamic look-ahead interval.

5. The dynamic look-ahead fusion robot control method based on the VLA model according to claim 1, characterized in that, The prospective action sequence is generated by the server calling the VLA model and based on the latest observation information.

6. The dynamic look-ahead fusion robot control method based on the VLA model according to claim 1, characterized in that, The process of evaluating whether to accept a look-ahead action sequence based on an execution queue and a look-ahead action sequence using a cost function includes: Retrieve the remaining unexecuted sequence from the execution queue; Extract the overlapping action sequence from the look-ahead action sequence, starting from the time the look-ahead action sequence is returned by the server, and whose length is equal to the length of the remaining sequence in the queue; Input the remaining sequence in the queue and the overlapping action sequence into the cost function to obtain the current cost and the look-ahead cost; If the difference between the prospective cost and the current cost is greater than the cost threshold, then the prospective action sequence is not accepted; otherwise, the prospective action sequence is accepted.

7. The dynamic look-ahead fusion robot control method based on the VLA model according to claim 6, characterized in that, The cost function is calculated as follows: Take the action sequence to be calculated, and based on each control command in the action sequence, estimate the expected robot end pose corresponding to the control command through a forward kinematics model; Calculate the distance between the expected robot end-effector pose and the spatial location of the task target, and sum all distances as the cost.

8. The dynamic look-ahead fusion robot control method based on the VLA model according to claim 1, characterized in that, The fusion of the execution queue and the look-ahead action sequence to update the execution queue includes: Retrieve the remaining unexecuted sequence from the execution queue; Extract the sequence from the look-ahead action sequence that starts from the look-ahead action sequence returned by the server and has a length equal to the length of the remaining sequence in the queue as the overlapping action sequence; The remaining sequence in the queue and the overlapping action sequence are weighted and fused to generate a fused sequence; Invoke the sequence replanning operation of the execution queue to replace the remaining unexecuted sequences in the execution queue with the fused sequences; Invoke the sequence expansion operation of the execution queue to add the future expansion sequence (excluding overlapping action sequences) from the lookahead action sequence to the end of the execution queue.

9. The dynamic look-ahead fusion robot control method based on the VLA model according to claim 8, characterized in that, The step of weighted fusion of the remaining sequence in the queue and the overlapping action sequence to generate a fused sequence includes: The fusion weights are calculated based on the current cost and the forward cost, as follows: ； in, To integrate weights, and These are the forward-looking costs and the current costs, respectively. It is a smoothing factor; By fusion weights The remaining sequence in the queue and the overlapping action sequence are linearly weighted to generate a fused sequence.

Images