A control method and device for a dual-arm robot based on visual language action models

By standardizing the processing of multi-view images and body state data of the robot, and combining them with a preset motion generation model, the problems of the executability and stability of visual language motion models on real robots were solved, and stable operation in a home environment was achieved.

CN123103683APending Publication Date: 2026-11-17BEIJING SIHETIANDI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611561221.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-10
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, visual language action models have poor executability and smoothness of action execution on robots. Especially in home environments with a variety of objects, random positions, and frequent occlusion, the actions generated by the model are difficult to use directly to control real robots. Furthermore, the time spent on model inference and communication can easily lead to discontinuities between adjacent generated actions.

Method used

By standardizing the multi-view image data and body state data of the robot, standardized model input data is constructed. The model generates action blocks using preset action generation model. The model actions are adapted into robot-executable actions according to the preset action space mapping relationship, forming closed-loop control and coordinating the actions of the robotic arm, gripper and moving base.

Benefits of technology

It improves the executableness of model-generated actions on real robots and the smoothness of action execution, reduces robotic arm tremors and sudden gripper movements, and achieves stable operation in unstructured environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123103683A_ABST
    Figure CN123103683A_ABST
Patent Text Reader

Abstract

This application relates to the field of robotics technology and discloses a control method and device for a dual-arm robot based on a visual language action model. The method includes: acquiring multi-view image data, body state data, and natural language task instructions for the robot; standardizing the multi-view image data and body state data to obtain standardized multi-view image data and standardized state data; constructing model input data based on the standardized multi-view image data, standardized state data, and natural language task instructions; processing the model input data through a preset action generation model to obtain action blocks including multiple model actions; adapting each model action to the action space according to a preset action space mapping relationship to obtain robot actions; sending robot actions to the robot at a preset control frequency and acquiring updated observation data to construct model input data for the next inference cycle.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and in particular to a control method and apparatus for a dual-arm robot based on a visual language action model. Background Technology

[0002] With the development of embodied intelligence technology, service robots need to perform tasks such as recognition, grasping, carrying, and sorting based on users' natural language commands in real-world environments. For example, in a home environment, there are many types of objects, their placement is random, and occlusion is frequent. Robots are usually equipped with cameras with multiple perspectives, as well as actuators such as robotic arms and grippers. They also generate actions based on collected observation data through motion generation models to control the robot to perform corresponding tasks.

[0003] In the process of controlling a robot through motion generation models, there are often differences between the input space and motion space used by the model during training and the observation space and execution space of the real robot. The motions generated by the model are difficult to use directly to control the real robot. Furthermore, the frequency of motion generation by the model is usually lower than the frequency of robot execution of control commands. The time consumption of model inference and communication can also easily lead to discontinuity between two adjacent generated motions.

[0004] Improving the executableness and smoothness of model-generated actions on robots has become an urgent technical problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides a dual-arm robot control method and apparatus based on a visual language action model, which can improve the executableness of model-generated actions on real robots and the smoothness of action execution. Specifically, the embodiments of this application disclose the following technical solutions.

[0006] The first aspect of this application provides a dual-arm robot control method based on a visual language action model. The method includes: acquiring multi-view image data of the robot, robot body state data, and natural language task instructions; wherein the body state data includes the joint states of the left robotic arm, the right robotic arm, the left gripper, and the right gripper; performing image standardization processing on the multi-view image data to obtain standardized multi-view image data, and performing state standardization processing on the body state data to obtain standardized state data; constructing model input data based on the standardized multi-view image data, standardized state data, and natural language task instructions; and generating model input data through a preset action generation model. The input data is processed to obtain action blocks; each action block includes multiple model actions arranged in execution order; according to a preset action space mapping relationship, the action space of each model action in the action block is adapted to obtain the robot action corresponding to each model action; the robot action includes the control variables of the left robotic arm joint, the right robotic arm joint, the left gripper, and the right gripper; the robot action is sent to the robot at a preset control frequency, and the updated observation data when the robot executes the robot action is obtained; the updated observation data includes updated multi-view image data and updated body state data, and the updated observation data is used to construct the model input data for the next inference cycle.

[0007] In this embodiment, multi-view image data, ontology state data, and natural language task instructions collectively constitute the input to the preset action generation model. Multi-view image data describes the robot's environment and visual information near the end effector of the robotic arm; ontology state data describes the robot's current mechanical state; and natural language task instructions describe the task the user expects the robot to complete. The preset action generation model outputs action blocks, where multiple model actions have an execution order. By adapting the model actions to the action space, the actions in the model space can be converted into robot actions that can be received by the actual robot control interface.

[0008] In some embodiments, the robot further includes a mobile base, and the body state data includes base motion state data of the mobile base. The robot actions also include base motion control quantities for controlling the mobile base. The base motion state data includes base linear velocity and base angular velocity, and the base motion control quantities include base linear velocity control quantities and base angular velocity control quantities.

[0009] By incorporating the base motion state into the body state data and the base motion control quantity into the robot's actions, the preset motion generation model can generate actions related to the robotic arm's actions, gripper actions, and base motion in a unified motion space, thereby supporting the robot to adjust its position during grasping, handling, and placement.

[0010] In some embodiments, image standardization processing is performed on multi-view image data to obtain standardized multi-view image data, and state standardization processing is performed on ontology state data to obtain standardized state data. This includes: converting the color channel format of each image in the multi-view image data, and scaling and filling it according to a preset image size to obtain standardized multi-view image data; normalizing the joint states of the left and right robotic arms according to a preset state value range to obtain normalized joint states; correcting the orientation of each joint dimension in the normalized joint states according to a preset orientation mask to obtain standardized joint states; mapping the left and right gripper states to a preset gripper normalization interval according to a preset gripper value range to obtain standardized gripper states; and determining standardized state data based on the standardized joint states and standardized gripper states.

[0011] Both image normalization and state normalization are performed to form a data format that the model can recognize. Image normalization and state normalization can be performed sequentially or in parallel; this application does not limit the execution order of the two processes.

[0012] In some embodiments, model input data is constructed based on standardized multi-view image data, standardized state data, and natural language task instructions, including: mapping standardized multi-view image data to image field data, standardized state data to state field data, and natural language task instructions to language field data according to preset model input fields; generating an image mask for marking the valid states of each standardized image in the standardized multi-view image data; and combining the image field data, state field data, language field data, and image mask into model input data.

[0013] Image masks can be used to indicate whether each normalized image channel is valid. For example, when a camera image channel is successfully acquired and passes integrity verification, the mask value corresponding to that normalized image channel can indicate that it is valid; when a camera image channel is missing, damaged, or fails time alignment verification, the mask value corresponding to that normalized image channel can indicate that it is invalid.

[0014] In some embodiments, according to a preset motion space mapping relationship, motion space adaptation processing is performed on each model motion in the motion block to obtain the robot motion corresponding to each model motion, including: determining the dimensional correspondence between the model motion and the robot motion; according to a preset motion direction mask, the motion dimensions corresponding to the preset motion direction mask in each model motion are oriented to obtain the oriented correction motion; according to a preset gripper mapping relationship, the standardized gripper control quantities in the oriented correction motion are denormalized into left gripper control quantities and right gripper control quantities respectively to obtain the preliminary adapted motion; according to the dimensional correspondence relationship, the preliminary adapted motion is dimensionally sorted and / or dimensionally truncated to obtain the robot motion.

[0015] In some embodiments, when the number of action dimensions of the model actions is less than the number of action dimensions of the robot actions, dimension completion processing can be performed on the initially adapted actions. For example, missing dimensions not provided by the model actions in the robot actions can be filled with default neutral control variables. The default neutral control variable can be a control variable that does not cause the corresponding moving part to produce active movement, or it can be a control variable that remains consistent with the current state of the robot.

[0016] In some embodiments, the model input data is processed by a preset action generation model to obtain action blocks, including: acquiring historical periodic action blocks, inference delay steps, and execution window length; wherein, the inference delay steps are determined based on model inference time, communication time, and a preset control frequency; determining action continuity constraints based on historical periodic action blocks, inference delay steps, and execution window length; the action continuity constraints are used to constrain the differences between the actions of the generated action blocks within a preset time alignment interval and the corresponding actions in the historical periodic action blocks, the preset time alignment interval being determined based on the inference delay steps and execution window length; inputting the model input data into the preset action generation model, and guiding and correcting the action sampling process based on the action continuity constraints to obtain action blocks.

[0017] The historical action block can be an action block generated in the previous inference cycle, or an action block that was generated before the current inference cycle and has not yet been partially executed. By using the inference delay steps and the execution window length, the interval between the new action block and the historical action block that needs to be time-aligned can be determined, so that the newly generated action block remains continuous with the historical action block in the vicinity of actions that the robot has already executed or is about to execute.

[0018] In some embodiments, when the preset inference frequency of the preset action generation model is less than the preset control frequency, sending robot actions to the robot according to the preset control frequency includes: writing robot actions into an action buffer; when there are cached robot actions in the action buffer that have not yet been executed, performing weighted fusion of the cached robot actions and the actions of the robot actions at corresponding times during the overlapping execution period to obtain a fused action; when there are no cached robot actions in the action buffer that have not yet been executed, determining the robot action as a fused action; when the action difference between two adjacent fused actions is greater than a preset difference threshold, inserting an intermediate action between the two adjacent fused actions according to a preset interpolation step number, and writing the intermediate action into the action buffer; reading the action to be sent from the action buffer according to the preset control frequency and action execution order, and sending the action to be sent to the robot; wherein the action to be sent includes at least one of a fused action and an intermediate action.

[0019] This application's embodiments coordinate the difference between the model's inference frequency and the robot's control frequency through an action buffer, and perform weighted fusion during the overlapping execution period of action blocks, which can reduce abrupt action changes when switching between adjacent action blocks. By inserting intermediate actions between adjacent fused actions with large action differences, the changes in the robot's underlying control commands can also be made more continuous.

[0020] A second aspect of this application provides a dual-arm robot control device based on a visual language action model. The device includes: an acquisition module configured to acquire multi-view image data of the robot, robot body state data, and natural language task instructions; wherein the body state data includes the joint states of the left robotic arm, the joint states of the right robotic arm, the state of the left gripper, and the state of the right gripper; a standardization module configured to perform image standardization processing on the multi-view image data to obtain standardized multi-view image data, and to perform state standardization processing on the body state data to obtain standardized state data; a construction module configured to construct model input data based on the standardized multi-view image data, the standardized state data, and the natural language task instructions; and a model processing module configured to use preset motion... The generative model processes the model input data to obtain action blocks; each action block includes multiple model actions arranged in execution order; the action generation module is configured to perform action space adaptation processing on each model action in the action block according to a preset action space mapping relationship to obtain the robot action corresponding to each model action; the robot action includes the control variables of the left robotic arm joint, the right robotic arm joint, the left gripper, and the right gripper; the control module is configured to send robot actions to the robot at a preset control frequency and acquire updated observation data when the robot executes robot actions; the updated observation data includes updated multi-view image data and updated body state data, and the updated observation data is used to construct the model input data for the next inference cycle.

[0021] A third aspect of this application provides a computer-readable storage medium storing a computer program for executing the dual-arm robot control method based on a visual language action model provided in any of the embodiments of the first aspect above.

[0022] A fourth aspect of this application provides an electronic device, which includes a processor and a memory for storing processor-executable instructions; the processor is configured to read executable instructions from the memory and execute the instructions to implement the dual-arm robot control method based on a visual language action model provided in any of the embodiments of the first aspect above.

[0023] The fifth aspect of this disclosure provides a computer program product, which includes a computer program that, when executed by a processor, implements the dual-arm robot control method based on a visual language action model as described in the first aspect.

[0024] The dual-arm robot control method based on a visual language action model provided in this disclosure standardizes the robot's multi-view image data and body state data, and constructs model input data based on the standardized multi-view image data, standardized state data, and natural language task instructions. This converts the observation data of the real robot into an input format that can be processed by a preset action generation model. The preset action generation model processes the model input data to obtain action blocks including multiple model actions, and adapts each model action into a robot action that can be executed by the robot according to a preset action space mapping relationship. Then, the robot action is sent to the robot at a preset control frequency, and updated observation data is obtained to construct the model input data for the next inference cycle, forming a closed-loop control. This improves the executability of the model-generated actions on the real robot and the smoothness of action execution. Attached Figure Description

[0025] Figure 1 A schematic diagram of a dual-arm robot control system framework based on a visual language action model, provided for an embodiment of the application; Figure 2 A schematic flowchart of a dual-arm robot control method based on a visual language action model is provided for an embodiment of this application; Figure 3 A schematic diagram of a dual-arm robot control device based on a visual language action model is provided for an embodiment of this application; Figure 4 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0026] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0027] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0028] With the development of embodied intelligence technology and home service robot technology, robots need to complete tasks such as recognition, grasping, carrying, and tidying up based on natural language instructions in real home environments. The home environment is a typical unstructured environment: there are many types of objects in the home with different shapes, the placement of objects often changes, and objects often obstruct each other; moreover, the natural language instructions for the same type of task are expressed in a variety of ways, such as "grab the clothes on the table", "pick up the banana", "move the cup to the coffee table", etc.

[0029] One implementation in related technologies employs a layered architecture comprising an object detection model, pose estimation, and motion planning. First, the object detection model detects the target object in the image. Then, pose estimation determines the object's position and orientation. Finally, the motion planning module generates the robotic arm's trajectory and controls its execution. While this layered architecture is effective in structured industrial scenarios, it is less suitable for home environments. Home environments present diverse object shapes, frequent occlusion, and significant lighting variations. Furthermore, the task objectives often require semantic understanding using natural language. Relying solely on fixed-category object detection and preset grasping postures is ill-suited to the diverse types of objects, significant pose variations, and flexible task expression. Moreover, errors in intermediate stages such as detection, pose estimation, and motion planning accumulate at each stage, impacting the final grasping success rate.

[0030] Another approach is to employ an end-to-end robot control strategy, where the model is trained using teaching data and then directly outputs actions based on observation data. However, this type of end-to-end strategy is typically designed for single robotic arms, fixed cameras, or specific experimental platforms, lacking a unified adaptation mechanism for dual robotic arms, wrist cameras, and mobile bases. When the joint orientation, gripper's physical value range, motion dimensions, and base control method of the real robot differ from the model's training data, issues such as reversed motion directions of the left and right robotic arms, incorrect gripper opening / closing ratios, and abnormal base speeds can easily occur, making it difficult to stably transfer the model to a real home robot. Furthermore, if the model's output actions are directly sent to the robot, abrupt changes can easily occur between adjacent generated actions when the model inference frequency is lower than the robot's control frequency and the model inference and communication times are long. This can cause issues such as robotic arm tremors, wrist posture jumps, or sudden gripper opening / closing.

[0031] To address the aforementioned issues, this disclosure provides a dual-arm robot control method based on a visual language action model. This method standardizes the robot's multi-view image data and body state data, and constructs model input data based on the standardized multi-view image data, standardized state data, and natural language task instructions. This converts the observation data of the real robot into an input format that can be processed by a preset action generation model. The preset action generation model processes the model input data to obtain action blocks comprising multiple model actions. These model actions are then adapted into robot actions executable by the robot according to a preset action space mapping relationship. The robot actions are then sent to the robot at a preset control frequency, and updated observation data is acquired to construct the model input data for the next inference cycle, forming a closed-loop control. This improves the executableness of the model-generated actions on the real robot and the smoothness of action execution.

[0032] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0033] Figure 1 This is a schematic diagram of a dual-arm robot control system based on a visual language action model, provided as an embodiment of this disclosure. The following is in conjunction with... Figure 1 The application scenarios of the embodiments of this disclosure will be described.

[0034] like Figure 1 As shown, the dual-arm robot control system based on visual language action model may include robot 100 and control device 120.

[0035] For example, robot 100 can be a dual-arm robot designed for home service scenarios. Figure 1 As shown, the robot 100 may include a high-position camera 110, a left wrist camera 111, a right wrist camera 112, a left robotic arm 113, a right robotic arm 114, a left gripper 115, a right gripper 116, and a mobile base 117.

[0036] In some examples, the high-position camera 110 is positioned at the head of the robot 100 and may also be referred to as a head camera or global camera. The high-position camera 110 is used to acquire an overall image of the environment in which the robot 100 is located. For example, the high-position camera 110 can acquire images that include the desktop, the ground, furniture, and the spatial relationships between the objects in order to determine the overall position of the target object 130 in the environment.

[0037] like Figure 1As shown, the target object 130 can be placed on the placement platform 140 (such as a desktop, coffee table, etc.). The image captured by the high-position camera 110 can simultaneously include the target object 130 and the placement platform 140, thereby reflecting the positional relationship between the target object 130 and the placement platform 140.

[0038] In some examples, the left wrist camera 111 is positioned on the wrist of the left robotic arm 113 (i.e., the end of the left robotic arm 113 near the left gripper 115) to acquire local images near the left gripper 115; the right wrist camera 112 is positioned on the wrist of the right robotic arm 114 (i.e., the end of the right robotic arm 114 near the right gripper 116) to acquire local images near the right gripper 116. The wrist cameras can move with the corresponding robotic arm. After the gripper approaches the target object 130, the wrist cameras can acquire local detail images such as the relative position between the gripper and the target object 130, the edge of the target object 130, the handle, or the graspable area, thus compensating for the shortcomings of the high-position camera 110, which is easily obscured by the robotic arm itself at close range and makes it difficult to see the grasping details.

[0039] For example, the left robotic arm 113 and the right robotic arm 114 may each include multiple joints. For example, they may each include a shoulder joint, an elbow joint, a wrist joint, etc., and each joint can rotate under the drive of a corresponding joint motor. The left gripper 115 is connected to the end of the left robotic arm 113, and the right gripper 116 is connected to the end of the right robotic arm 114. The left gripper 115 and the right gripper 116 can open or close under the drive of the drive mechanism to grip or release the target object 130. The movable base 117 is used to move the robot 100 as a whole. For example, the movable base 117 can be a wheeled base, and the robot 100 can move in the home environment by controlling the linear velocity and angular velocity of the base.

[0040] It should be noted that, Figure 1 The structure of the robot 100 shown is merely an example. In other examples, the robot 100 may not include the movable base 117, i.e., the robot 100 may be a fixed dual-arm robot; or, the robot 100 may include two or more cameras, which is not limited in this embodiment.

[0041] For example, the control device 120 is used to execute the dual-arm robot control method based on a visual language action model provided in this embodiment of the disclosure. The control device 120 can be a local computing device (e.g., a computing unit mounted on the robot) installed on the robot 100, or a remote server communicatively connected to the robot 100. Alternatively, some functions of the control device 120 may be deployed on the local computing device of the robot 100, and some functions may be deployed on the remote server; this embodiment of the disclosure does not limit this. The control device 120 and the robot 100 can communicate via a wired network or a wireless network, such as a local area network (LAN) or a wireless local area network (WLAN).

[0042] In some examples, such as Figure 1 As shown, the control device 120 may deploy a preset action generation model 121, which can be a visual language action model used to generate action blocks based on the input model input data. The control device 120 can also perform data acquisition and standardization processing, model input data construction, action space adaptation and buffer smoothing, etc. The specific processes of the above processing will be described in detail in the following embodiments.

[0043] like Figure 1 As shown, robot 100 can send the collected multi-view image data, body state data, and other observation data to control device 120; control device 120 can send the processed robot actions to robot 100 according to a preset control frequency to control the left robotic arm 113, right robotic arm 114, left gripper 115, right gripper 116, and mobile base 117 to perform corresponding actions (such as grasping and carrying the target object 130); during the execution of robot actions, robot 100 can also feed back updated observation data to control device 120, and update the observation data to build the model input data for the next inference cycle, thereby forming a closed-loop control of first observing, inferring from the observation results, then controlling the robot to perform corresponding actions based on the inference results, and then observing the robot to perform actions again.

[0044] The following describes in detail, with reference to the accompanying drawings, the dual-arm robot control method based on visual language action model provided in the embodiments of this disclosure.

[0045] Figure 2 This is a flowchart illustrating a dual-arm robot control method based on a visual language action model, provided in an embodiment of this disclosure. This dual-arm robot control method based on a visual language action model can be applied to the aforementioned... Figure 1The dual-arm robot control system shown is based on a visual language action model and can be applied, for example, to control device 120 in a robot control system. Figure 2 As shown, the dual-arm robot control method based on visual language action model may include steps 210 to 260 as shown below.

[0046] Step 210: Acquire multi-view image data of the robot, robot body state data, and natural language task instructions.

[0047] In some examples, the control device can acquire multi-view image data of the robot, robot body state data, and natural language task instructions at the beginning of each inference cycle. These data together constitute the robot's observation data at the current moment.

[0048] For example, multi-view image data refers to image data captured by cameras on the robot from multiple perspectives. Figure 1 Taking the robot shown as an example, the multi-view image data can include a global image captured by a high-position camera, a left wrist image captured by a left wrist camera, and a right wrist image captured by a right wrist camera. The global image reflects the overall scene of the robot's environment, such as the spatial relationship between the target object and the tabletop or obstacles; the left and right wrist images reflect local details near the corresponding grippers, such as the relative positional relationship between the grippers and the target object.

[0049] Assuming the robot is The observation data at time t can be expressed in the form shown in formula (1):

[0050] In formula (1), Indicates that the robot is Observational data at any given time; Indicates the high-position camera is in Global images captured at different times, where the superscript Indicates a global view; Indicates the left wrist camera is in Images of the left wrist captured at any time. Indicates the camera on the right wrist. Images of the right wrist captured at all times. Indicates that the robot is The body's state data is collected in real time. This indicates instructions for natural language tasks.

[0051] For example, the body state data is data used to reflect the current motion state of the robot body. The body state data includes the joint states of the left robotic arm, the joint states of the right robotic arm, the state of the left gripper, and the state of the right gripper. Among them, the joint state of the left robotic arm may include the joint angles of each joint of the left robotic arm (e.g., the rotation angle of each joint), and the joint state of the right robotic arm may include the joint angles of each joint of the right robotic arm; the state of the left gripper is used to indicate the opening and closing degree of the left gripper, such as the opening and closing angle or the opening and closing stroke of the left gripper, and the state of the right gripper is used to indicate the opening and closing degree of the right gripper, such as the opening and closing angle or the opening and closing stroke of the right gripper.

[0052] In some examples, the ontology state data can be represented as a state vector as shown in the following formula (2):

[0053] In formula (2), Indicates that the robot is The entity's state data at any given moment; It represents the joint state of the left robotic arm, for example, it can be a vector composed of the joint angles of each joint of the left robotic arm; Indicates the state of the left gripper; Indicates the joint status of the right robotic arm; Indicates the state of the right gripper; This indicates the linear velocity of the moving base; This represents the angular velocity of the moving base. It should be noted that if the robot does not include the moving base, the linear velocity of the base may not be included in the body state data. and base angular velocity .

[0054] For example, the control device can read joint information of each joint of the left and right robotic arms, the opening and closing amounts of the left and right grippers, and the motion state of the moving base from the robot's controller to obtain the body state data. For instance, the control device can send a state query request to the robot controller according to a preset sampling period and receive joint encoder readings, gripper position feedback, and base speed feedback returned by the robot controller.

[0055] For example, a natural language task instruction is an instruction input by the user in natural language form to describe the content of the task to be performed. For example, a natural language task instruction may be "grab the clothes on the table", "pick up the banana", "move the cup to the coffee table", etc. Natural language task instructions can be obtained by the user through voice input and then through speech recognition, or they can be directly input by the user through text input. This embodiment of the disclosure does not limit this.

[0056] In some examples, since the acquisition times of various cameras and robot controllers may not be completely consistent, the control device can also perform timestamp alignment or approximate alignment of multi-view image data and body state data to form observation data under a unified control cycle. For example, the control device can combine images and body states acquired at the same time or within adjacent preset time periods into observation data for the same inference cycle.

[0057] In some embodiments, the robot may further include a mobile base, the body state data may further include the base motion state data of the mobile base, and the robot actions may further include base motion control quantities for controlling the mobile base; wherein, the base motion state data includes the base linear velocity and the base angular velocity, and the base motion control quantities include the base linear velocity control quantity and the base angular velocity control quantity.

[0058] For example, the base motion state data is used to reflect the current motion of the moving base. The base linear velocity is used to indicate the translational speed of the moving base along the direction of travel, and the base angular velocity is used to indicate the rotational speed of the moving base about the vertical direction.

[0059] For example, for a wheeled mobile base, the control device can calculate the linear velocity and angular velocity of the base based on the wheel speeds of the left and right wheels. The linear velocity control variable is used to control the translational speed of the mobile base, and the angular velocity control variable is used to control the steering speed of the mobile base.

[0060] The dual-arm robot control method based on visual language action model provided in this disclosure incorporates the base motion state data into the body state data and incorporates the base motion control quantity into the robot action. This enables the preset action generation model to coordinate the actions of the dual arms, grippers, and moving base in a unified strategy, thereby supporting the robot to complete continuous tasks that require simultaneous movement and operation.

[0061] Step 220: Perform image standardization processing on the multi-view image data to obtain standardized multi-view image data, and perform state standardization processing on the ontology state data to obtain standardized state data.

[0062] In some examples, the output format, resolution, color channel order, and data dimensions of different cameras may vary. Furthermore, the joint value range, joint orientation, and gripper physical range of the real robot may differ from the state space used during the training of the preset motion generation model. Directly inputting the raw observation data into the model would lead to inconsistencies between the training and deployment phases. Therefore, the control device can perform image standardization on multi-view image data and state standardization on the ontology state data to convert the observation data of the real robot into a standard form that the model can process.

[0063] It should be noted that there is no strict order of execution between image normalization processing of multi-view image data and state normalization processing of ontology state data. In some examples, the two can be executed in parallel, for example, by different processing threads; in other examples, they can be executed in any order, and this disclosure does not limit this.

[0064] In some embodiments, step 220, which involves performing image standardization processing on the multi-view image data to obtain standardized multi-view image data, may include: converting the color channel format of each image in the multi-view image data, and scaling and filling it according to a preset image size to obtain standardized multi-view image data.

[0065] For example, color channel format conversion is used to unify the color channel format of various images to the color channel format required by the model. For instance, some camera output images are in BGR (Blue-Green-Red) channel order, while the images used during model training are in RGB (Red-Green-Blue) channel order. In this case, the BGR format image can be converted to RGB format. As another example, grayscale images or images with transparency channels can also be converted to three-channel RGB format.

[0066] For example, the preset image size is a fixed input size required by the preset action generation model, such as 224 pixels × 224 pixels. Since the original resolutions of the various cameras may be different (e.g., the resolution of a high-position camera is 1920 pixels × 1080 pixels, and the resolution of a wrist camera is 340 pixels × 480 pixels), the control device can scale each image to the preset image size, and if the scaled image does not match the scaled image size, the short side of the image is padded (e.g., pixels of a preset color, such as black pixels, are filled on the top, bottom, left, or right sides of the image) so that the padded image size matches the preset image size.

[0067] In some examples, the above image normalization process can be represented as shown in the following formula (3):

[0068] In formula (3), This refers to the image after standardization, i.e., the standardized image; This indicates a color channel format conversion operation, used to unify the input image into the RGB color channel format; This indicates scaling and filling operations, used to scale an image and fill it to a fixed size; Indicates the height of the preset image size; Indicates the width of the preset image size; This represents the original image before standardization. Each image in the multi-view image data can be processed according to formula (3), and the standardized images together constitute the standardized multi-view image data.

[0069] In some embodiments, the state standardization process performed on the body state data in step 220 to obtain standardized state data may include: normalizing the joint states of the left and right robotic arms according to a preset state value range to obtain normalized joint states; correcting the orientation of each joint dimension in the normalized joint states according to a preset orientation mask to obtain standardized joint states; mapping the left and right gripper states to a preset gripper normalization interval according to a preset gripper value range to obtain standardized gripper states; and determining standardized state data based on the standardized joint states and standardized gripper states.

[0070] For example, the preset state value range is a pre-configured range of joint state values ​​used during model training, such as the normalized interval [-1, 1] for each joint. Since the physical value range of each joint in a real robot (e.g., the range of motion of a joint expressed in angles or radians) is usually inconsistent with the value range used during the training of the preset motion generation model, the control device can normalize each joint dimension in the joint states of the left and right robotic arms according to the preset state value range, mapping the values ​​of each joint dimension to the preset state value range to obtain the normalized joint state.

[0071] For example, different robots or different models of joints may have different definitions of the positive direction of rotation. For instance, for the same joint, the joint angle increases when the joint rotates clockwise on the training data acquisition platform, while the joint angle decreases when the same joint rotates clockwise on the real robot. If no processing is done, the action output by the model will be executed in the opposite direction on the real robot.

[0072] Therefore, the control device can perform orientation correction on each joint dimension in the normalized joint state according to a preset orientation mask. The preset orientation mask can be a vector with a value of 1 or -1 that corresponds one-to-one with the joint dimension: for joint dimensions whose joint orientation definition is consistent with the training data, the corresponding mask element has a value of 1, and the value of the joint dimension remains unchanged; for joint dimensions whose joint orientation definition is opposite to the training data, the corresponding mask element has a value of -1, and the value of the joint dimension is inverted, thereby completing the orientation correction and obtaining the standardized joint state.

[0073] In some examples, the process of normalizing and correcting the orientation of ontology state data can be expressed as shown in the following formula (4):

[0074] In formula (4), This represents standardized state data, i.e., the standard input state of the preset action generation model; Indicates the preset direction mask. It is a vector that corresponds one-to-one with each dimension of the state vector, and its elements take the value 1 or -1; This represents a normalization operation, used to map each dimension in the ontology state data to a preset range of state values. This represents the ontology state data before standardization. It should be noted that for dimensions such as base linear velocity and base angular velocity, if their direction definitions differ from the training data, direction correction can also be performed using the corresponding elements in the preset direction mask.

[0075] For example, the physical representation of the opening and closing amount of the gripper may differ on different robots. For instance, some grippers represent the opening and closing angle, while others represent the opening and closing stroke (such as the distance between the two grippers), and the physical value ranges also vary. The control device can map the left gripper state and the right gripper state to a preset gripper normalization interval (such as the [0,1] interval) according to the preset gripper value range (i.e., the minimum and maximum values ​​of the gripper physical quantity), respectively, to obtain the standardized gripper state. The above mapping process can be expressed in the form of the following formula (5):

[0076] In formula (5), This indicates the standardized gripper state, which is the gripper opening and closing ratio mapped to the preset gripper normalization range; It represents the actual physical quantity of the gripper, such as the opening angle or opening stroke of the gripper; This indicates the minimum value within the preset range of gripper values, such as the angle or stroke when the gripper is fully closed; This represents the maximum value within the preset range of gripper values, such as the angle or stroke when the gripper is fully open. For example, when the opening and closing stroke range of a certain gripper is 0 mm to 80 mm, and the current opening and closing stroke is 40 mm, the standardized gripper state is (40-0) / (80-0)=0.5.

[0077] In some examples, the control device can concatenate standardized joint states and standardized gripper states into a unified state vector to obtain standardized state data. When the robot includes a mobile base, the standardized state data may also include the normalized and orientation-corrected linear and angular velocities of the base.

[0078] The dual-arm robot control method based on a visual language action model provided in this disclosure converts, scales, and fills multi-view image data in color channel format, and normalizes, corrects orientation, and maps gripper data. This transforms the observation data of the real robot into a standard form that can be processed by the preset action generation model. This reduces model input errors caused by inconsistencies in camera format, image size, joint orientation, and gripper physical range during deployment, and provides a foundation for the stable operation of the model on the real robot.

[0079] Step 230: Construct model input data based on standardized multi-view image data, standardized state data, and natural language task instructions.

[0080] In some examples, the preset action generation model has fixed requirements for the field organization of the input data. The control device can organize standardized multi-view image data, standardized state data and natural language task instructions into model input data according to the preset model input fields.

[0081] In some embodiments, step 230 may include: mapping standardized multi-view image data to image field data, standardized state data to state field data, and natural language task instructions to language field data according to preset model input fields; generating an image mask for marking the effective states of each standardized image in the standardized multi-view image data; and combining the image field data, state field data, language field data, and image mask into model input data.

[0082] For example, the preset model input fields are the input field names agreed upon by the preset action generation model. For instance, the preset model input fields may include: an image field (for carrying the global image), a field (for carrying the left wrist image), and a field (for carrying the right wrist image); a state field (for carrying standardized state data); and a language field (for carrying natural language task instructions). The control device can map the standardized image corresponding to the high-position camera to the global image field, the standardized image corresponding to the left wrist camera to the left wrist image field, and the standardized image corresponding to the right wrist camera to the right wrist image field, obtaining image field data; map the standardized state data to the state field, obtaining state field data; and map the natural language task instructions to the language field, obtaining language field data.

[0083] For example, an image mask is used to mark the valid state of each standardized image in the standardized multi-view image data. For instance, for each standardized image, the image mask may include a corresponding mask element. If the standardized image is valid (i.e., the camera has currently captured an image), the corresponding mask element has a value of 1 (or true); if the standardized image is invalid (e.g., the camera is malfunctioning or completely occluded, and a preset blank image is used to fill the gap), the corresponding mask element has a value of 0 (or false). Through the image mask, the model can distinguish whether each image is valid, avoiding interference from invalid image data in action generation.

[0084] In some examples, the model input data can be represented in the form shown in formula (6):

[0085] In formula (6), express The model input data at each time point; This represents image field data; This represents the field in the image field data used to carry the global image; This represents the field in the image field data used to carry the image of the left wrist; This represents the field in the image field data used to carry the image of the right wrist; Represents an image mask; This represents the mask element in the image mask that corresponds to the global image; This represents the mask element in the image mask that corresponds to the image of the left wrist; This represents the mask element in the image mask that corresponds to the image of the right wrist; This represents the status field data. Represents standardized state data; This indicates language field data; This indicates instructions for natural language tasks.

[0086] The dual-arm robot control method based on a visual language action model provided in this disclosure maps standardized multi-view image data, standardized state data, and natural language task instructions to corresponding field data according to preset model input fields, and generates image masks that mark the valid states of each image path. The above data and image masks are combined into model input data, realizing the format unification between training data, model input, and real robot observation. This allows natural language tasks, visual information, and ontology states to be integrated under a unified preset action generation model input structure, providing a basis for action generation.

[0087] Step 240: Process the model input data using a preset action generation model to obtain action blocks.

[0088] In this embodiment of the disclosure, the preset action generation model is a pre-trained Vision-Language-Action Model (VLA), also known as a visual-language-action model, which is used to generate action blocks based on the model input data.

[0089] For example, the visual language action model can be a robot strategy model that unifies the modeling of visual observation, language instructions, robot body state and robot action generation, and can generate action blocks including multiple model actions based on multi-view image data, body state data and natural language task instructions.

[0090] For example, the visual-language-action model can be a model obtained by fine-tuning the pi05 visual-language-action basic model based on home scene teaching data. Here, pi05 is a visual-language-action basic model based on flow matching for action generation. This disclosure does not limit the specific network structure, training method, or action generation method of the visual-language-action model.

[0091] For example, the preset action generation model can be abstracted as a policy function, and its processing of the model input data can be expressed as shown in the following formula (7):

[0092] In formula (7), express Action blocks that are constantly output; This represents the policy function corresponding to the preset action generation model; express The model input data at each time point.

[0093] For example, an action block includes multiple model actions arranged in execution order. The model actions are actions output by the preset action generation model and represented in the standard action space defined during the training of the preset action generation model. An action block can be represented in the form shown in formula (8) below:

[0094] In formula (8), Indicates an action block; This indicates the first model action executed in the action block; This indicates the second model action executed within the action block; This indicates the last model action executed in the action block; This indicates the temporal length of action prediction, i.e., the number of model actions included in the action block. It is an integer greater than or equal to 2. For example, The value can be set to 30, meaning the preset motion generation model generates motion blocks containing 30 model actions each time. Compared to predicting only a single instantaneous action each time, the motion blocks can implicitly contain continuous action stages such as approaching the target, adjusting posture, closing the gripper, lifting the object, moving to the target area, and placing it, making them more suitable for home-based tasks.

[0095] The following is an exemplary description of the internal processing of a preset action generation model. In some examples, the internal processing of a preset action generation model may include four stages: visual encoding, language encoding, state encoding, and action generation.

[0096] In the visual encoding stage, the visual encoder can extract features from each standardized image in the standardized multi-view image data to obtain visual features, as shown in the following formula (9):

[0097] In formula (9), Indicates visual features; This represents a visual encoder, such as an encoder with a Vision Transformer (ViT) structure. Represents the standardized global image; This represents a standardized image of the left wrist. This represents a standardized image of the right wrist.

[0098] In the language encoding stage, the language encoder can extract features from the natural language task instructions to obtain language features, as shown in the following formula (10):

[0099] In formula (10), Indicate linguistic features; Represents a language encoder (such as an encoder with a text Transformer structure). This indicates instructions for natural language tasks.

[0100] In the state encoding stage, the state encoder can extract features from the standardized state data to obtain state features, as shown in the following formula (11):

[0101] In formula (11), Fs represents the state feature; Es represents the state encoder, which can be a multi-layer perceptron (MLP). This represents standardized state data.

[0102] Then, the visual features, language features, and state features can be fused using the multimodal fusion module to obtain fused features, and action blocks can be generated based on the fused features using the action generation module, as shown in the following formulas (12) and (13):

[0103]

[0104] In formula (12), Indicates fusion characteristics; This represents a multimodal fusion module (such as a fusion module based on cross-attention). In formula (13), This indicates the action generation module; This indicates the fusion feature.

[0105] In some examples, the action generation module can employ action generation methods based on flow matching or diffusion models. In this type of action generation, action generation can be modeled as a continuous transformation process from noise to real action: let the initial noise action be... The final generated action is The preset motion generation model learns to describe the velocity field that changes with time. During reasoning, starting from the time step... Points to time step Each update step can be represented as shown in the following formula (14):

[0106] In formula (14), Indicates the first The intermediate actions corresponding to each time step; Indicates the first The intermediate actions corresponding to each time step; Indicates the time step size between two adjacent time steps; This represents the velocity field learned by the preset action generation model, where Represents the parameters of the model; This represents the time corresponding to the k-th time step; This represents the model input data. That is, formula (14) means that the preset action generation model estimates the action update direction based on the current intermediate action state, time parameters and model input data, and obtains the next intermediate action state along that direction.

[0107] After completing multiple sampling steps, the final action block can be obtained: .

[0108] In some embodiments, step 240 may include: acquiring historical periodic action blocks, inference delay steps, and execution window length; wherein, the inference delay steps are determined based on model inference time, communication time, and a preset control frequency; determining action continuity constraints based on historical periodic action blocks, inference delay steps, and execution window length; the action continuity constraints are used to constrain the differences between the actions of the generated action block within a preset time alignment interval and the corresponding actions in the historical action block, the preset time alignment interval being determined based on the inference delay steps and execution window length; inputting model input data into a preset action generation model, and guiding and correcting the action sampling process based on the action continuity constraints to obtain action blocks.

[0109] Traditional action block strategies involve the model regenerating action blocks periodically. These newly generated action blocks may be discontinuous with action blocks from the previous cycle that were not yet completed. In real-world robot execution, this can manifest as arm tremors, wrist pose jumps, or sudden changes in gripper position. To address this, embodiments of this disclosure introduce historical cycle action blocks, inference delay steps, and execution window length to impose action continuity constraints on newly generated action blocks.

[0110] For example, a historical cycle action block is an action block that has been generated before the current inference cycle. In some examples, a historical cycle action block can be an action block generated in the previous inference cycle. For instance, when the control device sends the model input data for the current inference cycle to the preset action generation model, it can simultaneously send the action block generated in the previous inference cycle to the model. A historical cycle action block can be represented in the form shown in the following formula (15):

[0111] In formula (15), Indicates a historical cycle action block; This indicates the first model action executed in the historical cycle action block; This indicates the second model action executed in the historical cycle action block; This represents the last model action executed in the historical cycle action block; This indicates the time domain length of the action prediction.

[0112] For example, the inference delay steps are used to represent the delay between sending model input data and receiving an action block in control steps. The inference delay steps are determined based on the model inference time, communication time, and preset control frequency, and can be expressed, for example, in the form shown in the following formula (16):

[0113] In formula (16), Indicates the number of inference delay steps; The time required for model inference This is the preset control frequency.

[0114] For example, if the model inference time is 150 milliseconds, the communication time is 50 milliseconds, and the preset control frequency is 50 Hz, then the inference delay steps are... That is, when the preset action generation model returns a new action block, the robot has already executed about 10 control steps according to the historical cycle action blocks.

[0115] For example, the execution window length is the number of actions planned to be actually executed in the newly generated action block, and the execution window length can be less than or equal to the action prediction time domain length H. The new action block to be generated can be represented in the form shown in the following formula (17):

[0116] In formula (17), This represents a newly generated action block; This indicates the first model action in the new action block; This indicates the second model action in the new action block; This indicates the last model action in the new action block.

[0117] For example, due to the existence of inference latency, the first few steps of a new action block correspond in time to actions in historical cycle action blocks that the robot has already executed or is about to execute. Therefore, the inference latency steps can be used to determine the actions. and execution window length Determine the preset time alignment interval And within this interval, the new and old action blocks are constrained to remain continuous. The action continuity constraint can be expressed as the loss function form shown in the following formula (18):

[0118] In formula (18), This represents the loss value corresponding to the action continuity constraint. Indicates the sequence number of the action within the time alignment interval; Indicates the first The time weight of each action; Indicates the first action in the new action block Each model action; This represents the k-th model action in the historical cycle action block. By making... Keeping it as small as possible can constrain new action blocks to a preset time alignment interval. The difference between the actions within the block and the corresponding actions in the historical cycle should be minimized.

[0119] In some examples, time weights The decay method can be set using either linear decay or exponential decay, for example, it can be expressed as shown in the following formula (19):

[0120] In formula (19), Indicates the first The time weight of each action; Indicates the action sequence number; Represents the exponential decay coefficient. A constant greater than 0; Indicates the first The time interval between the execution time of each action and the current execution time. Through the above time weights, actions closer to the current execution time (i.e., the smaller the sequence number k) have, the greater their weight, and the more they need to be consistent with historical periodic action blocks; actions further back have smaller weights, allowing the preset action generation model to adjust according to the latest observation data.

[0121] In some examples, during the motion sampling process based on flow matching or diffusion models, motion continuity constraints can be added as guiding terms to the motion generation process: the preset motion generation model first predicts the velocity field based on the current model input data, then calculates the deviation based on historical periodic motion blocks, and corrects the velocity field (i.e., the motion sampling direction), as shown in the following formula (20):

[0122] In formula (20), This represents the corrected velocity field; This represents the original velocity field predicted by the model; This represents the guiding weight, used to control the degree of influence of action continuity constraints on the generation of new actions; This represents the gradient of the loss value with respect to the action, corresponding to the action continuity constraint. Guiding weights. When the size is too large, the model will rely excessively on historical periodic action blocks, resulting in a slower response to new observation data; guiding weights If the value is too small, the new and old action blocks may not be continuous. Therefore, an upper limit can be set for the guiding weight, as shown in the following formula (21):

[0123] In formula (21), This represents the currently calculated guiding weight; This represents the preset maximum guiding weight. By setting an upper limit on the guiding weight, we can ensure the continuity between old and new action blocks while avoiding the model's over-reliance on historical actions, which would reduce its responsiveness to new observations.

[0124] The dual-arm robot control method based on visual language action model provided in this disclosure introduces historical periodic action blocks, inference delay steps, and execution window length during the action generation process. It applies action continuity constraints to the actions of new action blocks within a preset time alignment interval and performs guidance correction during action sampling. This ensures that the newly generated action blocks and the historical action blocks that have not yet been executed remain continuous within the execution window, avoiding abrupt changes in action after each re-inference. This reduces the probability of robotic arm tremors, wrist posture jumps, and sudden changes in gripper.

[0125] Step 250: According to the preset motion space mapping relationship, perform motion space adaptation processing on each model motion in the motion block to obtain the robot motion corresponding to each model motion.

[0126] In some examples, the model's actions are represented by the standard action space defined during model training, while the space required by the control interface of the real robot is the robot's execution space. There may be inconsistencies between the two in the number of joints, gripper range, base control method, and positive direction of action.

[0127] Therefore, the control device can adapt the actions of each model in the action block into robot actions according to the preset action space mapping relationship. These robot actions include control quantities for the left robotic arm joints, the right robotic arm joints, the left gripper, and the right gripper. If the robot includes a movable base, the robot actions may also include base motion control quantities.

[0128] In some examples, the model action can be represented by formula (22): The adaptation relationship between the model's actions and the robot's actions, as shown in the vector form, can be expressed as shown in formula (23): .

[0129] In formula (22), Indicates model actions; This represents the value of the first dimension in the model's actions; This represents the value of the second dimension in the model's actions; This represents the value of the nth dimension in the model's action. In formula (23), Indicates robot actions; This represents the adaptation function corresponding to the preset action space mapping relationship; This indicates the model's actions.

[0130] For example, for a dual-arm robot, the motion layout of the robot's movements can be represented as shown in the following formula (24):

[0131] In formula (24), Indicates robot actions; These represent the first to the second joint control values ​​of the left robotic arm. The control quantities for each joint dimension are represented by the superscript indicating the joint number. This refers to the number of joints in a single arm. Indicates the control amount of the left gripper; These represent the first to the second joint control values ​​of the right robotic arm. Control quantity per joint dimension; Indicates the control amount of the right gripper; Indicates the linear velocity control value of the base; This represents the angular velocity control value of the base. If the robot does not include a moving base, the robot's motion may not include the linear velocity control value of the base. and base angular velocity control amount .

[0132] In some embodiments, step 250 may include: determining the dimensional correspondence between model actions and robot actions; correcting the direction of the action dimensions corresponding to the preset action direction mask in each model action according to the preset action direction mask to obtain the direction-corrected action; inverting the standardized gripper control quantities in the direction-corrected action to the left gripper control quantity and the right gripper control quantity respectively according to the preset gripper mapping relationship to obtain the preliminary adapted action; and performing dimensional sorting and / or dimensional truncation processing on the preliminary adapted action according to the dimensional correspondence to obtain the robot action.

[0133] For example, the dimensional correspondence is used to indicate which action dimension in the model's motion corresponds to which action dimension in the robot's motion. For instance, dimensions 1 through 6 in the model's motion correspond to joints 1 through 6 of the left robotic arm, dimension 7 corresponds to the left gripper, dimensions 8 through 13 correspond to joints 1 through 6 of the right robotic arm, dimension 14 corresponds to the right gripper, dimension 15 corresponds to the linear velocity of the base, and dimension 16 corresponds to the angular velocity of the base. The dimensional correspondence can be pre-configured based on the motion space definition during model training and the control interface definition of the real robot.

[0134] For example, the preset motion direction mask corresponds to the preset direction mask in step 220, and is used to correct the direction of motion dimensions in the model motion that are inconsistent with the actual robot direction definition. This direction correction process can be expressed in the form of the following formula (25):

[0135] In formula (25), Indicates a direction correction action; Indicates the preset action direction mask; These represent the first to the second preset motion direction masks. The possible values ​​of each element; Indicates the first in the preset action direction mask The possible values ​​of each element, The value can be 1 or -1; This indicates the number of dimensions for the model's actions. For action dimensions with consistent directional definitions, the corresponding mask element is set to 1, and the value of that dimension remains unchanged. For action dimensions with opposite directional definitions, the corresponding mask element is set to -1, and the value of that dimension is inverted. Through this directional correction, situations such as the left and right robotic arms moving in opposite directions or the base turning in opposite directions can be avoided.

[0136] For example, the gripper dimension in the direction correction action is a standardized gripper control quantity, that is, the gripper opening and closing ratio represented by a preset gripper normalization interval (e.g., [0,1]). The control device can denormalize the standardized gripper control quantity into left gripper control quantity and right gripper control quantity respectively according to the preset gripper mapping relationship to obtain the preliminary adaptation action. The denormalization process can be expressed as shown in the following formula (26):

[0137] In formula (26), This represents the gripper control quantity obtained after inverse normalization, i.e., the actual gripper angle, stroke, or the opening / closing amount required by the controller. This represents the standardized gripper control quantity, that is, the gripper opening and closing ratio within a preset gripper normalization range, for example... For example, when the standardized gripper control value is 0.5 and the gripper stroke range is 0 mm to 80 mm, the gripper control value obtained by inverse normalization is: Millimeters. The left and right grippers can each correspond to their respective preset gripper value ranges, thus obtaining the control values ​​for the left and right grippers respectively.

[0138] For example, after obtaining the initial adaptation action, the control device can perform dimension sorting and / or dimension truncation processing on the initial adaptation action according to the dimension correspondence to obtain the robot action. Dimension sorting is used to adjust the arrangement order of each dimension in the initial adaptation action to the arrangement order required by the robot control interface; dimension truncation is used to truncate the effective dimensions when the number of dimensions of the initial adaptation action is greater than the number of dimensions of the robot action. Dimension truncation can be expressed in the form shown in the following formula (27):

[0139] In formula (27), This represents the executable action obtained after dimension truncation; Indicates the action performed before the cutoff; Indicates the number of valid dimensions that need to be retained; Indicates taking The 0th to the (m-1)th dimensions.

[0140] In some examples, when the number of dimensions in the model's actions is less than the number of dimensions in the robot's actions, default neutral control variables can be used to fill in the missing action dimensions in the model's actions, so that the number of action dimensions after completion matches the number of robot action dimensions. The default neutral control variable is the control variable that keeps the corresponding actuator in its current state. For example, for a missing joint dimension, the default neutral control variable could be the current joint angle; for a missing gripper dimension, it could be the current opening / closing amount of the gripper; and for a missing base dimension, it could be zero velocity. By filling in the missing dimensions with default neutral control variables, the model's actions can be adapted to a robot control interface with a larger number of dimensions, while avoiding control anomalies caused by missing dimensions.

[0141] The dual-arm robot control method based on visual language action model provided in this disclosure converts model actions in the model training space into robot actions in the real robot execution space through adaptation processing such as dimensional correspondence, preset action direction mask, preset gripper mapping relationship, and dimensional sorting, dimensional truncation, and dimensional padding. This solves the problem of inconsistency between the training space, model space, and real robot execution space, reduces the adaptation cost when migrating the model to different robot platforms, and reduces deployment problems such as opposite action directions of the left and right robotic arms, incorrect gripper opening and closing ratios, and abnormal base speed.

[0142] Step 260: Send robot actions to the robot according to the preset control frequency, and obtain updated observation data when the robot performs robot actions.

[0143] In some examples, the control device can send robot actions to the robot at a preset control frequency to control the robot's execution; and during the robot's execution of the actions, it can acquire updated observation data collected by the robot. This updated observation data includes updated multi-view image data and updated ontological state data, which are used to construct the model input data for the next inference cycle.

[0144] For example, the preset control frequency is the frequency at which the robot's underlying system executes control commands; for example, the preset control frequency could be 50 Hz. The preset inference frequency of the preset motion generation model is usually lower than the preset control frequency; for example, the preset inference frequency could be 5 Hz. The relationship between the two can be expressed as shown in the following formula (28):

[0145] In formula (28), Indicates the preset control frequency; This represents the preset inference frequency, which is the preset frequency at which the action generation model generates action blocks. Since the preset inference frequency is lower than the preset control frequency, each time the model generates an action block, the multiple actions within that action block can be executed sequentially by the robot over multiple control cycles.

[0146] In some embodiments, when the preset inference frequency of the preset action generation model is less than the preset control frequency, step 260, which involves sending robot actions to the robot according to the preset control frequency, may include: writing robot actions into an action buffer; when there are cached robot actions in the action buffer that have not yet been executed, performing weighted fusion of the cached robot actions and the actions of the robot actions at corresponding times during the overlapping execution period to obtain a fused action; when there are no cached robot actions in the action buffer that have not yet been executed, determining the robot action as a fused action; when the action difference between two adjacent fused actions is greater than a preset difference threshold, inserting an intermediate action between the two adjacent fused actions according to a preset interpolation step number, and writing the intermediate action into the action buffer; reading the action to be sent from the action buffer according to the preset control frequency and action execution order, and sending the action to be sent to the robot; wherein the action to be sent includes at least one of a fused action and an intermediate action.

[0147] For example, the control device can adopt an asynchronous structure: the inference thread requests new action blocks from the preset action generation model at a preset inference frequency, and the control thread reads actions from the action buffer and issues them to the robot at a preset control frequency. The action buffer is used to cache robot actions to be executed, and can be, for example, a queue arranged in the order of action execution.

[0148] For example, when the robot action corresponding to the new action block in the current inference cycle arrives, if there are still cached robot actions in the action buffer that have not yet been executed (i.e., actions that have not yet been completed in the action block of the previous inference cycle), it indicates that there is an overlapping execution period between the cached robot actions and the newly arrived robot actions. The control device can perform weighted fusion on the actions of the cached robot actions and the newly arrived robot actions at corresponding times within the overlapping execution period to obtain the fused action. The weighted fusion process can be expressed as shown in the following formula (29):

[0149] In formula (29), Indicates the first in the fusion action The action at that moment; Indicates the first The fusion weights corresponding to each time point The value can be between 0 and 1; This indicates that the first robot action has been cached. The action at that moment; This indicates the first action of the newly arrived robot. Actions at any given moment. For example, weight fusion. It can be done at any time The linear decrease in the value of the action buffer allows for a smooth transition between old and new actions, as the earlier phase of the overlapping execution period retains more of the old actions and the later phase gradually transitions to the new actions. If there are no cached robot actions in the action buffer that have not yet been executed, it means that the actions of the previous cycle have been completed, and the newly arrived robot actions can be directly identified as fused actions.

[0150] For example, a preset difference threshold is used to measure whether the change in motion between two adjacent fused actions is too large. The motion difference can be the maximum value of the difference between the corresponding dimensions in the two adjacent fused actions or the L2 norm. When the motion difference between two adjacent fused actions is greater than the preset difference threshold, it indicates that the abrupt change between the two actions is large, and direct execution may cause the robotic arm to shake. At this time, the control device can insert an intermediate action between the two adjacent fused actions according to the preset interpolation steps and write the intermediate action into the action buffer. The interpolation process can be expressed as shown in the following formula (30):

[0151] In formula (30), Indicates the inserted first An intermediate action; This indicates the earlier fusion action among two adjacent fusion actions, i.e., the starting action of the interpolation; This indicates the later fusion action among two adjacent fusion actions, i.e., the target action for interpolation; This indicates the preset number of interpolation steps, i.e., the number of intermediate actions to be inserted; Indicates the sequence number of the intermediate action. The value ranges from 1 to Integers. By inserting between two adjacent fusion actions. A linear transition in intermediate motions can make the changes in robot movements smoother.

[0152] For example, the control device can read actions to be sent from the action buffer according to a preset control frequency and action execution order, and send the actions to be sent to the robot. The actions to be sent include at least one of fused actions and intermediate actions. For instance, robot actions, after weighted fusion and interpolation processing, are sequentially written into the action buffer according to the execution order. Each time the control thread reaches a control cycle, it reads one action to be sent from the head of the action buffer and sends it to the robot controller, which then distributes it to the controllers of the left and right robotic arms and the mobile base. Since the joint control quantities of the left and right robotic arms, the left and right grippers, and the base motion control quantities all originate from the same robot action, synchronous control of the two arms, grippers, and mobile base can be achieved.

[0153] In some examples, while the robot is performing its actions, the control device can continuously collect and update observation data, including updating multi-view image data and updating the robot's state data. For example, when the target object is slightly moved, the gripper's approach angle deviates, or the wrist camera observes new occlusions, the updated observation data can reflect these changes. The control device can then construct the model input data for the next inference cycle based on the updated observation data, enabling the model to generate corresponding adjustment actions in the next inference cycle, thereby improving the task completion rate.

[0154] In some examples, the closed-loop control process described above can be expressed as shown in the following formula (31):

[0155] In formula (31), express Observational data at any given time; This represents the reasoning process of the preset action generation model; Indicates an action block; This indicates motion space adaptation processing performed according to a preset motion space mapping relationship; This indicates that the robot is performing robot actions; This indicates the updated observation data for the next moment; the arrow indicates the direction of data flow. In other words: retrieve the current observation data. Preset action generation model generates action blocks. Motion space adaptation converts motion blocks into robot actions that the robot can execute; the robot controller executes the robot actions; new and updated observation data are collected. ; Reasoning and control to enter the next reasoning cycle.

[0156] The dual-arm robot control method based on a visual language action model provided in this disclosure performs weighted fusion of cached robot actions and newly arrived robot actions during overlapping execution periods through an action buffer. When the action difference between adjacent fused actions exceeds a preset difference threshold, an intermediate action is inserted, and the action to be sent is sent to the robot according to a preset control frequency. This alleviates the problem of discontinuous actions caused by the inconsistency between the model inference frequency, communication delay, and robot control frequency, reduces robotic arm jitter and end-effector posture abrupt changes, and enables the dual arms and grippers to maintain a more stable motion trajectory during continuous grasping and placement. At the same time, by acquiring updated observation data to construct model input data for the next inference cycle, closed-loop control is achieved, enabling the robot to continuously correct its actions according to environmental changes.

[0157] The following example illustrates the dual-arm robot control method based on a visual language action model provided in this disclosure.

[0158] For example, suppose a user inputs the natural language task command "pick up the banana" into the robot, and the target object is a banana placed on a platform. Assume a preset control frequency... 50 Hz, preset inference frequency The frequency is 5 Hz, the motion prediction time domain length H is 30, the execution window length E is 20, the preset image size is 224 pixels × 224 pixels, the model inference time is 150 milliseconds, the communication time is 50 milliseconds, the left and right robotic arms each include 6 joints, the linear velocity of the moving base ranges from -0.5 m / s to 0.5 m / s, and the opening and closing stroke range of the left and right grippers is 0 mm to 80 mm.

[0159] In one inference cycle, the control device first acquires the global image captured by the high-position camera, the left wrist image captured by the left wrist camera, the right wrist image captured by the right wrist camera, the six joint angles of the left robotic arm, the six joint angles of the right robotic arm, the opening and closing stroke of the left gripper (e.g., currently 60 mm), the opening and closing stroke of the right gripper (e.g., currently 60 mm), the linear velocity of the moving base (e.g., currently 0 m / s) and the angular velocity of the base (e.g., currently 0 radians / s), and the natural language task instruction "pick up the banana".

[0160] Next, the control device converts the three images from BGR to RGB color channel format, and scales and fills them to a standardized 224 pixel × 224 pixel image. The joint angles of the left and right robotic arms are normalized to the [-1, 1] interval according to a preset state value range. Joint dimensions with opposite orientation definitions are inverted and corrected according to a preset orientation mask. For example, if the orientation definition of the second joint of the right robotic arm 114 is opposite to the training data, the element corresponding to that joint in the preset orientation mask is set to -1. The opening and closing strokes of the left and right grippers are mapped to the preset gripper normalization interval [0, 1]. For example, the standardized gripper state corresponding to 60 mm is (60-0) / (80-0) = 0.75. The base linear velocity and base angular velocity are normalized to the [-1, 1] interval. The standardized joint states, gripper states, and base states are then concatenated into standardized state data.

[0161] Then, the control device maps the standardized global image to the field. Map the standardized left wrist image to the field Map the standardized right wrist image to a field Generate an image mask where all mask elements are 1, and map the normalized state data to the field. Map "Pick up a banana" to a field The input data for the model is obtained by combining these data.

[0162] Then, the control device sends the model input data, along with the historical action blocks generated in the previous inference cycle, the inference delay steps, and the execution window length, to the preset action generation model. The inference delay steps are among the data points. The execution window length is E=20, and the preset time alignment interval is [10,20). The preset action generation model obtains fusion features through visual encoding, language encoding, state encoding, and multimodal fusion. During the action sampling process, a guided correction based on action continuity constraints is applied to the velocity field, so that the difference between the 10th to 19th model actions in the new action block and the corresponding actions in the historical periodic action blocks is constrained, and finally an action block containing 30 model actions is generated.

[0163] Next, the control device performs motion space adaptation processing on the motion of each model in the motion block: according to the preset motion direction mask, the dimensions defined in the opposite direction of the second joint of the right robotic arm are corrected; the standardized gripper control quantities are denormalized into left gripper control quantities and right gripper control quantities respectively. For example, the standardized gripper control quantity 0.3 is denormalized into a gripper stroke control quantity of 0.3×(80-0)+0=24 mm; then, according to the dimensional correspondence, each dimension is sorted into the motion layout of the control quantities of the 6 joints of the left robotic arm, the control quantity of the left gripper, the control quantity of the 6 joints of the right robotic arm, the control quantity of the right gripper, the linear velocity control quantity of the base, and the angular velocity control quantity of the base, so as to obtain the robot motion corresponding to each model motion.

[0164] Finally, the control device writes the robot's actions into the action buffer, performs weighted fusion of the cached robot actions within the overlapping execution period and the newly arrived robot actions, and inserts an intermediate action when the action difference between adjacent fused actions exceeds a preset difference threshold. The control thread reads the actions to be sent from the action buffer at a preset control frequency of 50 Hz and sends them to the robot. The robot's left and right robotic arms move according to the received joint control values, the left and right grippers open and close according to the received gripper control values, and the moving base moves according to the received base linear velocity and base angular velocity control values, causing the robot to move to the vicinity of the placement platform and pick up the banana. During execution, the control device continuously acquires and updates observation data to construct the model input data for the next inference cycle, enters the next round of inference and control, until the task of "picking up the banana" is completed.

[0165] Figure 3 This is a schematic diagram of the structure of a dual-arm robot control device based on a visual language action model, provided as an embodiment of this disclosure. Figure 3 As shown, the dual-arm robot control device 300 based on a visual language action model may include an acquisition module 310, a standardization module 320, a construction module 330, a model processing module 340, an action generation module 350, and a control module 360. Wherein: The acquisition module 310 is configured to acquire multi-view image data of the robot, robot body state data, and natural language task instructions; wherein, the body state data includes the joint state of the left robotic arm, the joint state of the right robotic arm, the state of the left gripper, and the state of the right gripper.

[0166] The standardization module 320 is configured to perform image standardization processing on multi-view image data to obtain standardized multi-view image data, and to perform state standardization processing on ontology state data to obtain standardized state data.

[0167] Module 330 is configured to construct model input data based on standardized multi-view image data, standardized state data, and natural language task instructions.

[0168] The model processing module 340 is configured to process the model input data by generating a model through preset actions to obtain an action block; wherein the action block includes multiple model actions arranged in the execution order.

[0169] The motion generation module 350 is configured to perform motion space adaptation processing on each model motion in the motion block according to a preset motion space mapping relationship, so as to obtain the robot motion corresponding to each model motion; wherein, the robot motion includes the control quantities of the left robotic arm joint, the control quantities of the right robotic arm joint, the control quantities of the left gripper and the control quantities of the right gripper.

[0170] The control module 360 ​​is configured to send robot actions to the robot at a preset control frequency and acquire updated observation data when the robot performs robot actions; wherein, the updated observation data includes updated multi-view image data and updated body state data, and the updated observation data is used to construct the model input data for the next inference cycle.

[0171] In some embodiments, the robot further includes a mobile base, the body state data includes the base motion state data of the mobile base, and the robot actions include base motion control quantities for controlling the mobile base; wherein, the base motion state data includes the base linear velocity and the base angular velocity, and the base motion control quantities include the base linear velocity control quantity and the base angular velocity control quantity.

[0172] In some embodiments, the standardization module 320 is configured to: convert the color channel format of each image in the multi-view image data, and scale and fill it according to a preset image size to obtain standardized multi-view image data; normalize the joint states of the left and right robotic arms according to a preset state value range to obtain normalized joint states; correct the orientation of each joint dimension in the normalized joint states according to a preset orientation mask to obtain standardized joint states; map the left and right gripper states to a preset gripper normalization interval according to a preset gripper value range to obtain standardized gripper states; and determine standardized state data based on the standardized joint states and standardized gripper states.

[0173] In some embodiments, the construction module 330 is configured to: map standardized multi-view image data to image field data, standardized state data to state field data, and natural language task instructions to language field data according to preset model input fields; generate an image mask for marking the valid states of each standardized image in the standardized multi-view image data; and combine the image field data, state field data, language field data, and image mask into model input data.

[0174] In some embodiments, the motion generation module 350 is configured to: determine the dimensional correspondence between model motions and robot motions; perform directional correction on the motion dimensions corresponding to the preset motion direction mask in each model motion according to the preset motion direction mask to obtain directional correction motions; denormalize the standardized gripper control quantities in the directional correction motions into left gripper control quantities and right gripper control quantities respectively according to the preset gripper mapping relationship to obtain preliminary adapted motions; and perform dimensional sorting and / or dimensional truncation processing on the preliminary adapted motions according to the dimensional correspondence to obtain robot motions.

[0175] In some embodiments, the model processing module 340 is configured to: acquire historical periodic action blocks, inference delay steps, and execution window length; wherein, the inference delay steps are determined based on model inference time, communication time, and a preset control frequency; determine action continuity constraints based on historical periodic action blocks, inference delay steps, and execution window length; the action continuity constraints are used to constrain the differences between the actions of the generated action blocks within a preset time alignment interval and the corresponding actions in historical action blocks, the preset time alignment interval being determined based on the inference delay steps and execution window length; input model input data into a preset action generation model, and guide and correct the action sampling process based on the action continuity constraints to obtain action blocks.

[0176] In some embodiments, when the preset inference frequency of the preset action generation model is less than the preset control frequency, the control module 360 ​​is configured to: write robot actions into an action buffer; when there are cached robot actions in the action buffer that have not yet been executed, perform weighted fusion of the cached robot actions and the actions of the robot actions at corresponding times during the overlapping execution period to obtain a fused action; when there are no cached robot actions in the action buffer that have not yet been executed, determine the robot action as a fused action; when the action difference between two adjacent fused actions is greater than a preset difference threshold, insert an intermediate action between the two adjacent fused actions according to a preset interpolation step number, and write the intermediate action into the action buffer; read the action to be sent from the action buffer according to the preset control frequency and action execution order, and send the action to be sent to the robot; wherein, the action to be sent includes at least one of a fused action and an intermediate action.

[0177] It should be noted that the dual-arm robot control device based on visual language action model provided in this disclosure is used to execute the dual-arm robot control method based on visual language action model provided above. Therefore, the beneficial technical effects it can achieve can be referred to the corresponding beneficial technical effects in the above method embodiments, and will not be repeated here.

[0178] Exemplary electronic devices Figure 4A structural diagram of an electronic device provided in this disclosure includes at least one processor 410 and a memory 420.

[0179] The processor 410 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0180] The memory 420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 410 may execute one or more computer program instructions to implement the visual language action model-based dual-arm robot control method and / or other desired functions of the various embodiments of this disclosure described above.

[0181] In one example, the electronic device 400 may also include an input device 430 and an output device 440, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0182] The input device 430 may also include, for example, a keyboard, a mouse, etc.

[0183] The output device 440 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0184] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device 400 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 400 may include any other suitable components depending on the specific application.

[0185] Exemplary computer program products and computer-readable storage media In addition to the methods and devices described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the visual language action model-based dual-arm robot control method of various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0186] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0187] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the visual language action model-based dual-arm robot control method of various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0188] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0189] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0190] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A control method for a dual-arm robot based on a visual language action model, characterized in that, The method includes: The robot acquires multi-view image data, robot body state data, and natural language task instructions; wherein, the body state data includes the joint states of the left robotic arm, the joint states of the right robotic arm, the state of the left gripper, and the state of the right gripper. The multi-view image data is subjected to image standardization processing to obtain standardized multi-view image data, and the ontology state data is subjected to state standardization processing to obtain standardized state data. Based on the standardized multi-view image data, the standardized state data, and the natural language task instructions, model input data is constructed; The input data of the model is processed by a preset action generation model to obtain action blocks; wherein, the preset action generation model is a visual language action model, and the action block includes multiple model actions arranged in execution order; According to the preset motion space mapping relationship, the motion space adaptation processing is performed on each of the model actions in the motion block to obtain the robot action corresponding to each model action; wherein, the robot action includes the left robotic arm joint control amount, the right robotic arm joint control amount, the left gripper control amount and the right gripper control amount. The robot's actions are sent to the robot at a preset control frequency, and updated observation data is acquired when the robot performs the actions. The updated observation data includes updated multi-view image data and updated body state data, and is used to construct the model input data for the next inference cycle.

2. The method according to claim 1, characterized in that, The robot also includes a mobile base, the body state data also includes the base motion state data of the mobile base, and the robot actions also include base motion control quantities for controlling the mobile base. The base motion state data includes the base linear velocity and the base angular velocity, and the base motion control quantities include the base linear velocity control quantity and the base angular velocity control quantity.

3. The method according to claim 1, characterized in that, The step of performing image standardization processing on the multi-view image data to obtain standardized multi-view image data, and performing state standardization processing on the ontology state data to obtain standardized state data, includes: The color channel format of each image in the multi-view image data is converted, and the images are scaled and filled according to the preset image size to obtain the standardized multi-view image data. According to the preset state value range, the joint states of the left robotic arm and the right robotic arm are normalized to obtain normalized joint states. According to the preset orientation mask, the orientation of each joint dimension in the normalized joint state is corrected to obtain the standardized joint state. According to the preset gripper value range, the left gripper state and the right gripper state are mapped to the preset gripper normalization interval respectively to obtain the standardized gripper state; The standardized state data is determined based on the standardized joint state and the standardized gripper state.

4. The method according to claim 1, characterized in that, The process of constructing model input data based on the standardized multi-view image data, the standardized state data, and the natural language task instructions includes: According to the preset model input fields, the standardized multi-view image data is mapped to image field data, the standardized state data is mapped to state field data, and the natural language task instructions are mapped to language field data; Generate an image mask to mark the effective state of each standardized image in the standardized multi-view image data; The image field data, the state field data, the language field data, and the image mask are combined to form the model input data.

5. The method according to claim 1, characterized in that, The step of performing motion space adaptation processing on each model action in the action block according to a preset motion space mapping relationship to obtain the robot action corresponding to each model action includes: Determine the dimensional correspondence between the model actions and the robot actions; According to the preset action direction mask, the action dimension corresponding to the preset action direction mask in each of the model actions is corrected in direction to obtain the direction-corrected action; According to the preset gripper mapping relationship, the standardized gripper control quantities in the direction correction action are inversely normalized to the left gripper control quantity and the right gripper control quantity, respectively, to obtain the preliminary adaptation action; Based on the dimensional correspondence, the preliminary adaptation actions are sorted and / or truncated according to dimensions to obtain the robot actions.

6. The method according to claim 1, characterized in that, The process of processing the input data of the model through a preset action generation model to obtain action blocks includes: Obtain historical cycle action blocks, inference delay steps, and execution window length; wherein, the inference delay steps are determined based on model inference time, communication time, and the preset control frequency; Action continuity constraints are determined based on the historical periodic action blocks, the inference delay steps, and the execution window length. The action continuity constraints are used to constrain the differences between the actions of the generated action blocks within a preset time alignment interval and the corresponding actions in the historical action blocks. The preset time alignment interval is determined based on the inference delay steps and the execution window length. The model input data is input into the preset action generation model, and the action sampling process is guided and corrected based on the action continuity constraint to obtain the action block.

7. The method according to claim 1, characterized in that, When the preset inference frequency of the preset action generation model is less than the preset control frequency, sending the robot action to the robot according to the preset control frequency includes: Write the robot's actions into the action buffer; When there are cached robot actions that have not yet been executed in the action buffer, the cached robot actions and the corresponding actions of the robot actions at the same time during the overlapping execution period are weighted and fused to obtain the fused action; If there is no cached robot action that has not yet been executed in the action buffer, the robot action is determined as the fused action; When the difference between two adjacent fusion actions is greater than a preset difference threshold, an intermediate action is inserted between the two adjacent fusion actions according to a preset number of interpolation steps, and the intermediate action is written into the action buffer. The robot reads the actions to be sent from the action buffer according to the preset control frequency and action execution order, and sends the actions to be sent to the robot; wherein, the actions to be sent include at least one of the fused actions and the intermediate actions.

8. A dual-arm robot control device based on a visual language action model, characterized in that, include: The acquisition module is configured to acquire multi-view image data of the robot, the robot's body state data, and natural language task instructions; wherein, the body state data includes the joint states of the left robotic arm, the joint states of the right robotic arm, the state of the left gripper, and the state of the right gripper. The standardization module is configured to perform image standardization processing on the multi-view image data to obtain standardized multi-view image data, and to perform state standardization processing on the ontology state data to obtain standardized state data. The construction module is configured to construct model input data based on the standardized multi-view image data, the standardized state data, and the natural language task instructions; The model processing module is configured to process the model input data by generating a model through preset actions to obtain an action block; wherein the action block includes multiple model actions arranged in execution order; The motion generation module is configured to perform motion space adaptation processing on each of the model motions in the motion block according to a preset motion space mapping relationship, so as to obtain the robot motion corresponding to each of the model motions; wherein, the robot motion includes the left robotic arm joint control amount, the right robotic arm joint control amount, the left gripper control amount, and the right gripper control amount. The control module is configured to send the robot actions to the robot at a preset control frequency and acquire updated observation data when the robot performs the robot actions; wherein, the preset action generation model is a visual language action model, and the updated observation data includes updated multi-view image data and updated ontology state data, and the updated observation data is used to construct the model input data for the next inference cycle.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the dual-arm robot control method based on a visual language action model as described in any one of claims 1-7.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the dual-arm robot control method based on the visual language action model as described in any one of claims 1-7.