A dual-arm robot task assembly method based on a behavior tree and a VLA

Abstract

The application discloses a kind of based on behavior tree and VLA's dual-arm robot task assembly method, specifically includes: when task starts, behavior tree is executed from root node, signal is sent to child node by fixed frequency;Proposition received by parsing perception module is extracted target object and its spatial relationship by conditional node, and three-dimensional information and attribute are obtained using visual recognition algorithm, to judge whether meet the requirement;Action node generates initial path according to motion planning module, trajectory optimization is carried out according to the dynamics constraint of manipulator, and deviation and disturbance in motion process are monitored in real time, timely adjustment is ensured to complete task, wherein user instruction and scene image information are integrated, joint angle and end effector instruction are generated by means of pre-trained multi-modal VLA model, guide dual-arm efficient cooperation, until successfully complete assembly operation.

Description

Technical Field

[0001] This invention relates to the field of dual-arm robot assembly, and mainly to a dual-arm robot assembly method based on behavior tree and VLA. Background Technology

[0002] VLA (Vision-Language-Action) was first formally proposed in July 2023 when Google DeepMind released the RT-2 model, sparking widespread academic attention to the integration of multimodal perception and robot motion. Subsequently, Stanford University and other institutions released the OpenVLA open-source framework in 2024, further promoting industry enthusiasm for VLA research and its diversified development.

[0003] VLA unifies the three modalities of vision, language, and action into a single semantic space, enabling robots to directly map image observations and text commands to physical actions in an end-to-end manner, significantly enhancing contextual understanding and generalization capabilities.

[0004] Traditional robotic systems typically employ modular designs, such as training perception, planning, and control modules separately. This approach is susceptible to errors in individual modules and struggles to generalize across tasks. In contrast, VLA (Virtual Language Algorithm) reduces the cumulative errors propagated between modules and, thanks to the rich semantic understanding provided by pre-trained large-scale visual language models, enables robots to perform zero-shot or few-shot executions even in unseen scenarios. Furthermore, VLA models, by extracting knowledge from massive amounts of internet data, acquire powerful semantic reasoning capabilities, thus supporting the understanding and execution of open-domain instructions, offering greater flexibility than traditional robots that can only execute predefined actions.

[0005] However, as a black-box model, VLA's core drawback lies in the lack of transparency in its decision-making process. Due to the extreme complexity of its parameters and internal mechanisms, users and developers find it difficult to understand how the model arrives at specific conclusions based on the input, leading to serious reliability issues. Specifically, this manifests as follows: (1) Difficult to debug: When the model produces erroneous or biased output, it is difficult to trace the root cause of the error and make targeted fixes.

[0006] (2) Safety: The unpredictability and uninterpretability of the output of robots when performing tasks such as assembly make them difficult to trust, which ultimately limits their practical application and deployment in such tasks.

[0007] Behavior trees are an action-switching model that was initially used to design the behavior of characters in video games. Their advantages can be summarized as follows: (1) Interpretability. The decision-making logic is intuitively displayed through a hierarchical tree structure. The semantics of its node types (sequence, selection, parallel, etc.) are clear, and the hierarchy is well-defined, making the robot's decision-making process resemble a flowchart. Figure 1 The code is readable. During execution, the behavior path can be tracked in real time through node status (running / success / failure), making it easy for developers to quickly locate decision motivations and fault points, significantly improving the transparency of robot behavior and debugging efficiency.

[0008] (2) Safety. Through a clear tree-like hierarchy and explicit node return status (success / failure), behavior trees ensure that when a child node fails, the parent node (such as sequence or selection) will respond according to predetermined logic, such as executing a backup plan or safety behavior. This determinism prevents the robot from falling into an unknown state, while the modular design facilitates the isolation and repair of specific faults, thereby ensuring the reliability and controllability of the system behavior.

[0009] However, behavior trees have two core drawbacks when applied to the fine manipulation of robots. First, fine manipulation often requires rapid and smooth switching and fusion of robotic arm joint movements, while the discrete node transition mechanism of behavior trees struggles to smoothly achieve such continuous state adjustments. Second, it has weak modeling capabilities for underlying continuous dynamics, relying heavily on manual debugging to embed complex physical constraints into the node logic, resulting in poor robustness of the system when facing uncertainties. Summary of the Invention

[0010] To address the above shortcomings, this invention combines behavior trees with VLA (Virtual Automation). Behavior trees are used to manage the overall assembly task of a dual-arm robot. For known, repetitive tasks, motion planning algorithms are employed to improve the reliability of dual-arm operations. For intricate assembly processes, VLA is used to achieve generalized assembly operations and improve the success rate. Simultaneously, the assembly process is monitored in real-time using behavior trees. When the robot is in an unsafe state, it switches to motion planning mode to ensure the safety of the task. According to a first aspect of this invention, a dual-arm robot task assembly method based on behavior trees and VLA is proposed, specifically including: When a task starts, the behavior tree is executed starting from the root node, sending signals to child nodes at a fixed frequency. When a leaf node receives a signal: the condition node returns Success or Failure based on the received proposition, and the action node returns Success, Failure, or Running based on the operation execution result; the intermediate nodes perform logical judgments based on the status returned by the child nodes and pass the status upwards. The system uses a perception module to receive proposition requests from condition nodes in the behavior tree, parses the propositions to extract target objects and spatial relationships, calls a visual recognition algorithm to obtain the three-dimensional information and attributes of the target objects, calculates the relative relationships between objects, matches the proposition requirements, and returns Success if the requirements are met, otherwise it returns Failure. Using the motion planning module, if the leaf node is an action node, the initial path of the two arms is generated according to the target pose. The time parameterization and trajectory smoothing are performed according to the dynamic constraints of the robotic arm. The two arms are then synchronously controlled to move along the planned trajectory, and the trajectory deviation and external interference are monitored in real time. If a deviation is found, the trajectory is adjusted or the speed is compensated until the target action is completed. The system receives scene image information and user input commands, inputs pre-trained multimodal VLA model to generate joint angles of the two arms and opening and closing commands of the end effector, and sends them to the dual-arm controller for execution; during the execution of the action, the VLA module iteratively generates subsequent action commands based on the current scene information until the assembly operation is completed.

[0011] Furthermore, the behavior tree includes a root node, at least one intermediate node, and at least one leaf node, wherein the intermediate node includes a sequence node and a backtrack node, and the leaf node includes a condition node and an action node.

[0012] Furthermore, the VLA module includes the following training phases: In the first stage, multi-robot, multi-task demonstration data is used to pre-train the multimodal Transformer model to learn the alignment relationship between visual features, language commands and action representations in the same high-dimensional space. In the second stage, the basic VLA model is deployed to a real robot or high-fidelity simulation environment. The model performance is optimized through instruction fine-tuning and reinforcement learning, and supervised fine-tuning is performed using successful data. In the third stage, a mature VLA model is obtained, which generates the joint angles of the two arms and the opening and closing commands of the end effector based on the scene information and instructions, and iteratively generates actions based on the execution feedback until the task is completed.

[0013] Furthermore, the VLA module iteratively generates subsequent action instructions based on the current scene information until the assembly operation is completed. This includes generating a dual-arm synchronous trajectory and monitoring trajectory errors and speed deviations in real time during execution. If the deviation deviates from the expected task, the local trajectory is replanned or speed compensation is performed.

[0014] Furthermore, the execution logic for task assembly specifically includes: The behavior tree nodes are visited sequentially. First, the task completion judgment condition node is visited. If the task is not completed, the safety condition judgment and component distance judgment condition nodes are visited. The corresponding action node is selected and executed according to the output result of the perception module, including stopping operation, returning to the initial position, approaching the component, or assembly operation. During the assembly operation, the status of the two arms and the VLA output result are monitored in real time. If the safety condition is not met, the stop operation step is triggered. When all assembly operations are completed and the task completion judgment returns "Success", the task execution ends.

[0015] Furthermore, the status of child nodes is continuously polled during the assembly task execution to achieve dynamic task adjustment and safety control.

[0016] Furthermore, the action of approaching the component follows a preset fixed trajectory.

[0017] Furthermore, the behavior tree is characterized by sending Tick signals at a fixed frequency and determining the execution path in real time based on the return status of the child nodes from bottom to top.

[0018] According to a second aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0019] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: This invention clarifies the task's operational flow through behavior trees, constraining the boundaries of robot motion planning algorithms and VLA usage.

[0020] 1) When the task begins, the motion planning algorithm is used to move both arms to the vicinity of the components quickly and stably, avoiding the low efficiency and slow decision-making problems that may occur when using VLA throughout the large-scale movement.

[0021] 2) Once the arms reach the vicinity of the components, the VLA is responsible for fine assembly, leveraging its strengths in handling minor uncertainties, visual guidance, and dexterous operation.

[0022] 3) During assembly, the robot's operating status is monitored in real time using a behavior tree. If unsafe actions are encountered, such as excessive contact torque, the VLA's control over the arms is stopped through the node's interruption mechanism to ensure task safety.

[0023] 4) After VLA stops, use the motion planning algorithm again to quickly and stably move both arms to the vicinity of the components and restart the assembly task.

[0024] 5) The overall task is represented by a tree structure of behavior trees, which is intuitive and highly interpretable, making it convenient for engineers to debug.

[0025] 6) In addition, behavior trees have modular characteristics, and the nodes do not interfere with each other, which makes it convenient for engineers to maintain and expand them later.

[0026] Overall, this invention significantly improves the efficiency, reliability, and security of tasks while ensuring a high success rate. Attached Figure Description

[0027] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0028] Figure 1 A flowchart of a dual-arm robot task assembly method based on behavior tree and VLA according to an embodiment of the present invention is shown.

[0029] Figure 2 A schematic diagram of the workflow of a sensing module according to an embodiment of the present invention is shown.

[0030] Figure 3 A schematic diagram of the workflow of a motion planning module according to an embodiment of the present invention is shown.

[0031] Figure 4 A schematic diagram of the VLA training process according to an embodiment of the present invention is shown.

[0032] Figure 5 A schematic diagram of the assembly task flow of a dual-arm robot according to an embodiment of the present invention is shown.

[0033] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0034] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] Figure 1 A flowchart of a dual-arm robot task assembly method based on behavior tree and VLA according to an embodiment of the present invention is shown. The method consists of four modules: behavior tree module, VLA module, perception module, and motion planning module. Specific steps include: S1. When a task starts, the behavior tree is executed starting from the root node, and signals are sent to the child nodes at a fixed frequency. When a leaf node receives a signal: the condition node returns Success or Failure based on the received proposition, and the action node returns Success, Failure or Running based on the operation execution result; the intermediate node performs logical judgment based on the status returned by the child node and passes the status upward. The behavior tree module takes a behavior tree structure as input, including a root node, at least one intermediate node, and at least one leaf node. The intermediate nodes include sequence nodes and backtracking nodes, while the leaf nodes include condition nodes and action nodes. The behavior tree serves as the robot's control framework, monitoring the robot's overall task process. The output is the behavior tree's return status, which has three categories: Success, Running, and Failure, representing task execution success, task execution in progress, and task execution failure, respectively.

[0037] A behavior tree consists of a root node, intermediate nodes, and leaf nodes. The root node is the top node of each behavior tree. Intermediate nodes are further divided into sequence nodes and fallback nodes. Each intermediate node contains at least one child node, used to implement logical jumps between child nodes. Leaf nodes do not contain child nodes and are divided into action nodes and condition nodes, used to implement condition judgment and action execution.

[0038] The execution of a behavior tree begins at the root node, which generates a signal of a fixed frequency and passes it down to its child nodes. Each child node is idle when it does not receive a signal. The operating mechanisms of the two types of intermediate nodes and the two types of leaf nodes are explained below.

[0039] Sequential nodes: When a sequential node receives a signal, it traverses its child nodes from left to right. When a child node returns Failure / Running, the sequential node stops traversing and returns Failure / Running. A sequential node only returns Success when all its child nodes return Success.

[0040] Back node: When a back node receives a tick, it traverses its child nodes from left to right. When a child node returns Success / Running, the back node stops traversing and returns Success / Running. The back node only returns Failure when all child nodes return Failure.

[0041] Action Node: When an action node receives a signal, it begins to execute the corresponding operation. If the operation is completed, it returns "Success"; if it fails, it returns "Failure"; if it is in progress, it returns "Running".

[0042] Condition node: When a condition node receives a signal, it determines whether the predefined proposition is correct. If correct, it returns Success; otherwise, it returns Failure.

[0043] Therefore, the root node determines the state of a behavior tree based on the execution feedback passed from bottom to top.

[0044] In a specific embodiment of the present invention, if the behavior space is unstable, a Tick acceleration mechanism is activated. When the robot performs an action and exhibits behavioral abnormalities, such as increased trajectory deviation (first derivative of deviation > threshold), drastic changes in joint torque, detection of approaching obstacles, increased dual-arm collaboration error, or decreased confidence level of the VLA module output, the system automatically adjusts the Tick period from, for example, 100ms to 20ms. Where U(t) is the comprehensive instability index and k represents the sensitivity coefficient. When performing simple actions, such as approaching a fixed trajectory of a component, where visual requirements are low, the trajectory is stable, or the decision does not change much, Tick automatically increases from, for example, 10ms to 100ms, thereby saving computational resources.

[0045] S2. Utilize the perception module to receive proposition requests from condition nodes in the behavior tree, parse the propositions to extract target objects and spatial relationships, call the visual recognition algorithm to obtain the three-dimensional information and attributes of the target objects, calculate the relative relationships between objects, match the proposition requirements, and return Success if the requirements are met, otherwise return Failure.

[0046] The input to the perception module is the proposition request sent by the condition nodes in the behavior tree, and the output is the judgment of the corresponding proposition. If the judgment is true, it returns Success; if the judgment is false, it returns Failure.

[0047] The workflow of the perception module is as follows: Figure 2 As shown. During the assembly task, the system needs to determine whether a certain propositional condition is true based on the visual information of the current environment. The system first analyzes the proposition and extracts the target objects or spatial relationships involved, such as descriptive conditions like "the red cube is to the left of the blue cylinder".

[0048] Subsequently, the system invokes the visual recognition module to run a scene perception algorithm to obtain 3D information about relevant objects, including their position, pose, and category attributes. After successfully obtaining the scene data, the system further calculates the relative positional relationships between multiple objects in the proposition and matches and verifies these relationships against the proposition requirements. For example, it determines whether the spatial distance is within a threshold range, whether the orientation relationship meets the requirements, and whether all target objects have been detected.

[0049] If the analysis results show that the propositional conditions are fully met, the system returns "Success," and the behavior tree determines the condition node as true, thus allowing the task flow to continue. Conversely, if the recognition fails or the conditions are not met, "Failure" is returned, the condition node is determined as false, and the task flow will select an alternative path or retry the perception based on the behavior tree logic.

[0050] This process ensures that the system can accurately judge complex environmental conditions based on visual and spatial understanding capabilities, providing support for overall task planning and decision-making.

[0051] S3. Using the motion planning module, if the leaf node is an action node, the initial path of the two arms is generated according to the target pose. The time parameterization and trajectory smoothing are performed according to the dynamic constraints of the robotic arm. The two arms are then synchronously controlled to move along the planned trajectory. The trajectory deviation and external interference are monitored in real time. If a deviation is found, the trajectory is adjusted or the speed is compensated until the target action is completed.

[0052] The motion planning module takes as input the target pose from the action nodes in the behavior tree and outputs the joint angles of both arms and the opening / closing command P=[ , ].

[0053] The workflow of the motion planning module is as follows: Figure 3 As shown, to ensure safe and precise synchronous movement of the robot's two arms under spatial position, velocity, and dynamic constraints, a complete planning and control process is required. First, based on the task requirements, the pose information of the target object is input, and the system initiates a path search algorithm to generate a collision-free initial path for both arms from their current position to the target position. Then, according to the robot arm's time parameterization rules and dynamic constraints, the initial path is smoothed to obtain a synchronous trajectory for both arms that satisfies velocity, acceleration, and joint constraints.

[0054] Once the trajectory is generated, a synchronization waiting phase begins, ensuring both arms are ready to execute actions based on the same time reference. Upon entering the command execution phase, the controller sends synchronization trajectory commands to both arms and monitors their motion status in real time, including trajectory errors, speed deviations, and external interference. If asynchrony in the arm trajectories is detected, the system will trigger corresponding adjustment strategies, such as replanning local trajectories or performing speed compensation, to restore arm synchronization.

[0055] In the aforementioned motion planning and trajectory optimization process, the cooperative cost function, which includes the interdependence of both arms, has the objective function: S = ,in, and The first term represents the independent cost of the left and right robotic arms, while the latter terms with λ are "coupling terms".

[0056] This represents the relative pose coupling cost, keeping both arms within a stable "cooperative frame." The specific formula is as follows: ,in, and Indicates the position of the left and right arm ends. This indicates the desired relative posture of the left and right arms during the task (e.g., keeping the clamps at both ends parallel).

[0057] This represents the cost of synchronized movement, allowing both arms to "do things simultaneously," rather than "one arriving first and the other half a beat behind." The specific formula is as follows: The parameters represent the velocities and accelerations of the left and right arms, respectively.

[0058] This represents the geometric constraint cost of a collaborative task. When two arms manipulate the same part, the object itself restricts the relative movement of the two arms. For example, when two arms are jointly moving a board, the shape of the board determines the positional relationship between the two hands. The specific formula is as follows: ,in This indicates the consistency of the gripping points of both arms on the geometric model of the object.

[0059] This represents the cost of occlusion constraints, preventing the two arms from blocking each other's visual field, ensuring the effectiveness of the perception module, and controlling the angle between the end effector and the camera's line of sight within a preset threshold.

[0060] Finally, once both arms have completed the planned trajectory and reached the target position, the system ends the synchronous motion task and enters the task exit or next stage process.

[0061] S4. Receive scene image information and user input instructions, input the pre-trained multimodal VLA model to generate dual-arm joint angle and end effector opening and closing instructions, and send them to the dual-arm controller for execution; during the action execution process, the VLA module iteratively generates subsequent action instructions based on the current scene information until the assembly operation is completed.

[0062] S5. The execution logic for task assembly also includes: The behavior tree nodes are visited sequentially. First, the task completion judgment condition node is visited. If the task is not completed, the safety condition judgment and component distance judgment condition nodes are visited. The corresponding action node is selected and executed according to the output result of the perception module, including stopping operation, returning to the initial position, approaching the component, or assembly operation. During the assembly operation, the status of the two arms and the VLA output result are monitored in real time. If the safety condition is not met, the stop operation step is triggered. When all assembly operations are completed and the task completion judgment returns "Success", the task execution ends.

[0063] If the sensing module determines that the robotic arms are not near the component based on their status, it returns a Failure. At this point, the "Approaching Component" action node is accessed. This node, through the system's motion planning module, controls the arms to move closer to the component. Since the component is deployed in a fixed area, this only requires controlling the arms to complete a relatively fixed trajectory movement; visual recognition is not needed.

[0064] After completing the operation to approach the component, the node returns "Success". At this point, the action node "Assembly Operation" is accessed, which completes the assembly operation by calling the VLA module. VLA then continuously collects visual and auditory information and outputs the joint angle information of both arms.

[0065] This joint angle information is sent to the controller of the two arms to control the movement of each joint, achieving precise movements of the arms. Simultaneously, after each control operation, the VLA continues to output joint angle information based on the current scene information, iteratively controlling the arms to complete a series of precise movements, thereby completing the component assembly operation.

[0066] The VLA module receives scene image information acquired by the sensing module and user input; its output consists of the joint angles of the two arms and the opening / closing commands of the end effector. Each robotic arm has seven degrees of freedom and one end effector opening / closing degree of freedom. The VLA outputs a total of 16 parameters, represented as: P=[ , ],in, This indicates the output parameters of the left arm. =[ , , , , , , , ],in , , , , , , These represent the angles of motion for each joint. This indicates the opening and closing angle of the end effector. This represents the output parameters of the right arm. =[ , , , , , , , ],in, , , , , , , These represent the angles of motion for each joint. This indicates the opening and closing angle of the end effector.

[0067] The training process of VLA is divided into three stages, such as Figure 4 As shown.

[0068] 1) First stage: Pre-training; Extensive demonstration data (such as videos, joint torques, and trajectories) from multiple robots and tasks are used, containing basic "action-visual scene" correspondences. This data is then input into a Transformer-based multimodal pre-trained model. Through self-supervised learning objectives, the model learns to align visual features, language commands, and basic action representations in the same high-dimensional space, thereby obtaining a basic VLA model.

[0069] 2) Second stage: Instruction fine-tuning and reinforcement learning; This is an iterative optimization process designed to adapt the model to the real-world environment. The base VLA model is deployed to a real robot or high-fidelity simulation environment, interacting with the environment based on received instructions. During this interaction, the system continuously collects data, including: environmental state, actions taken by the model, the results of these actions (success / failure / degree), and corresponding natural language instructions. Using the collected high-quality interaction data (especially successful data), the model is fine-tuned through supervised learning. Finally, the fine-tuned model is systematically evaluated to determine its performance, such as task success rate, safety, and efficiency, to see if it meets deployment standards.

[0070] 3) Third stage: After the first and subsequent phases, an optimized and mature model is obtained, capable of understanding complex instructions and executing precise movements in real-world environments. This model can be applied to practical scenarios. Upon receiving instructions and scenario information, it generates joint angles and opening / closing commands. Simultaneously, VLA generates joint angles and opening / closing commands for the next movement in real time based on the execution feedback from both arms, gradually completing the task.

[0071] S7. During the assembly process, the behavior tree periodically traverses all child nodes. If the bi-arm joint angle information output by VLA deviates from the task expectation, the "Safety Condition Determination" node will return Failure when accessed. At this time, the system will jump to step S3, thereby ensuring the safety of the overall task.

[0072] S8. If assembly is complete, the condition node "Task Completion Judgment" will return "Success". Then, the process will jump to step S2, indicating that the task is complete.

[0073] The flowchart of the assembly task of the dual-arm robot is as follows: Figure 5 As shown, when a task starts, the system first uses the perception module to determine if the task has been completed: if so, the root node directly returns "Success," and the process ends; if not, it continues to determine if the system is in a safe state. If it is unsafe, the motion planning module will stop the current operation and return to the initial position; if it is safe, it further determines whether it has approached the target component. If it has not yet approached, the motion planning module will execute the operation to approach the component; if it has approached, it will call the VLA module to perform the assembly operation. The entire process embodies the structured control logic of hierarchical decision-making, conditional judgment, and module invocation.

[0074] The advantages of traditional robotic arm motion planning are high precision, reliability, and determinism: in structured environments, for known, repetitive tasks, motion planning can generate extremely accurate and stable trajectories, ensuring task success. However, its disadvantage is weak generalization ability.

[0075] The advantage of VLA lies in its strong generalization and flexibility: it can understand natural language instructions and generalize to unseen objects and scenarios. It can respond to abstract instructions such as "put the apple in that blue plate." However, the disadvantages of VLA are reliability and safety: the output is random and may produce "hallucinations" or unsafe actions.

[0076] This invention combines behavior trees with VLA (Variable Allocation). Behavior trees are used to manage the overall assembly task of a dual-arm robot. For known, repetitive tasks, motion planning algorithms are employed to improve the reliability of dual-arm operations. For intricate assembly processes, VLA is used to achieve generalized assembly operations, increasing the success rate. Simultaneously, the behavior tree monitors the assembly process in real time, switching to motion planning mode when the robot is in an unsafe state to ensure the safety of the task.

[0077] In summary, this invention clarifies the task's operational flow through behavior trees, constrains the usage boundaries of robot motion planning algorithms and VLA, and improves the safety, execution efficiency, and success rate of collaborative assembly of dual-arm robots.

[0078] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing electronic devices according to embodiments of the present application. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0079] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0080] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a liquid crystal display (LCD) and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card and a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0081] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0082] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0085] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device: when a task is started, executes a behavior tree starting from the root node, sending signals to child nodes at a fixed frequency; the condition node extracts the target object and its spatial relationships by parsing the proposition received by the perception module, and uses a visual recognition algorithm to obtain three-dimensional information and attributes to determine whether the requirements are met; the action node generates an initial path according to the motion planning module, optimizes the trajectory according to the dynamic constraints of the robotic arm, and monitors deviations and interferences during the movement process in real time, making timely adjustments to ensure task completion. This integrates user instructions and scene image information, and uses a pre-trained multimodal VLA model to generate joint angles and end effector instructions to guide efficient collaboration between the two arms until the assembly operation is successfully completed.

[0086] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for assembling a dual-arm robot task based on behavior tree and VLA, characterized in that, include: When a task starts, the behavior tree is executed starting from the root node, sending signals to child nodes at a fixed frequency. When a leaf node receives a signal: the condition node returns Success or Failure based on the received proposition, and the action node returns Success, Failure, or Running based on the operation execution result; the intermediate nodes perform logical judgments based on the status returned by the child nodes and pass the status upwards. The system uses a perception module to receive proposition requests from condition nodes in the behavior tree, parses the propositions and extracts the target objects and spatial relationships, uses a visual recognition algorithm to obtain the three-dimensional information and attributes of the target objects, calculates the relative relationships between objects, matches the proposition requirements, and returns Success if the requirements are met, otherwise it returns Failure. Using the motion planning module, if the leaf node is an action node, the initial path of the two arms is generated according to the target pose. The time parameterization and trajectory smoothing are performed according to the dynamic constraints of the robotic arm. The two arms are then synchronously controlled to move along the planned trajectory, and the trajectory deviation and external interference are monitored in real time. If a deviation is found, the trajectory is adjusted or the speed is compensated until the target action is completed. The system receives scene image information and user input commands, inputs a pre-trained multimodal VLA model to generate dual-arm joint angle and end effector opening and closing commands, and sends the dual-arm joint angle and end effector opening and closing commands to the dual-arm controller for execution. During the action execution process, the VLA module iteratively generates subsequent action commands based on the current scene information until the assembly operation is completed.

2. The dual-arm robot task assembly method based on behavior tree and VLA according to claim 1, characterized in that, The behavior tree includes a root node, at least one intermediate node, and at least one leaf node, wherein the intermediate node includes a sequence node and a backtrack node, and the leaf node includes a condition node and an action node.

3. The dual-arm robot task assembly method based on behavior tree and VLA according to claim 1, characterized in that, The VLA module includes the following training phases: In the first stage, multi-robot, multi-task demonstration data is used to pre-train the multimodal Transformer model to learn the alignment relationship between visual features, language commands and action representations in the same high-dimensional space. In the second stage, the basic VLA model is deployed to a real robot or high-fidelity simulation environment. The model performance is optimized through instruction fine-tuning and reinforcement learning, and supervised fine-tuning is performed using successful data. In the third stage, a mature VLA model is obtained, which generates the joint angles of the two arms and the opening and closing commands of the end effector based on the scene information and instructions, and iteratively generates actions based on the execution feedback until the task is completed.

4. The dual-arm robot task assembly method based on behavior tree and VLA according to claim 1, characterized in that, The VLA module iteratively generates subsequent action instructions based on the current scene information until the assembly operation is completed. It also generates a synchronous trajectory for both arms and monitors trajectory errors and speed deviations in real time during execution. If the trajectory deviates from the expected task, the local trajectory is replanned or speed compensation is performed.

5. The dual-arm robot task assembly method based on behavior tree and VLA according to claim 1, characterized in that, The execution logic for task assembly also includes: The behavior tree nodes are visited sequentially. First, the task completion judgment condition node is visited. If the task is not completed, the safety condition judgment and component distance judgment condition nodes are visited. The corresponding action node is selected and executed according to the output result of the perception module, including stopping operation, returning to the initial position, approaching the component, or assembly operation. During the assembly operation, the status of the two arms and the VLA output result are monitored in real time. If the safety condition is not met, the stop operation step is triggered. When all assembly operations are completed and the task completion judgment returns "Success", the task execution ends.

6. The dual-arm robot task assembly method based on behavior tree and VLA according to claim 5, characterized in that, During the assembly task execution, the status of the child nodes is continuously polled to achieve dynamic task adjustment and safety control.

7. The dual-arm robot task assembly method based on behavior tree and VLA according to claim 5, characterized in that, The action of approaching the component follows a preset fixed trajectory.

8. The dual-arm robot task assembly method based on behavior tree and VLA according to claim 1, characterized in that, The behavior tree sends Tick signals at a fixed frequency and determines the execution path in real time based on the return status of the child nodes from bottom to top.

9. A computer program product, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

10. A computing system, characterized in that, It includes a processor and a memory, the processor being configured to perform the method as described in any one of claims 1-8.

Images