A visual language planning method and system applied to long sequence object rearrangement task of multi-arm robot

Abstract

This invention provides a visual language planning method and system for long-term object rearrangement tasks in multi-arm robots. The method includes: a visual language task decomposition module that fine-tunes a multimodal large language model based on a hierarchical dataset, combining perceptual anchoring and temporal inference loss to decompose high-level language instructions and visual observations into multiple sets of semantically coherent candidate sub-task decomposition schemes; a dual-criteria sub-task selection module that filters and optimizes candidate schemes from both consistency and spatial dispersion dimensions to obtain the optimal master scheme and convert it into executable action primitives; and a multi-arm action allocation module based on proximal policy optimization that combines a composite reward function consisting of motion distance penalty, parallelism reward, and collision penalty to generate collision-free, high-parallelism multi-arm cooperative execution trajectories. This invention effectively alleviates the semantic illusion problem of multimodal large language models and compensates for their shortcomings in long-term spatial inference and multi-arm cooperative planning capabilities.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology and relates to a visual language planning method and system for long-term object rearrangement tasks of multi-arm robots. Background Technology

[0002] With the development of embodied intelligence and multi-robot collaboration technologies, robots are gradually evolving from structured industrial scenarios with single arms and single tasks to complex multi-arm collaborative tasks in unstructured environments such as industrial assembly, home services, and intelligent sorting. In these scenarios, multi-arm robots need to complete long-term object rearrangement operations based on natural language instructions, gradually breaking down high-level semantic tasks into continuously executable sub-tasks, and achieving efficient execution through multi-arm collaboration. This places higher demands on the consistency of task reasoning, the feasibility of action generation, and the parallel efficiency of multi-arm scheduling. However, significant challenges remain in practical applications: On the one hand, as the task chain grows, existing multimodal large language models are prone to semantic shifts, object recognition errors, and inconsistent spatial relationships during task decomposition, resulting in unexecutable or disjointed subtasks; on the other hand, during the execution of multiple robotic arms, their motion paths, obstacle avoidance constraints, and spatial interference are highly coupled, and existing methods often lack unified modeling from the semantic layer to the execution layer, either focusing only on high-level task decomposition while ignoring parallel feasibility, or only performing low-level motion planning without semantic guidance, leading to low efficiency in collaboration between robotic arms and problems such as path conflicts, excessively long waiting times, and increased collision risks.

[0003] Furthermore, existing methods are mainly divided into policy methods based on end-to-end learning and task decomposition methods based on multimodal large language models. The former can directly generate actions but is difficult to handle long-term reasoning and has limited generalization ability. The latter has certain semantic understanding ability but lacks constraints on physical feasibility and spatial distribution, making it difficult to adapt to multi-arm parallel execution scenarios. At the same time, although traditional multi-robot scheduling methods can optimize paths and obstacle avoidance, they lack language understanding and high-level planning capabilities, making it difficult to meet the needs of complex semantic-driven tasks.

[0004] In summary, existing technologies generally suffer from a disconnect between high-level semantic reasoning and low-level collaborative execution in long-term multi-arm reordering tasks, making it difficult to simultaneously ensure subtask consistency, execution parallelism, and operational safety. There is an urgent need for a planning method that integrates visual language reasoning and multi-arm collaborative optimization to improve the execution efficiency and stability of robot systems in complex scenarios. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a visual language planning method and system for long-term object rearrangement tasks of multi-armed robots, so as to alleviate the semantic illusion of multimodal large language models and the insufficient long-term spatial reasoning and multi-armed collaborative planning capabilities.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A visual language planning method for long-term object rearrangement tasks in multi-arm robots is proposed. This method constructs a unified planning framework integrating visual language task decomposition, dual-criteria subtask selection, and deep reinforcement learning-based multi-arm scheduling. The method specifically includes the following steps: S1: Obtain robot task rescheduling instructions Visual observation sequence With scene state The Visual Language Task Decomposition Module (VLDec) generates multiple sets of semantically coherent candidate subtask decomposition schemes. ; S2: Based on the candidate subtask decomposition scheme The optimal main solution is selected through the dual-criteria subtask selection module DiSS and transformed into an executable action primitive A, resulting in a high-quality execution solution. ; S3: Based on the aforementioned high-quality execution scheme By combining the state and motion constraints of the robotic arm, an efficiency-oriented multi-arm collaborative allocation strategy is generated through the deep reinforcement learning multi-arm motion allocation module RLAlloc. S4: Based on the multi-arm collaborative allocation strategy, generate a collision-free, highly parallel execution trajectory, and complete the long-term object rearrangement task of the multi-arm robot based on the trajectory.

[0007] Furthermore, in step S1, the visual language task decomposition module VLDec is constructed based on a multimodal large language model, and its process for generating candidate subtask decomposition schemes includes: The multimodal large language model is fine-tuned using a hierarchical dataset consisting of a perceptual alignment dataset, a temporal reasoning dataset, and a task decomposition dataset. During fine-tuning, the perceptual anchoring loss used to align visual and textual features is jointly optimized, along with the temporal inference loss used to ensure the coherence of the subtask sequence, so that the generated multiple sets of candidate decomposition schemes have semantic consistency.

[0008] Furthermore, in step S2, the process of the dual-criteria subtask selection module DiSS filtering to obtain the optimal main solution and converting it into an executable action primitive A specifically includes: For each candidate decomposition scheme, a consistency evaluation is performed, the existence probability and placement effectiveness of the objects involved in the sub-tasks are calculated, a consistency score is obtained, and a set of feasible schemes with a consistency score exceeding a preset threshold is retained. The subtasks in the set of feasible solutions are encoded into a shared embedding space and decoded into action primitives that include grasping pose and placement pose; Based on the spatial distribution of the action primitives, calculate the spatial dispersion score of each feasible solution; The feasible solution with the highest spatial dispersion score is selected as the optimal master solution.

[0009] Furthermore, in step S3, the multi-arm action allocation module RLAlloc is trained based on the proximal policy optimization algorithm, and its process of generating a multi-arm cooperative allocation strategy includes: Feature encoding is performed on the state of the robotic arm and the motion primitives to be assigned, and the compatibility score and geometric proximity between the robotic arm and the motion primitives are calculated. By path decoding, motion primitives are assigned to each robotic arm based on the compatibility score and geometric proximity. The composite reward function used in training consists of a weighted sum of a distance penalty term, a parallelism reward term, and a collision penalty term, in order to optimize the total task completion time.

[0010] Furthermore, the composite reward function Specifically, it is expressed as follows: in, This is a motion distance penalty term used to reduce the total movement path of the robotic arm; This is a parallelism reward item, used to increase the proportion of simultaneous execution by multiple arms; This is a collision penalty measure used to avoid interference and deadlock in the arm space; , , These are the weighting coefficients.

[0011] Furthermore, in step S4, after generating the execution trajectory and before executing the task, a feasibility verification process for the execution trajectory is also included, specifically including: Verify whether the motion primitives and multi-arm collaborative allocation strategy meet spatial constraints and obstacle avoidance conditions, including checking the legality of the grasping and placement postures, verifying that the minimum distance between robotic arms is greater than a preset safety threshold, and ensuring that there are no path intersections.

[0012] Furthermore, after a task fails, a replanning process is also included, specifically: Re-trigger the visual language task decomposition module to generate new candidate subtask decomposition schemes; The dual-criteria subtask selection module re-selects the optimal main scheme from the new candidate subtask decomposition schemes and updates the action primitives and multi-arm collaborative allocation strategy. The trajectory is regenerated based on the updated multi-arm cooperative allocation strategy and an attempt is made to execute it.

[0013] This invention also provides a visual language planning system for long-term object rearrangement tasks in multi-arm robots, the system comprising: The acquisition module is used to obtain reordering instructions. Visual observation sequence With scene state ; The Visual Language Task Decomposition Module (VLDec) is used to generate multiple [data / resources] based on the information acquired by the acquisition module. A semantically coherent candidate subtask decomposition scheme; The dual-criteria subtask selection module DiSS is used to select the optimal main scheme from the candidate subtask decomposition schemes based on the dual criteria of consistency and spatial dispersion, and transform it into an executable action primitive. The deep reinforcement learning multi-arm action allocation module RLAlloc is used to generate an efficiency-oriented multi-arm collaborative allocation strategy based on the action primitives and the state and action constraints of the robotic arm. The execution module is used to generate collision-free and parallel execution trajectories according to the multi-arm cooperative allocation strategy, and to complete the long-term object rearrangement task of the multi-arm robot.

[0014] The present invention also provides a multi-armed robot, including a processor, a memory, and a communication bus, wherein the processor executes a computer program stored in the memory to implement the visual language planning method described above.

[0015] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the visual language planning method as described above.

[0016] The beneficial effects of this invention are as follows: We construct a hierarchical task decomposition system (VLDec module) for joint visual-language reasoning. Through perceptual anchoring, temporal reasoning, and self-optimization iteration, we generate high-quality candidate subtasks, solving the problems of semantic inconsistency and physical infeasibility in long temporal reasoning, and significantly improving the accuracy and coherence of subtasks. Design a dual-criteria screening mechanism based on consistency and spatial dispersion (DiSS module) to complete the validity verification of subtasks and the evaluation of parallel potential before execution, prioritize the execution scheme that is conducive to multi-arm concurrency, reduce inter-arm interference and waiting, and significantly improve the parallelism and execution efficiency of multi-arm collaboration. We propose an efficiency-oriented multi-arm allocation strategy based on deep reinforcement learning (RLAlloc module), which jointly optimizes motion distance, parallelism and obstacle avoidance constraints, so as to minimize the total task completion time while ensuring collision-free safe execution. When deviations occur during task execution, candidate decomposition schemes can be quickly regenerated and the optimal plan can be screened again without restarting the task from scratch. This effectively reduces the risk of failure caused by single inference errors and significantly improves the overall success rate and robustness of complex long-term time-series rearrangement tasks.

[0017] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 An overview diagram of a visual language planning system for long-term object rearrangement tasks of multi-arm robots provided in this application embodiment; Figure 2 This is a schematic diagram of the overall framework of the three modules of MRoPlanner provided in an embodiment of this application; Figure 3 A simulation and real-world flowchart illustrating the multi-arm robot performing spelling and block rearrangement tasks, as provided in the embodiments of this application. Figure 4 A schematic diagram illustrating the robustness and consistency analysis of long-sequence task inference provided in this application embodiment; Figure 5 This is a schematic diagram illustrating the multi-arm scalability and spatial dispersion effects provided in the embodiments of this application (Note: the bar chart represents parallelism, the line represents speedup ratio; different curves represent the change in completion time with dispersion score under 2 / 3 / 4-arm configurations). Figure 6 This is a schematic diagram comparing the execution effects of the RLAlloc strategy and the greedy strategy in a real environment provided for embodiments of this application. Detailed Implementation

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0021] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0022] This invention proposes a visual language planning framework for long-term multi-arm object rearrangement tasks, applicable to multi-arm robots performing complex multi-step operations such as object classification, arrangement, spelling, and placement based on natural language instructions in desktop scenarios. Addressing issues such as semantic inconsistency and physical infeasibility of subtasks in long-term tasks, as well as motion coupling, collision waiting, and low execution efficiency in multi-arm collaboration, this invention proposes an integrated planning and scheduling method, mainly comprising three stages: First, constructing a visual language-guided task decomposition module, decomposing high-level language instructions and visual observations into multiple sets of semantically coherent candidate subtask plans; Second, constructing a dual-criteria subtask selection module, selecting the main plan with the highest consistency and optimal spatial dispersion from the candidate plans, and mapping the subtasks to robot-executable action primitives; Third, constructing a multi-arm action allocation module based on deep reinforcement learning, efficiently allocating action primitives to each robotic arm, maximizing parallelism and minimizing the total task completion time while avoiding collisions, ultimately enabling the multi-arm robot to stably and efficiently complete long-term rearrangement tasks.

[0023] The present invention provides a visual language planning method and system for long-term object rearrangement tasks in multi-arm robots, mainly comprising: In a first aspect, this application provides a visual language planning method for long-term object rearrangement in multi-arm robots; the method mainly includes: S1: Obtain instructions for long-sequence reordering tasks Visual observation sequence With scene state The Visual Language Task Decomposition Module (VLDec) generates multiple sets of semantically coherent candidate subtask decomposition schemes. ; S2: Based on the candidate subtask decomposition scheme The consistency and spatial dispersion evaluation are completed through the dual-criteria subtask selection module DiSS, the optimal master scheme is selected and transformed into an executable action primitive A; S3: Based on the optimal master scheme and action primitive A, combined with the state and action constraints of the robotic arm, an efficiency-oriented multi-arm collaborative allocation strategy is generated through the deep reinforcement learning multi-arm action allocation module RLAlloc. S4: Based on the multi-arm collaborative allocation strategy, generate a collision-free, highly parallel execution trajectory, and complete the long-term object rearrangement task of the multi-arm robot based on the trajectory.

[0024] By adopting the above technical solution, firstly, in the visual language task decomposition module VLDec, a multimodal large language model is fine-tuned using a hierarchical dataset to perform perceptual anchoring and temporal reasoning for long-term complex tasks, generating multiple sets of semantically consistent candidate decomposition schemes to alleviate the sub-task inconsistency problem caused by long-term reasoning; subsequently, in the dual-criteria sub-task selection module DiSS, the optimal scheme is selected from both semantic feasibility and spatial parallelism, and the sub-tasks are accurately mapped to robot-executable action primitives, improving the adaptability of subsequent multi-arm execution; in the multi-arm action allocation module RLAlloc, based on near... The end-to-end strategy optimizes the PPO training efficiency-oriented allocation strategy, combining three mechanisms: motion distance penalty, parallelism reward, and collision penalty, to optimize the multi-arm collaborative path. By completing high-quality sub-task decomposition and spatial constraint calibration before multi-arm execution, the semantic illusion problem of large language models can be effectively alleviated, and the deficiencies in long-term spatial reasoning and multi-arm collaborative planning capabilities can be made up for. This significantly improves the success rate, execution parallelism, and operational safety of the reordering task. Ultimately, higher task success rates and shorter completion times are achieved in both simulation and real-world scenarios, providing reliable support for the efficient and stable execution of long-term reordering tasks by multi-arm robots.

[0025] Secondly, this application provides a visual language planning system for long-term object rearrangement in multi-arm robots; the system includes modules for executing the method in the first aspect or any possible implementation of the first aspect: the system includes: an acquisition module for acquiring long-term rearrangement task instructions. Visual observation sequence With scene state The system includes: a visual language task decomposition module (VLDec) for generating multiple sets of semantically coherent candidate subtask decomposition schemes; a dual-criteria subtask selection module (DiSS) for filtering the optimal scheme from two dimensions and converting it into executable action primitives; a deep reinforcement learning multi-arm action allocation module (RLAlloc) for generating efficiency-oriented multi-arm collaborative allocation strategies; and an execution module for generating collision-free execution trajectories based on the allocation strategy to complete the multi-arm long-term object rearrangement task.

[0026] Thirdly, this application provides a multi-armed robot, including a processor, a memory, and a communication bus, wherein the communication bus is used to realize a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the method described in any of the preceding claims.

[0027] Fourthly, this application also provides a computer-readable storage medium storing a computer program; the computer program can be executed by a processor to implement the method described above.

[0028] Fifthly, this application also provides a computer program product, including a computer program that can be executed by a processor to implement the method described above.

[0029] To facilitate understanding of this invention, the following explanations are provided for key terms involved: Long-field rearrangement tasks refer to robotic desktop tasks that involve a large number of objects, multiple steps, and strong semantic constraints, such as arranging blocks by color, spelling out city names with letters, or placing objects with the same texture into corresponding containers. The subtask sequence is usually 6 to 18 steps long and requires the subtasks to maintain a high degree of consistency in both semantics and physical space.

[0030] Multi-arm collaboration refers to two, three or more robotic arms working together to perform the same task. This reduces the overall time by executing non-conflicting sub-tasks in parallel, while avoiding problems such as collisions between arms, path crossings, and unnecessary waiting.

[0031] Visual language planning refers to combining visual images acquired by the camera with natural language commands input by the user to automatically break down high-level complex tasks into low-level executable subtasks and align semantics with physical space.

[0032] Subtask consistency refers to the fact that each subtask obtained from the decomposition is semantically consistent with the task objective and physically consistent with the current scene state, including dimensions such as whether the object exists, whether the position is legal, and whether the action is executable.

[0033] Spatial dispersion refers to the sparseness of the distribution of the operation positions corresponding to each subtask in the robot's workspace. The higher the dispersion, the less likely path conflicts will occur when multiple arms execute in parallel, and the higher the overall execution efficiency.

[0034] Makespan, or total task completion time, refers to the total time consumed from task initiation to the completion of all subtasks. It is a core indicator for measuring the efficiency of multi-tasking collaboration.

[0035] A motion primitive refers to the smallest motion unit that can be directly executed by the robot's underlying controller. It is usually composed of grasping pose and placement pose, and contains three-dimensional coordinates and attitude information.

[0036] The hierarchical dataset consists of three levels of data: perceptual alignment, temporal reasoning, and task decomposition. It is used to progressively fine-tune multimodal large language models and improve their decomposition quality and temporal coherence under long-view tasks.

[0037] Dual-criteria evaluation refers to evaluating candidate decomposition plans from two dimensions: subtask consistency and spatial dispersion, in order to select the optimal master plan that is most suitable for efficient parallel execution by multiple arms.

[0038] Figure 1 This application provides an overview diagram of a visual language planning system for long-term object rearrangement tasks in multi-arm robots. Figure 2 This is a schematic diagram of the overall framework of the three modules of MRoPlanner provided in an embodiment of this application. Figure 3 This is a simulation and real-world flowchart illustrating the process of a multi-armed robot performing spelling and block rearrangement tasks, as provided in the embodiments of this application.

[0039] The visual language planning system for long-field-of-view object rearrangement tasks in multi-arm robots provided in this invention mainly includes three modules: a visual language-guided task decomposition module (VLDec), a dual-criteria subtask selection module (DiSS), and a deep reinforcement learning-based multi-arm action allocation module (RLAlloc). Specifically: The first phase employs a Visual Language-Guided Task Decomposition Module (VLDec), which receives task language instructions and visual observation sequences, and outputs multiple sets of candidate subtask plans with temporal coherence and semantic consistency. This includes the following steps: Step S11: Receive high-level language instructions for the long-view rearrangement task. The instructions contain the operation objectives and constraints, such as "spell the name of a common city" or "arrange all the blocks in two rows by color".

[0040] Step S12: Receive the desktop visual observation sequence acquired by the robot camera. Environmental text description It includes scene information such as object category, location, color, texture, and quantity.

[0041] Step S13: Receive visual observations, environmental descriptions, and task instructions. Through a multimodal large language model fine-tuned with a hierarchical dataset, generate multiple sets of different candidate subtask decomposition plans.

[0042] Step S14: Perform preliminary organization of multiple candidate plans to ensure that each subtask has a clear description of the object, action, and position, forming a structured candidate plan set.

[0043] Step S15: Output a set of candidate plans, which will be used as input to the subsequent dual-criteria subtask selection module for filtering.

[0044] Specifically, the process of generating candidate subtask plans by the visual language-guided task decomposition module can be divided into five core parts: data input, hierarchical dataset training, joint feature learning, candidate plan generation, and master plan self-refinement, as detailed below: The module first receives input data, including visual observation sequences. Environmental description With task instructions The visual observation sequence is acquired by an RGB-D camera, containing color images and depth information, which can provide clues such as the spatial position, outline, and distance of objects; the environmental description presents the composition and layout of objects on the desktop in text form; the task instructions are natural language input by the user, used to clarify the final task objective and guide the model to generate a sequence of sub-tasks aligned with the objective.

[0045] During the model training phase, this invention constructs a hierarchical dataset for fine-tuning a multimodal large language model. This dataset consists of three complementary subsets, which can be represented as follows: .in, For the perception alignment subset, the data is in the form of observation and environment description pairing, which is used to enable the model to learn the spatial correspondence between vision and text; It is a subset of temporal reasoning, and the data is in the form of continuous observations and continuous subtask pairings, which is used to learn the dependencies between adjacent steps; The task decomposition subset contains complete visual and language instructions and reference subtask sequences, used to learn the overall decomposition logic of long-vision tasks. The training process randomly samples batches from a unified dataset to achieve multi-objective joint optimization.

[0046] In the feature encoding stage, the module uses a frozen visual encoder to extract image features and maps these visual features to the language space through a lightweight projection layer, enabling the multimodal large language model to understand both visual and textual information simultaneously. During training, the model optimizes two loss functions simultaneously. The first is a token-level perceptual loss used to align visual and textual features, which can be expressed as: .in The length of the token for the subtask. The model predicts the first Sub-tasks One token, The first loss is for the real token, ensuring that the subtask description remains consistent with the real-world scenario. The second loss is the subtask-level inference loss, used to guarantee the temporal coherence of long sequences of subtasks, and can be expressed as: .in This represents the total number of steps in the subtask. The model predicts the first Sub-tasks For the true sub-task, this loss ensures that the decomposition sequence always progresses towards the final task objective. The total loss is the sum of the two, i.e. .

[0047] After completing basic training, the module introduces a master plan self-refinement mechanism. The optimal master plan is obtained through subsequent dual-criteria selection, and then this optimal plan replaces low-quality plans in the dataset. The model is then fine-tuned to gradually generate higher-quality candidate plans more suitable for multi-arm execution. The model base uses Qwen2.5-VL-7B and is fine-tuned using LoRA, updating only the visual projection layer, cross-modal projection layer, and attention component. This improves multimodal task inference capabilities while maintaining efficiency.

[0048] After the above process, the module can output multiple sets of structured candidate plans. Each set of plans consists of a series of executable and semantically clear subtasks, such as "place the blue letter S in the first position of the right column" and "place the red square in the third position of the first row", providing high-quality input for subsequent filtering and execution.

[0049] The second stage employs a dual-criteria subtask selection module (DiSS) to receive a set of candidate plans from VLDec, select the main plan with the highest consistency and optimal spatial dispersion, and map the subtasks to action primitives executable by the robot. This includes the following steps: Step S21: Perform multimodal consistency evaluation on each sub-task in each group of candidate plans to obtain the overall consistency score of each group of plans.

[0050] Step S22: Filter out candidate plans with insufficient consistency according to the preset threshold, and retain only the set of feasible plans that are physically feasible and semantically valid.

[0051] Step S23: Map all subtasks in the feasible plan to action primitives containing grasping pose and placement pose, and obtain the specific position of each subtask in three-dimensional space.

[0052] Step S24: Calculate the spatial dispersion score of each feasible plan based on the spatial position of the action primitives. The higher the dispersion, the more suitable it is for multi-arm parallelism.

[0053] Step S25: Select the plan with the highest spatial dispersion from the feasible plans as the final master plan, and output the corresponding action primitive sequence.

[0054] Specifically, the core of the dual-criteria subtask selection module lies in the joint screening of consistency verification and dispersion evaluation, which can simultaneously ensure the executability of the plan and the efficiency of multi-arm execution. The specific implementation is as follows: During the consistency evaluation phase, the module evaluates each subtask from two dimensions: object existence and placement effectiveness. Given the current observation... Environmental description Task instructions With candidate subtasks The multimodal large language model outputs the consistency score for this subtask: in Indicates the first The first plan The consistency score for each subtask is between 0 and 1. The overall consistency score for the plan is obtained by averaging the scores of all subtasks within the plan. .in For the first The total number of subtasks in each plan. In this embodiment, the consistency threshold is set to 0.8, and only plans with scores higher than the threshold are retained to form a set of feasible plans: This step eliminates unreliable plans that fail due to non-existent objects, invalid positions, or contradictory actions, ensuring that subsequent execution will not fail due to task errors.

[0055] In the subtask-to-action primitive mapping stage, the module uses the CLIPort architecture to encode the text subtask and visual observation into the same embedding space, and decodes them to obtain the robot-executable pick-place action primitives: in To capture the three-dimensional coordinates of the location, To capture the posture, The three-dimensional coordinates of the placement position, This mapping determines the placement orientation. Through this mapping, the text subtask is transformed into spatial coordinates that can be used for path planning, providing a basis for dispersion calculation.

[0056] In the spatial dispersion assessment phase, the module uses the Nearest Neighbor Index (NNI) to measure the spatial sparsity of the operation positions. First, the grab positions of all action primitives are extracted to form a point set. Then, the average nearest neighbor distance and point density are calculated to obtain the dispersion index: .in The average nearest neighbor distance between the capture points. Point density is the ratio of the number of target objects to the area of ​​the workspace. A higher value indicates a more dispersed distribution, and thus, safer multi-arm parallelism. For easier comparison, the value is normalized to the 0-1 range using the sigmoid function to obtain the dispersion score: in To control the coefficient of normalized kurtosis, It is the sigmoid activation function.

[0057] Finally, the module selects the plan with the highest dispersion score from the feasible plans as the optimal master plan: The master plan simultaneously satisfies the semantic and physical consistency of subtasks and the spatial distribution is suitable for multi-arm parallel execution, providing optimal input for subsequent multi-arm scheduling.

[0058] The third stage employs a deep reinforcement learning-based multi-arm action allocation module (RLAlloc) to receive the action primitive sequence corresponding to the optimal master plan. With the goal of minimizing the total task completion time, it efficiently allocates actions to each robotic arm and implements obstacle avoidance, waiting, and parallel scheduling. This includes the following steps: Step S31: Construct a multi-arm collaborative Markov decision process, defining the system state, robotic arm state, actions to be assigned, reward function, and interaction constraints.

[0059] Step S32: Independently encode the state and motion primitives of the robotic arm, and calculate the compatibility score between the arm and the motion through multi-head attention.

[0060] Step S33: Integrate geometric features such as compatibility score, arm-to-action distance, and action displacement to obtain the joint arm-action representation.

[0061] Step S34: Based on the joint representation and RestHead control signal, generate motion assignments or waiting instructions for each robotic arm.

[0062] Step S35: Use the PPO algorithm to train an efficiency-oriented strategy to maximize the cumulative reward, enabling the model to learn the allocation strategy with the shortest time, highest parallelism, and no collisions.

[0063] Step S36: Output the multi-arm collaborative motion sequence and control each robotic arm to execute in parallel until all tasks are completed.

[0064] This embodiment first constructs a multi-arm collaborative Markov decision process to uniformly define the system state, robotic arm state, actions to be assigned, reward function, and interaction constraints. The multi-arm action allocation module based on deep reinforcement learning aims for optimal efficiency, employing proximal policy optimization (PPO) to achieve end-to-end collaborative scheduling. The overall optimization objective is to minimize the total expected task completion time, i.e.: in Assign strategies to actions. This represents the total task completion time. During the state representation phase, the... Step System Status Composed of the states of all robotic arms and the actions to be assigned, it can be represented as: .in The total number of robotic arms, For the first The robotic arm in the first The state of a step is determined by the current and previous end effector poses. composition This is a sequence of action primitives to be assigned, thereby ensuring that the action assignment strategy can fully perceive global resources and execution constraints.

[0065] In the feature encoding stage, the module extracts state features using the arm encoder and motion encoder respectively, and calculates the compatibility score D between the arm and the motion through multi-head attention: in and These are the arm state encoder and the motion element encoder, respectively. This is a compatibility evaluation function, used to model the matching degree between the robotic arm's execution capabilities and the actions to be performed.

[0066] After calculating the compatibility score, the module further integrates geometric features such as arm-motion distance and motion displacement to obtain a joint arm-motion representation. The compatibility score, arm-motion distance, and motion displacement are normalized, concatenated, and input into a multilayer perceptron to obtain the final joint features. in The distance from the end effector of the robotic arm to the gripping point. To capture the motion displacement from the capture point to the placement point, This is a scene-scale normalization coefficient to ensure the consistency of features across different scenes.

[0067] To avoid spatial conflicts between arms due to excessive parallelism, the module introduces a Rest Head control unit, enabling the robotic arms to enter a waiting state. During the path decoding phase, the waiting control signals are fused, and action assignment instructions or waiting instructions for each robotic arm are generated based on the joint representation. in To wait for control signals, Path decoders are assigned to actions. The final allocation strategy is implemented using a mask probability distribution to avoid the same action being assigned repeatedly. ,in It is a binary mask used to mark whether the corresponding action has been assigned.

[0068] The module design employs an efficiency-oriented composite reward function that simultaneously implements distance-based penalties, reasonable parallel incentives, and inter-arm collision avoidance. The immediate reward expression is as follows: in The penalty is for the total distance traveled. For parallelism reward items, This is a penalty for arm-to-arm collisions. In this embodiment, the reward coefficient is set to [value missing]. During this period, the PPO algorithm is used to maximize the cumulative reward after discounts to achieve stable training. ,in This is a discount factor that enables the model to learn the optimal allocation strategy with the shortest completion time, highest parallelism, and no collisions.

[0069] Finally, after PPO training converges, the module outputs the globally optimal multi-arm collaborative action sequence. Based on this action sequence, the robot arms are controlled to perform operations synchronously and in parallel without conflict. The scheduling process is iterated until all sub-tasks are completed, thus achieving efficient execution of long-term multi-arm reordering tasks.

[0070] Through the above steps, the module can achieve optimal scheduling of multi-arm collaboration in complex long-view tasks, significantly improving parallelism, reducing total time consumption, and avoiding collisions, enabling multiple robotic arms to achieve efficient collaboration in desktop rescheduling tasks.

[0071] Optionally, considering the complexities of tabletop object placement in the real world, such as occlusion, positional offset, and texture similarity interference, and the long sub-task sequence of long-view tasks, the action allocation and execution sequence output by the embodiments of the present invention may not always be completely accurate or successful on the first attempt. Therefore, an iterative rollback and replanning strategy is also designed: when the multi-arm robot encounters interaction failures such as grasping failure, placement offset, inter-arm collision, or task interruption while executing a certain sub-task, the current execution process is immediately paused, and the robot returns to the dual-criteria sub-task selection module (DiSS). The suboptimal candidate decomposition plan generated in the visual language task decomposition module (VLDec) that has not been selected is called, and consistency verification and spatial dispersion evaluation are performed again to update the master plan and action primitives that are more suitable for the current real-time scene. Then, the updated action primitives are input into the multi-arm action allocation module (RLAlloc) based on deep reinforcement learning to regenerate a conflict-free and efficient multi-arm collaborative execution sequence, and the currently interrupted long-view rearrangement task is continued to be executed until the sub-task is completed or all candidate plans have been traversed.

[0072] To illustrate the effectiveness of this embodiment, the inventors evaluated 22 different multi-arm rearrangement tasks across two long-term benchmarks in a simulation environment and on a real robot platform, testing the overall performance of the present invention's technical solution and comparing it with five other different methods. The experiments covered two major standard datasets, Raven-Bench and VIMA-Bench (with tasks not appearing in the module training set), to verify the effectiveness, generalization ability, and execution efficiency of the present invention's technical solution.

[0073] The hardware used in the experiment consisted of three main parts: a multi-DOF Elephant robotic arm, an Intel RealSense D455 depth camera (for acquiring RGB-D visual observations), and a computing platform equipped with an NVIDIA RTX 4090 GPU, providing ample computing power for inference and training, and efficiently running the visual language decomposition, dual-criteria selection, and multi-arm reinforcement learning assignment modules. Task types included typical long-term sequential rearrangement tasks such as matching, moving, stacking, spelling, shape regularization, sorting, twisting, and following, covering 2-arm and 3-arm collaborative scenarios, and objects including various types such as cubes, letters, bowls, and tools.

[0074] In addition to using the overall success rate metric (SucR, %) to measure task completion rate, this embodiment employs two complementary metrics to evaluate task reasoning ability and multi-arm collaboration efficiency: At the task reasoning level, token accuracy (TokAcc, %) and semantic similarity (SemSim, ∈ [0,1]) are used to quantify the textual correctness and semantic alignment between the predicted subtask and the reference subtask; At the action allocation level, completion time (Mks, seconds), parallelism (Para, %), average waiting time (AWT, seconds), and speedup ratio (Speedup, ∈ [0,N]) are used to evaluate overall efficiency and inter-arm collaboration level. Higher parallelism and speedup ratio, and lower completion time and average waiting time, indicate higher collaboration efficiency.

[0075] To conduct baseline comparisons, this embodiment selects three representative methods: the first is the end-to-end method CLIPort, which directly maps visual language to low-level actions; the second is MLLM planning methods, including EmbodiedGPT, PAR, RoCo, and ReAct, all of which decompose tasks based on large models; and the third is traditional multi-arm scheduling methods, including greedy strategies, Hungarian algorithms, round-robin scheduling, and DRTA reinforcement learning allocation methods.

[0076] The specific explanations are as follows: (1) CLIPort: It directly outputs actions by imitation learning, without explicit subtask decomposition and multi-arm coordination; (2) EmbodiedGPT / PAR: It performs single-step or short-time decomposition based on visual language, without considering multi-arm parallelism; (3) RoCo / ReAct: It decomposes tasks through dialogue or reasoning, but lacks spatial constraints and efficiency optimization; (4) Traditional scheduling / DRTA: It only performs action-level allocation and does not have language and semantic reasoning capabilities.

[0077] To verify the comprehensive performance of the long-term multi-arm rearrangement visual language planning method, this embodiment conducted systematic experiments in a simulation environment and on a physical robot platform. First, the overall task performance was evaluated by comparing the method of this embodiment with existing baseline methods for visual language planning. The experimental results are shown in Table 1.

[0078] Table 1 Table 1 shows the task success rate and completion time on the Raven-Bench and VIMA-Bench test sets for two-arm and three-arm configurations, respectively, to characterize the method's ability to complete complex long-term multi-arm operation tasks. Quantitative results show that the method in this embodiment significantly outperforms existing baseline methods under all tasks and configurations. The average task success rate reaches 73.2% in the two-arm configuration and 63.7% in the three-arm configuration, with completion times reduced to 77.3s and 49.6s, respectively. Compared to the optimal baseline, this represents a maximum improvement of 30% in success rate and an 18% reduction in completion time. These results demonstrate that this embodiment, by combining visual language task decomposition, quality calibration semantic implementation, and efficiency-oriented multi-arm scheduling, can effectively improve task completion rate and reduce overall execution time.

[0079] Based on the overall performance verification, this embodiment further analyzes the semantic consistency and robustness of the method in long-term inference processes. To test the reliability of subtask generation under different time spans, the task decomposition effect is evaluated under different inter-frame intervals, and the experimental results are shown in Table 2.

[0080] Table 2 Table 2 records two metrics: token accuracy and semantic similarity, used to measure the textual accuracy and semantic alignment between the predicted sub-task and the reference sequence. Experiments show that the method in this embodiment maintains high inference quality under both continuous and interval observation conditions, with an average token accuracy of 96.5% and an average semantic similarity of 0.94, and performance fluctuation is less than 3% as the temporal interval increases. In contrast, traditional methods are prone to semantic breaks and sub-task misalignment in long-term inference, indicating that the hierarchical training and consistency constraints adopted in this embodiment can effectively maintain the stability and coherence of long-term inference.

[0081] To further clarify the role of each key component in long-running tasks, this embodiment performs ablation analysis on the task decomposition module. The experimental results are as follows: Figure 4 As shown, the method in this embodiment maintains a high token accuracy even as the length of the subtask sequence gradually increases. Removing the perceptual alignment training, temporal reasoning module, or consistency verification mechanism all cause significant performance degradation, which is more pronounced under long sequence conditions. This result demonstrates that joint learning of perceptual features and semantic reasoning, along with dual-criteria quality screening, are crucial to ensuring that long temporal tasks do not suffer from semantic bias, spatial relationship errors, or subtask logical confusion.

[0082] In terms of multi-arm action allocation, this embodiment focuses on evaluating collaborative execution efficiency and scheduling rationality. The experimental results are shown in Table 3.

[0083] Table 3 Table 3 compares the reinforcement learning allocation method of this embodiment with traditional greedy strategies, the Hungarian algorithm, round-robin scheduling, and DRTA learning-based allocation methods, statistically analyzing the completion time and average waiting time under different task sizes. The results show that the method of this embodiment can still maintain the shortest execution time and lowest waiting latency in intensive task scenarios, with an average waiting time reduction of 40.36%. Meanwhile, ablation experiments show that removing parallelism rewards or collision penalties significantly reduces execution efficiency, indicating that efficiency-oriented composite reward design plays a crucial role in improving collaborative efficiency and avoiding inter-arm conflicts.

[0084] Based on this, this embodiment further analyzes the influence of the system's multi-arm expansion capability and spatial dispersion. The experimental results are as follows: Figure 5 As shown. Among them. Figure 5 (a) Shows the trend of parallelism and speedup ratio as the number of robotic arms increases. The method in this embodiment maintains the highest parallelism and optimal acceleration effect under 2-arm, 3-arm and 4-arm conditions, and its scalability is significantly better than the comparison method. Figure 5 (b) The relationship between spatial dispersion and completion time is shown. As the spatial dispersion of subtasks increases, the overall execution time continuously decreases, and the decreasing trend is more pronounced with more arms. The above results indicate that higher spatial dispersion can reduce inter-arm interference and enhance parallel potential. This embodiment, through a dispersion-first scheme selection mechanism, can fully unleash the efficiency advantages of multi-arm collaboration.

[0085] Finally, this embodiment deployed the method on a real multi-arm robot platform for verification, and the experimental results are shown in Table 4. Figure 6 As shown in Table 4, subtask accuracy metrics are introduced. The degree of matching between instruction understanding and execution in real-world scenarios is evaluated through manual verification. The results show that the method in this embodiment maintains high subtask accuracy across various physical operation tasks. Removing the dual-criteria screening module significantly reduces accuracy, especially in stacked tasks with complex spatial relationships. Regarding execution efficiency, as... Figure 6 As shown, traditional greedy strategies are prone to problems such as robotic arm interference and pauses during collaboration. However, the method in this embodiment achieves smooth, conflict-free execution through reinforcement learning-based collaborative allocation, further reducing the actual completion time by approximately 7%. These results collectively demonstrate that the method in this embodiment not only exhibits excellent performance in simulation environments but can also be stably generalized to real robot systems, achieving significant improvements in semantic reliability, execution efficiency, and operational safety.

[0086] Table 4 To better execute the above method, this application also provides a multi-arm robot, including a processor, a memory, and a communication bus for enabling communication between the processor and the memory.

[0087] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for processing at least one function, and instructions for implementing the methods provided in the above embodiments; the data storage area may store data involved in the methods provided in the above embodiments.

[0088] Optionally, the memory may be a read-only memory, a random access memory, an electrically erasable programmable read-only memory, an optical disc, a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0089] The processor may include one or more processing cores, which execute instructions, programs, code sets, or instruction sets stored in memory, call data stored in memory, and perform various functions and process data as described in this application. The processor may be at least one of the following: application-specific integrated circuit, digital signal processor, field-programmable gate array, central processing unit, controller, microcontroller, and microprocessor.

[0090] A communication bus can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus. This communication bus can be categorized into address bus, data bus, and control bus, among others.

[0091] In an alternative embodiment, the robot may also include a communication interface for communication with other devices.

[0092] This application provides a computer-readable storage medium, including, for example, various media capable of storing program code such as a USB flash drive, portable hard drive, read-only memory, random access memory, magnetic disk, or optical disk. This computer-readable storage medium stores a computer program that can be loaded by a processor and execute the methods of the above embodiments.

[0093] This application also provides a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program containing program code for performing any of the methods described in any embodiment of this application, the computer program being downloadable and installable over a network, and / or installed from a removable medium.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A visual language planning method for long-temporal object rearrangement tasks in multi-arm robots, characterized in that, This method constructs a unified planning framework that integrates visual language task decomposition, dual-criteria subtask selection, and deep reinforcement learning multi-arm scheduling. The method specifically includes the following steps: S1: Obtain robot task rescheduling instructions Visual observation sequence With scene state The Visual Language Task Decomposition Module (VLDec) generates multiple sets of semantically coherent candidate subtask decomposition schemes. ; S2: Based on the candidate subtask decomposition scheme The optimal main solution is selected through the dual-criteria subtask selection module DiSS and transformed into an executable action primitive A, resulting in a high-quality execution solution. ; S3: Based on the aforementioned high-quality execution scheme By combining the state and motion constraints of the robotic arm, an efficiency-oriented multi-arm collaborative allocation strategy is generated through the deep reinforcement learning multi-arm motion allocation module RLAlloc. S4: Based on the multi-arm collaborative allocation strategy, generate a collision-free, highly parallel execution trajectory, and complete the long-term object rearrangement task of the multi-arm robot based on the trajectory.

2. The visual language planning method for long-term object rearrangement tasks in multi-arm robots according to claim 1, characterized in that, In step S1, the visual language task decomposition module VLDec is constructed based on a multimodal large language model, and its process of generating candidate subtask decomposition schemes includes: The multimodal large language model is fine-tuned using a hierarchical dataset consisting of a perceptual alignment dataset, a temporal reasoning dataset, and a task decomposition dataset. During fine-tuning, the perceptual anchoring loss used to align visual and textual features is jointly optimized, along with the temporal inference loss used to ensure the coherence of the subtask sequence, so that the generated multiple sets of candidate decomposition schemes have semantic consistency.

3. The visual language planning method for long-term object rearrangement tasks in multi-arm robots according to claim 2, characterized in that, In step S2, the process of the dual-criteria subtask selection module DiSS filtering to obtain the optimal main solution and converting it into an executable action primitive A specifically includes: For each candidate decomposition scheme, a consistency evaluation is performed, the existence probability and placement effectiveness of the objects involved in the sub-tasks are calculated, a consistency score is obtained, and a set of feasible schemes with a consistency score exceeding a preset threshold is retained. The subtasks in the set of feasible solutions are encoded into a shared embedding space and decoded into action primitives that include grasping pose and placement pose; Based on the spatial distribution of the action primitives, calculate the spatial dispersion score of each feasible solution; The feasible solution with the highest spatial dispersion score is selected as the optimal master solution.

4. The visual language planning method for long-term object rearrangement tasks in multi-arm robots according to claim 3, characterized in that, In step S3, the multi-arm action allocation module RLAlloc is trained based on the proximal policy optimization algorithm, and its process of generating a multi-arm cooperative allocation strategy includes: Feature encoding is performed on the state of the robotic arm and the motion primitives to be assigned, and the compatibility score and geometric proximity between the robotic arm and the motion primitives are calculated. By path decoding, motion primitives are assigned to each robotic arm based on the compatibility score and geometric proximity. The composite reward function used in training consists of a weighted sum of a distance penalty term, a parallelism reward term, and a collision penalty term, in order to optimize the total task completion time.

5. The visual language planning method for long-term object rearrangement tasks in multi-arm robots according to claim 4, characterized in that, The composite reward function Specifically, it is expressed as follows: in, This is a motion distance penalty term used to reduce the total movement path of the robotic arm; This is a parallelism reward item, used to increase the proportion of simultaneous execution by multiple arms; This is a collision penalty measure used to avoid interference and deadlock in the arm space; , , These are the weighting coefficients.

6. The visual language planning method for long-term object rearrangement tasks in multi-arm robots according to claim 5, characterized in that, In step S4, after generating the execution trajectory and before executing the task, a feasibility verification process for the execution trajectory is also included, specifically including: Verify whether the motion primitives and multi-arm collaborative allocation strategy meet spatial constraints and obstacle avoidance conditions, including checking the legality of the grasping and placement postures, verifying that the minimum distance between robotic arms is greater than a preset safety threshold, and ensuring that there are no path intersections.

7. The visual language planning method for long-term object rearrangement tasks in multi-arm robots according to claim 6, characterized in that, After a task fails, a replanning process is also included, specifically: Re-trigger the visual language task decomposition module to generate new candidate subtask decomposition schemes; The dual-criteria subtask selection module re-selects the optimal main scheme from the new candidate subtask decomposition schemes and updates the action primitives and multi-arm collaborative allocation strategy. The trajectory is regenerated based on the updated multi-arm cooperative allocation strategy and an attempt is made to execute it.

8. A visual language planning system for long-term object rearrangement tasks in multi-arm robots, characterized in that, The system includes: The acquisition module is used to obtain reordering instructions. Visual observation sequence With scene state ; The Visual Language Task Decomposition Module (VLDec) is used to generate multiple [data / resources] based on the information acquired by the acquisition module. A semantically coherent candidate subtask decomposition scheme; The dual-criteria subtask selection module DiSS is used to select the optimal main scheme from the candidate subtask decomposition schemes based on the dual criteria of consistency and spatial dispersion, and transform it into an executable action primitive. The deep reinforcement learning multi-arm action allocation module RLAlloc is used to generate an efficiency-oriented multi-arm collaborative allocation strategy based on the action primitives and the state and action constraints of the robotic arm. The execution module is used to generate collision-free and parallel execution trajectories according to the multi-arm cooperative allocation strategy, and to complete the long-term object rearrangement task of the multi-arm robot.

9. A multi-armed robot, characterized in that, It includes a processor, a memory, and a communication bus, wherein the processor executes a computer program stored in the memory to implement the visual language planning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the visual language planning method as described in any one of claims 1 to 7.

Images