A method and system for extending the actions of a single-arm proxy to dual-arm operation.

Abstract

This invention relates to a method and system for extending the actions of a single-arm agent to dual-arm operations, belonging to the field of robot control technology. The invention designs a framework for extending the actions of a single-arm agent to dual-arm operations. This framework utilizes a single-arm dataset to facilitate dual-arm operations, leveraging a broad single-arm dataset to facilitate dual-arm operations without requiring additional dual-arm data. This scheme introduces an attention extension mechanism that embeds semantic information into pixel-level data, allowing for the dynamic generation of action pairs. These actions are processed by a proposed TAN (Target Assignment Network), which assigns tasks between the two robotic arms, evaluates feasibility, and selects the optimal solution. Furthermore, this scheme can seamlessly transition back to single-arm operations when anomalies are detected, ensuring uninterrupted task execution. Experimental results show that the technical solution provided by this invention improves task efficiency while eliminating the cost of collecting dedicated dual-arm data.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology and relates to a method and system for extending the actions of a single-arm agent to dual-arm operations. Background Technology

[0002] In recent years, dual-arm robots have achieved remarkable success in numerous fields. They not only act as intelligent assistants in home environments, performing daily chores and caring for the elderly and children, but also demonstrate unparalleled precision and collaborative capabilities on industrial assembly lines, significantly improving production efficiency and product quality. Simultaneously, in agriculture, dual-arm robots, with their powerful operational capabilities and adaptability, provide strong support for crop planting, harvesting, and subsequent processing, contributing to the advancement of agricultural modernization. Behind these successes lies the continuous progress and innovation of dual-arm robot technology, particularly its outstanding performance in flexibility, coordination, and efficient execution of complex tasks.

[0003] However, with the continued growth in demand for dual-arm robots across various fields, a significant problem has gradually emerged: how to obtain high-quality dual-arm datasets to effectively train dual-arm manipulation strategies? This challenge mainly stems from the complexity and diversity of dual-arm interactions. In practical applications, dual-arm robots need to handle target coordination, differences in object properties, and various uncertainties in the real world, such as changes in ambient lighting and object position shifts, all of which greatly increase the difficulty and cost of data collection. Traditional data collection methods are often time-consuming and labor-intensive, and struggle to comprehensively cover all possible interaction scenarios, resulting in limited generalization ability of the trained models.

[0004] Furthermore, current dual-arm operation strategies are mostly based on predefined rigid rules, lacking sufficient flexibility and adaptability to cope with real-time changes. For example, if one arm malfunctions or encounters unexpected task irregularities during task execution, existing strategies often struggle to adjust in a timely manner, thus affecting the successful completion of the overall task. Therefore, developing a more dynamic and responsive dual-arm operation strategy is particularly important.

[0005] Against this backdrop, extending existing single-arm agent algorithms to the dual-arm manipulation domain has become a highly promising solution. The advantage of this approach lies in its ability to fully utilize existing large datasets of single-arm agents. Through algorithmic extension and transfer learning, it enables effective training of dual-arm agents, significantly reducing the cost and time of data collection. However, this extension process is not easy. Existing single-arm agent algorithms are primarily designed for sequential, single-objective tasks. When faced with scenarios requiring simultaneous manipulation of multiple objects or complex coordination tasks, their efficiency drops significantly. This is mainly because single-arm algorithms lack the ability to handle complex mechanisms such as the generation, allocation, interdependence, and spatial relationship coordination of dual-objective actions.

[0006] Therefore, to truly achieve efficient training and operation of dual-arm robots, deep innovation at the algorithmic level is necessary to develop new algorithms that can effectively manage the generation and allocation of dual-target actions and achieve efficient coordination and synchronization between the arms. Through such expansion and optimization, not only can existing single-arm datasets be fully utilized, but the execution efficiency and adaptability of dual-arm robots in various complex tasks can also be significantly improved, laying a solid foundation for the widespread application of dual-arm robot technology. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method and system for extending the actions of a single-arm proxy to dual-arm operation. This method and system provide a framework called ExS2D (Extending Actions of a Single-Arm Proxy to Dual-Arm Operation), which comprises two main components: 1) a target generation network; and 2) a target assignment network. The target generation network embeds semantic information into pixel-level data by introducing an attention extension mechanism, deeply mining and fully utilizing the features of semantic and visual data to dynamically generate action pairs. The target assignment network, based on the current state of the environment, assigns the action pairs dynamically generated by the target generation network to the two robotic arms, evaluates feasibility, and selects the optimal solution to achieve efficient dual-arm operation. Specifically, when an anomaly is detected in one robotic arm, the action pairs originally executed by that arm are automatically transferred to the other robotic arm to ensure uninterrupted task execution.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for extending the actions of a single-arm proxy to a two-arm operation, the method specifically includes the following steps:

[0010] S1: Construct an attention extension module that uses global visual information o and natural language instructions l to obtain the first picking point;

[0011] S2: Construct an object segmentation module to generate a mask for the first object based on the first pick point;

[0012] S3: Based on the first object mask, the attention extension module is used again to generate the second pickup point;

[0013] S4: Construct the transport module, which uses global visual information, natural language instructions, and the coordinates of the first pick point to obtain the coordinates of the first placement point;

[0014] S5: Based on obtaining the coordinates of the first placement point, the object segmentation module is used again to generate a mask for the first placement area;

[0015] S6: Based on obtaining the first placement area mask, the transport module is used again to generate the second placement point;

[0016] S7: Construct a state encoder that uses the target state diagram to obtain the encoded features of the target state diagram;

[0017] S8: Construct a cost encoder that uses the cost matrix to obtain the encoded features of the cost matrix;

[0018] S9: Construct a target assignment module that utilizes the encoded features output by the state encoder and the encoded features output by the cost encoder, and combines them with the motion planning library to achieve effective coordination, thereby assigning the target action to the corresponding robotic arm.

[0019] Furthermore, in step S1, an attention extension module is constructed, which obtains a probability map of the optimal pickup point for the first object by fusing natural language instructions with visual data. The pixel with the highest probability value on the probability map is the optimal pickup point for that object. The coordinates of the first pickup point are obtained by the following formula:

[0020]

[0021] Among them, Q pick It is the action value function for the picking operation, and (u,v) are the pixel coordinates on the image.

[0022] Further, in step S2, an object segmentation module is constructed. Based on the generated coordinates of the first picking point, a 64×64 pixel cropping block is cropped centered on that point. This cropping block can completely contain the first object. Then, the mask of the first object is obtained using SAM, and the global image containing the mask of the first object is generated. m1 This serves as new visual information for use in subsequent processes.

[0023] Furthermore, in step S3, the natural language instruction l and the global image o obtained in step S2 are used. m1The attention extension module is then used again to obtain a probability map of the optimal pickup point for the second object. Since the first object has already been processed by SAM, the attention extension module can focus its attention on the second object, thus obtaining the second pickup point more accurately. Similar to solving for the coordinates of the first pickup point, the pixel with the highest probability in the probability map is the optimal pickup point. The coordinates of the second pickup point are obtained by the following formula:

[0024]

[0025] In step S4, a transport module is constructed. The input visual information and natural language commands are consistent with those input in step S1. The transport module uses this data and combines it with the coordinates of the first pick point to generate a probability map about the first placement point. The pixel with the highest probability value in the probability map is the optimal placement point for the first object. The coordinates of the first placement point are obtained by the following formula:

[0026]

[0027] Among them, Q place It is the action value function for the placement operation, where Δτ is the pixel coordinates on the image.

[0028] Further, in step S5, a SAM module is constructed to crop an 8×8 pixel area centered on the first placement point. The SAM is then used to generate a mask for this area to ensure that the transport module focuses on generating the possible placement area for the second object. The global image containing the mask of the first placement point area is then... m2 This new visual information is used to solve for the second placement point in the subsequent process.

[0029] Furthermore, in step S6, the natural language instruction l and the global image o obtained in step S5 are combined. m2 In addition to the coordinates of the second pickup point obtained in step S3, the transport module is used again to obtain a probability map of the second placement point. The pixel with the highest probability value in the probability map is the optimal placement point for the second object. The coordinates of the second placement point are obtained by the following formula:

[0030]

[0031] Furthermore, in step S7, a state encoder is constructed, which captures the spatial relationship between the target and the two arms by processing the observations in the target state diagram. MLP and MHSA are used to embed the position data of the task and the robotic arms into the feature vector. The target node in the target state diagram contains the corresponding object's action posture in the Cartesian coordinate system, i.e., the object's pick-up and placement coordinates, expressed by the following formula:

[0032]

[0033] T1 and T2 represent the coordinates of the pick-up and place-up operations, respectively; the warehouse node in the target state diagram contains the starting position of the corresponding robotic arm, expressed by the following formula:

[0034]

[0035] Arm1 and Arm2 represent the Cartesian coordinates of the two robotic arms, respectively.

[0036] Furthermore, in step S8, a cost encoder is constructed, which collects execution costs from the cost matrix. As the robotic arm performs pick-up and place-down actions, the cost matrix dynamically changes with the constantly changing working state. The cost matrix consists of a 2×2 grid, and each grid cell contains two types of costs: movement cost c. mv and transfer costs c tf Here, movement cost represents the distance the robotic arm travels from its current position to the target object; transfer cost represents the distance the target object is moved from the pick-up position to the placement position; the solutions for both costs are expressed by the following formulas:

[0037]

[0038] In cases where operations are not feasible, such as assigning the same task to two arms or assigning a task that has already been completed, the cost in each individual grid cell will be set to infinity. The cost matrix captures the path length from the robotic arm's starting position to the two action pairs and the corresponding transfer costs:

[0039]

[0040] Wherein, cost(T) i Arm j () is the total path length from the starting position to the pick-up position and then to the delivery position of the robotic arm; when a robotic arm encounters a malfunction, the action that should have been performed by this robotic arm will be seamlessly transferred to another robotic arm to ensure uninterrupted execution of the action.

[0041] Furthermore, in step S9, a task allocation module is constructed, which uses an MHSA layer to fuse the collaborative information of the target node and the cost matrix, and then applies an MLP layer and a probability maximization operation to output the optimal task allocation strategy for both arms; the entire learning process optimizes the allocation of actions by minimizing the cross-entropy loss function.

[0042] The final output of the target assignment network is generated based on a cost matrix, where each task is assigned to a robotic arm with a lower execution cost. The output of the target assignment network is represented as follows:

[0043]

[0044] Where A(1) and A(2) represent two action pairs that have been assigned to the robotic arm Arm respectively. j and Arm k .

[0045] The present invention also provides a system for extending the actions of a single-arm agent to dual-arm operation.

[0046] The beneficial effects of this invention are as follows:

[0047] This invention provides a framework called ExS2D (extending single-arm proxy actions to dual-arm operation), which comprises two main components: 1) a target generation network and 2) a target assignment network. The target generation network embeds semantic information into pixel-level data through an attention extension mechanism, deeply mining and fully utilizing the features of both semantic and visual data to dynamically generate action pairs. The target assignment network, based on the current environment, assigns the dynamically generated action pairs to the two robotic arms, assesses feasibility, and selects the optimal solution to achieve efficient dual-arm operation. Notably, when an anomaly is detected in one robotic arm, the action pairs originally executed by that arm are automatically transferred to the other arm to ensure uninterrupted task execution.

[0048] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0049] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0050] Figure 1 This is a system block diagram of the present invention;

[0051] Figure 2 Illustrative example diagrams for generating modules for the target;

[0052] Figure 3 An illustrative example diagram of assigning modules to a target. Detailed Implementation

[0053] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0054] The present invention provides a framework for extending the actions of a single-arm agent to dual-arm operations, which is used to leverage datasets from single-arm operations to facilitate dual-arm operations. This invention proposes a novel framework comprising the following nine stages: 1) Constructing an attention extension module to obtain the coordinates of the first pickup point using visual information and natural language instructions; 2) Generating a mask for the corresponding object around the first pickup point using an object segmentation module; 3) Utilizing the attention extension module again to obtain the coordinates of the second pickup point based on visual data containing the mask of the first picked object and natural language instructions; 4) In the transport module, obtaining the coordinates of the first placement point by combining the initial visual data, natural language instructions, and the coordinates of the first pickup point; 5) Generating a mask for the placement area around the first placement point using the object segmentation module; 6) Utilizing the transport module again to obtain the coordinates of the second placement point based on visual data containing the mask of the first placement area, natural language instructions, and the coordinates of the second pickup point; 7) Constructing a state encoder to obtain encoding features about the target state diagram using the target state diagram; 8) Constructing a cost encoder to obtain encoding features about the cost matrix using the cost matrix; 9) Constructing a target allocation module, which utilizes the encoding features output by the state encoder and the cost encoder, and combines them with a motion planning library to achieve effective coordination, thereby allocating the target action to the corresponding robotic arm. This framework leverages a large number of readily available datasets of single-arm operations to facilitate two-arm operations, addressing the challenge of obtaining datasets for two-arm operations. Figure 1 This is a system block diagram of the present invention.

[0055] This embodiment involves four essential concepts, as follows:

[0056] The first concept: global visual information, which is a top-down RGB-D 3D reconstructed image captured by three noise-free cameras, each with a resolution of 640×480. Two cameras are located on the left and right sides of the table, respectively, and the remaining camera is located in front of the robotic arm. All three cameras are facing the center of the table.

[0057] The second concept: Natural language instructions. Natural language instructions are human-given instructions that are expected to enable the robotic arm to perform the tasks mentioned therein.

[0058] The third concept is the target state graph. The target state graph is constructed as a partially connected graph, reflecting the spatial relationships and task completion status within the current workspace. It consists of several target nodes and two warehouse nodes. Target nodes represent pick-up and place-down actions, with each node containing the object's pose in absolute Cartesian coordinates. Warehouse nodes represent the starting points of the two robotic arms.

[0059] The fourth concept: Cost Matrix. The cost matrix represents the collaborative cost of all potential operations in the current workspace state. These costs are influenced by goal-related information and the configuration of the two arms during operation execution.

[0060] Figure 2 An illustrative example diagram of the network generated for the target. Figure 3 An illustrative example diagram of the target assignment network. The system framework of this system mainly includes two components: the target generation network and the target assignment network. Figure 1 The diagram shown is a system block diagram of the present invention, wherein: Target Generation Network: The starting point of this network is to obtain the optimal pickup point coordinates of the target object by utilizing global visual information and natural language instructions, including the following steps:

[0061] Step 1: Input global visual information and natural language commands, and obtain the coordinates of the first object picking point through the attention extension module;

[0062] Step 2: Use the object segmentation module to obtain the mask of the first object;

[0063] Step 3: Based on the global visual information containing the mask of the first object and the natural language instructions, the coordinates of the picking point of the second object are obtained again using the attention extension module;

[0064] Step 4: Based on the global visual information, natural language instructions, and the coordinates of the first pick point, use the transport module to obtain the coordinates of the first placement point;

[0065] Step 5: Around the first placement point, use the object segmentation module again to generate a mask for the area of ​​the first placement point;

[0066] Step 6: Based on the global visual information including the mask of the first object placement area, the natural language command, and the coordinates of the second pick point, use the transport module again to obtain the coordinates of the second placement point.

[0067] Target Assignment Network: This network aims to assign the action pairs generated by the target generation network to two robotic arms with minimal execution cost, including the following steps:

[0068] Step 1: Input the target state diagram into the state encoder to obtain the encoded features of the target state diagram;

[0069] Step 2: Input the cost matrix into the cost encoder to obtain the encoded features of the cost matrix;

[0070] Step 3: Combine the encoded features output by the state encoder and the encoded features output by the cost encoder, and use the target assignment module to assign the target action pairs to the two robotic arms respectively.

[0071] In this embodiment, an attention extension module is constructed, which obtains a probability map of the optimal pickup point for the first object by fusing natural language instructions with visual data. The pixel with the highest probability value on the probability map is the optimal pickup point for that object. The coordinates of the first pickup point are obtained by the following formula:

[0072]

[0073] Among them, Q pick It is the action value function for the picking operation, and (u,v) are the pixel coordinates on the image.

[0074] It also includes: constructing an object segmentation module, which, based on the generated coordinates of the first pick point, crops a 64×64 pixel crop block centered on that point; this crop block can completely contain the first object; then, it uses SAM to obtain the mask of the first object, and then uses the global image containing the mask of the first object... m1 This serves as new visual information for use in subsequent processes.

[0075] Using natural language commands and the resulting global image. m1 The attention extension module is then used again to obtain a probability map of the optimal pickup point for the second object. Since the first object has already been processed by SAM, the attention extension module can focus its attention on the second object, thus obtaining the second pickup point more accurately. Similar to solving for the coordinates of the first pickup point, the pixel with the highest probability in the probability map is the optimal pickup point. The coordinates of the second pickup point are obtained by the following formula:

[0076]

[0077] A transport module is built. This module uses this data and the coordinates of the first pick point to generate a probability map about the first placement point. The pixel with the highest probability value in the probability map is the optimal placement point for the first object. The coordinates of the first placement point are obtained using the following formula:

[0078]

[0079] Among them, Q placeIt is the action value function for the placement operation, where Δτ is the pixel coordinates on the image.

[0080] Construct a SAM module that crops an 8×8 pixel region centered on the first placement point. Use SAM to generate a mask for this region to ensure the transport module focuses on generating the possible placement area for the second object. Then, generate a global image containing the mask of the first placement point region. m2 This new visual information is used to solve for the second placement point in the subsequent process.

[0081] In this embodiment, natural language instructions 1 and global image 0 are combined. m2 And the coordinates of the second pick point, then use the transport module again to obtain the probability map of the second placement point. The pixel with the highest probability value in the probability map is the optimal placement point for the second object; the coordinates of the second placement point are obtained by the following formula:

[0082]

[0083] A state encoder is constructed to capture the spatial relationship between the target and the robotic arms by processing observations in the target state graph. MLP and MHSA are used to embed the position data of the task and the robotic arms into the feature vector. The target nodes in the target state graph contain the corresponding object's action pose in the Cartesian coordinate system, i.e., the object's pick-up and placement coordinates, expressed by the following formula:

[0084]

[0085] T1 and T2 represent the coordinates of the pick-up and place-up operations, respectively; the warehouse node in the target state diagram contains the starting position of the corresponding robotic arm, expressed by the following formula:

[0086]

[0087] Arm1 and Arm2 represent the Cartesian coordinates of the two robotic arms, respectively.

[0088] Construct a cost encoder that collects execution costs from a cost matrix. As the robotic arm performs pick-and-place actions, the cost matrix dynamically changes with the evolving operational state. The cost matrix consists of a 2×2 grid, with each grid cell containing two types of costs: movement cost c. mv and transfer costs c tf Here, movement cost represents the distance the robotic arm travels from its current position to the target object; transfer cost represents the distance the target object is moved from the pick-up position to the placement position; the solutions for both costs are expressed by the following formulas:

[0089]

[0090] In cases where operations are not feasible, such as assigning the same task to two arms or assigning a task that has already been completed, the cost in each individual grid cell will be set to infinity. The cost matrix captures the path length from the robotic arm's starting position to the two action pairs and the corresponding transfer costs:

[0091]

[0092] Wherein, cost(T) i Arm j () is the total path length from the starting position to the pick-up position and then to the delivery position of the robotic arm; when a robotic arm encounters a malfunction, the action that should have been performed by this robotic arm will be seamlessly transferred to another robotic arm to ensure uninterrupted execution of the action.

[0093] A task allocation module is constructed, which uses an MHSA layer to fuse the collaborative information of the target node and the cost matrix. Then, an MLP layer and a probability maximization operation are applied to output the optimal task allocation strategy for both arms. The entire learning process optimizes the action allocation by minimizing the cross-entropy loss function.

[0094] The final output of the target assignment network is generated based on a cost matrix, where each task is assigned to a robotic arm with a lower execution cost. The output of the target assignment network is represented as follows:

[0095]

[0096] Where A(1) and A(2) represent two action pairs that have been assigned to the robotic arm Arm respectively. j and Arm k .

[0097] Experimental results show that the technical solution provided by this invention improves task efficiency while eliminating the cost of collecting specialized dual-arm data.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications should be covered within the scope of the claims of the present invention.

Claims

1. A method for extending the actions of a single-arm proxy to dual-arm operation, characterized in that: The method specifically includes the following steps: S1: Construct an attention extension module that uses global visual information o and natural language instructions l to obtain the first picking point; S2: Construct an object segmentation module to generate a mask for the first object based on the first pick point; S3: Based on the first object mask, the attention extension module is used again to generate the second pickup point; S4: Construct the transport module, which uses global visual information, natural language instructions, and the coordinates of the first pick point to obtain the coordinates of the first placement point; S5: Based on obtaining the coordinates of the first placement point, the object segmentation module is used again to generate a mask for the first placement area; S6: Based on obtaining the first placement area mask, the transport module is used again to generate the second placement point; S7: Construct a state encoder that uses the target state diagram to obtain the encoded features of the target state diagram; S8: Construct a cost encoder that uses the cost matrix to obtain the encoded features of the cost matrix; S9: Construct a target assignment module that utilizes the encoded features output by the state encoder and the encoded features output by the cost encoder, and combines them with the motion planning library to achieve effective coordination, thereby assigning the target action to the corresponding robotic arm. In step S1, an attention extension module is constructed, which obtains a probability map of the optimal pickup point for the first object by fusing natural language instructions with visual data. The pixel with the highest probability value on the probability map is the optimal pickup point for that object. The coordinates of the first pickup point are obtained by the following formula: Among them, Q pick It is the action value function for the picking operation, and (u,v) are the pixel coordinates on the image; In step S2, an object segmentation module is constructed. Based on the generated coordinates of the first picking point, a 64×64 pixel crop block is cropped centered on that point. This crop block can completely contain the first object. Then, the mask of the first object is obtained using SAM, and the global image containing the mask of the first object is generated. m1 As new visual information for use in subsequent processes; In step S7, a state encoder is constructed, which captures the spatial relationship between the target and the robotic arms by processing observations in the target state diagram. MLP and MHSA are used to embed the position data of the task and the robotic arms into the feature vector. The target node in the target state diagram contains the corresponding object's action pose in the Cartesian coordinate system, i.e., the object's pick-up and placement coordinates, expressed by the following formula: T1 and T2 represent the coordinates of the pick-up and place-up operations, respectively; the warehouse node in the target state diagram contains the starting position of the corresponding robotic arm, expressed by the following formula: Arm1 and Arm2 represent the Cartesian coordinates of the two robotic arms, respectively. In step S8, a cost encoder is constructed, which collects execution costs from the cost matrix. As the robotic arm performs pick-up and place-down actions, the cost matrix dynamically changes with the constantly changing working state. The cost matrix consists of a 2×2 grid, and each grid cell contains two types of costs: movement cost c. mv and transfer costs c tf Here, movement cost represents the distance the robotic arm travels from its current position to the target object; transfer cost represents the distance the target object is moved from the pick-up position to the placement position; the solutions for both costs are expressed by the following formulas: c mv =||Arm k -T i pk ||,c tf =||T i pk -T i pl || The cost matrix captures the path length from the robot arm's starting position to the two action pairs and the corresponding transfer costs: Wherein, cost(T) i Arm j () is the total path length from the starting position to the pick-up position and then to the delivery position of the robotic arm; when a robotic arm encounters a malfunction, the action that should have been performed by this robotic arm will be seamlessly transferred to another robotic arm to ensure uninterrupted execution of the action; In step S9, a task allocation module is constructed, which uses an MHSA layer to fuse the collaborative information of the target node and the cost matrix, and then applies an MLP layer and a probability maximization operation to output the optimal task allocation strategy for both arms; the entire learning process optimizes the allocation of actions by minimizing the cross-entropy loss function. The final output of the target assignment network is generated based on a cost matrix, where each task is assigned to a robotic arm with a lower execution cost. The output of the target assignment network is represented as follows: Where A(1) and A(2) represent two action pairs that have been assigned to the robotic arm Arm respectively. j and Arm k .

2. A method for extending single-arm proxy actions to dual-arm operations according to claim 1, characterized in that: In step S3, the natural language instruction l and the global image o obtained in step S2 are used. m1 The attention extension module is then used again to obtain a probability map of the optimal pickup point for the second object. Since the first object has already been processed by SAM, the attention extension module can focus its attention on the second object, thus obtaining the second pickup point more accurately. Similar to solving for the coordinates of the first pickup point, the pixel with the highest probability in the probability map is the optimal pickup point. The coordinates of the second pickup point are obtained by the following formula: In step S4, a transport module is constructed. The input visual information and natural language commands are consistent with those input in step S1. The transport module uses this data and combines it with the coordinates of the first pick point to generate a probability map about the first placement point. The pixel with the highest probability value in the probability map is the optimal placement point for the first object. The coordinates of the first placement point are obtained by the following formula: Among them, Q place It is the action value function for the placement operation, where Δτ is the pixel coordinates on the image.

3. A method for extending single-arm proxy actions to dual-arm operations according to claim 2, characterized in that: In step S5, a SAM module is constructed to crop an 8×8 pixel area centered on the first placement point. A mask is generated for this area using SAM to ensure that the transport module focuses on generating the possible placement area for the second object. The global image containing the mask of the first placement point area is then processed. m2 This new visual information is used to solve for the second placement point in the subsequent process.

4. A method for extending single-arm proxy actions to dual-arm operations according to claim 3, characterized in that: In step S6, the natural language instruction l and the global image o obtained in step S5 are combined. m2 In addition to the coordinates of the second pickup point obtained in step S3, the transport module is used again to obtain a probability map of the second placement point. The pixel with the highest probability value in the probability map is the optimal placement point for the second object. The coordinates of the second placement point are obtained by the following formula:

5. A system for extending the actions of a single-arm proxy to dual-arm operation, characterized in that: The system employs the method described in any one of claims 1 to 4.

Images