Multi-mechanical-arm collaborative long-view task analysis method for industrial edge

Abstract

The embodiment of the invention relates to an industrial-edge-oriented multi-mechanical-arm collaborative long-view task analysis method, device, equipment and medium, and the method comprises the steps: carrying out the multi-dimensional decomposition of original perception input data based on a fused visual representation segmentation algorithm, a global optical flow segmentation algorithm and a local motion information segmentation algorithm, clustering all the sub-tasks in the joint feature space based on a density spatial clustering algorithm to obtain sub-task segments with sub-task boundaries; performing structured representation on each sub-task fragment based on a structured semantic model, and constructing a dependency graph of the to-be-executed task according to the structured representation; sub-tasks of at least one mechanical arm are distributed according to the dependency graph of the to-be-executed task, the at least one mechanical arm is controlled to cooperatively execute all the sub-tasks according to demonstration trajectories, all the demonstration trajectories are obtained from a trajectory sharing library, all the sub-tasks are subsets of long-view tasks, and all the sub-tasks are subsets of the to-be-executed tasks. The invention aims to realize systematic transition from single mechanical arm demonstration to cooperative execution in a long-view task.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of multi-robot collaborative long-horizon task parsing method for industrial edge, and particularly relate to a multi-robot collaborative long-horizon task parsing method for industrial edge. BACKGROUND

[0002] With the rapid development of mechanical arms, Internet of Things and large language models, multi-robot systems with autonomous perception, decision-making and control capabilities are increasingly deployed in practical scenarios. Such systems have been widely used in production and manufacturing, warehousing and logistics, and home services. As the complexity of tasks increases, many practical tasks often have long time series, multiple stages, and subtasks that depend on each other. Such tasks are often referred to as "long-horizon tasks". Such characteristics pose a major challenge to robot learning, making task understanding more difficult and making execution planning more complex. Therefore, how to automatically decompose long-horizon tasks, organize subtasks based on time series structure, and implement collaborative execution in multi-robot systems has become a core challenge.

[0003] To address this challenge, there are two main directions in existing research: end-to-end learning and demonstration learning. End-to-end learning aims to directly map raw perception inputs to action outputs, usually implemented through a reinforcement learning framework, and optimized for task performance through interaction with the environment. For example, some methods decompose tasks into multiple stages and further divide each stage into atomic actions, thereby increasing reward density and enhancing training stability and efficiency. However, this type of method relies on artificially defined stage boundaries and is difficult to adapt to tasks with variable or flexible structures. On the other hand, demonstration learning is widely recognized as an intuitive and efficient learning paradigm, enabling robots to acquire task execution capabilities by imitating demonstrations from other agents or humans. Representative methods determine subtask boundaries by detecting peaks in visual feature changes, thereby achieving visual-based task decomposition. Although it has some universality, relying on a single dimension of visual signal often leads to inaccurate boundary detection, which in turn affects downstream policy learning. In addition, most existing methods mainly focus on single-robot execution and have not been extended to multi-robot collaboration scenarios, limiting their applicability in complex multi-robot systems. SUMMARY

[0004] Therefore, embodiments of the present application propose a multi-robot collaborative long-horizon task parsing method for industrial edge, aiming to achieve a systematic transition from single-robot demonstration to collaborative execution in long-horizon tasks.

[0005] To achieve the above object, the embodiment of the present application provides a multi-robot arm cooperative long-view task analysis method for an industrial edge, comprising: based on a fused visual representation segmentation algorithm, a global optical flow segmentation algorithm and a local motion information segmentation algorithm, multi-dimensional decomposition is performed on original perception input data to obtain corresponding subtasks, a density-based spatial clustering algorithm is used to cluster all the subtasks in a joint feature space to obtain subtask segments with subtask boundaries; a structured semantic model is used to structure each subtask segment, and a dependency graph of a to-be-executed task is constructed according to the structured representation; subtask allocation is performed for at least one robot arm according to the dependency graph of the to-be-executed task, and at least one robot arm is controlled to cooperatively execute each subtask according to a demonstration trajectory, wherein each demonstration trajectory is obtained from a trajectory sharing library, and each subtask is a subset of a long-view task.

[0006] Optionally, the multi-dimensional decomposition of the original perception input data based on the fused visual representation segmentation algorithm, the global optical flow segmentation algorithm and the local motion information segmentation algorithm comprises: a visual embedding encoder is used to extract a visual state vector of the original perception input data, and clustering analysis is used to determine all potential task boundaries, and a first candidate set is obtained according to the summary of all the task boundaries; an optical flow tensor of adjacent frames of the original perception input data is calculated; a pixel-level amplitude of each optical flow tensor is calculated; time sliding processing is performed on the pixel-level amplitude to obtain a pixel-level amplitude sequence, and a local maximum value in the pixel-level amplitude sequence is selected as a candidate segmentation point, and a second candidate set is obtained according to the summary of all the candidate segmentation points; a target segmentation mask of the original perception input data is obtained through a pre-trained target segmentation model, and a mask frame focusing on a target object is obtained through mask processing on a frame image with the target segmentation mask; a plurality of segmentation points are obtained according to the light flow disturbance in the mask frame, and a third candidate set is obtained according to the segmentation points.

[0007] Optionally, the structured representation comprises an action name, a precondition and an execution effect; the dependency graph of the to-be-executed task is constructed according to the structured representation, comprising: obtaining a natural language description corresponding to each subtask segment; using a large language model to convert the natural language description into a structured representation containing an action name, a precondition and an execution effect; constructing a dependent action relationship based on the logical relationship between the preconditions and the execution effects, wherein if the execution effect of action A contains the precondition of action B, action B depends on action A, and A→B represents a directed edge from action A to action B; if the preconditions and effects of two actions have no intersection and no conflict dependence, the two actions are determined as concurrent action relationship; the task dependency graph is determined based on the directed edge and the concurrent action relationship.

[0008] Optionally, the demonstration trajectory includes the position sequence and orientation quaternion sequence of the end effector: the sub-task allocation of at least one robotic arm according to the dependency graph of the task to be executed, and the control of at least one robotic arm to collaboratively execute each sub-task according to the demonstration trajectory, includes: obtaining the orientation quaternion of the gripper in the global coordinate system at each time step; calculating the base orientation quaternion and the base orientation inverse quaternion of the target robotic arm while keeping the position sequence of the end effector unchanged; and obtaining the base orientation quaternion in the local coordinate system of each target robotic arm by left-multiplying the base orientation quaternion by the base orientation inverse quaternion.

[0009] Optionally, the expression for obtaining the target segmentation mask of the original perceptual input data through the pre-trained target segmentation model, and performing masking processing on the frame image with the target segmentation mask to obtain the mask frame focused on the target object includes:

[0010] Indicates at time Focus on the mask frame of the target object. Fill the background area. Represents the target segmentation mask. Indicates time The original frame.

[0011] Optionally, the step of extracting the visual state vector of the original perceptual input data using a visual embedding encoder and determining all potential task boundaries through cluster analysis, and obtaining a first candidate set based on the sum of all task boundaries, includes: constructing a time-series embedding based on the embedding of each frame of the original perceptual input data, wherein the time-series embedding is the visual state vector extracted by the visual embedding encoder; using a hierarchical clustering algorithm to divide frames that are temporally adjacent and have similar motion features into the same cluster, and determining candidate segmentation points based on the boundaries of the clusters; and determining the first candidate set based on the boundaries between adjacent clusters.

[0012] Optionally, the expression for determining the first candidate set based on the boundary between adjacent clusters is: z k = max( C k ) = min( C k+1 ): in, C k Indicates the first k Clusters, C k+1 Indicates the first k +1 cluster.

[0013] To achieve the above object, the embodiment of the present application also provides an industrial edge-oriented multi-robot collaborative long-view task analysis device, comprising: a task decomposition module, configured to perform multi-dimensional decomposition on original perception input data based on a fused visual representation segmentation algorithm, a global optical flow segmentation algorithm and a local motion information segmentation algorithm to obtain corresponding subtasks, and perform clustering on all the subtasks in a joint feature space based on a density-based spatial clustering algorithm to obtain subtask segments with subtask boundaries; a task dependency modeling module, configured to perform structured representation on each subtask segment based on a structured semantic model, and construct a dependency graph of a to-be-executed task according to the structured representation; and a collaborative execution module, configured to perform subtask allocation of at least one robot according to the dependency graph of the to-be-executed task, and control the at least one robot to collaboratively execute each subtask according to a demonstration trajectory, wherein each demonstration trajectory is obtained from a trajectory sharing library, and each subtask is a subset of a long-view task.

[0014] To achieve the above object, the embodiment of the present application also provides a server, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the industrial edge-oriented multi-robot collaborative long-view task analysis method.

[0015] To achieve the above object, the embodiment of the present application also provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the industrial edge-oriented multi-robot collaborative long-view task analysis method.

[0016] The industrial edge-oriented multi-robot collaborative long-view task analysis method, device, equipment and medium provided by the embodiment of the present application comprise the following steps: performing multi-dimensional decomposition on original perception input data based on a fused visual representation segmentation algorithm, a global optical flow segmentation algorithm and a local motion information segmentation algorithm to obtain corresponding subtasks, performing clustering on all the subtasks in a joint feature space based on a density-based spatial clustering algorithm to obtain subtask segments with subtask boundaries, combining multi-dimensional information of semantic clustering, global optical flow and local disturbance to decompose a long-view task video, and aggregating segmentation boundaries by using the density-based spatial clustering algorithm to obtain candidate subtask segments, thereby avoiding inaccuracy caused by a single feature.

[0017] The structured semantic model is used for structured representation of each sub-task fragment, a dependency graph of the to-be-executed task is constructed according to the structured representation, a natural language description corresponding to the task segmentation is converted into a structured action representation by the large language model, the action representation includes an action name, a precondition and an execution effect, and a task dependency graph is constructed based on the logical relationship between the precondition and the execution effect, so as to identify the sequence relationship and parallel relationship between the sub-tasks, and realize the executable multi-robot task scheduling.

[0018] The sub-task distribution of at least one robot is performed according to the dependency graph of the to-be-executed task, and each sub-task is a subset of the long-view task, and the direction information in the demonstration trajectory is converted into an adjustment direction quaternion in the local coordinate system of the target robot, so as to avoid the inconsistency of the direction caused by the difference in robot deployment, and ensure the cooperative execution efficiency of the multi-robot in the global space. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of a multi-robot cooperative long-view task analysis method for an industrial edge provided in an embodiment of the present application; Figure 2 is an executable task dependency graph of a multi-robot cooperative long-view task analysis method for an industrial edge provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to make the readers better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present application, and the embodiments can be combined and referenced with each other without contradiction.

[0021] An embodiment of the present application proposes a multi-robot cooperative long-view task analysis method for an industrial edge, which is applied to an electronic device. The electronic device can be a terminal or a server. In the embodiments of the present application and the following embodiments, the server is taken as an example for description. The implementation details of the multi-robot cooperative long-view task analysis method for an industrial edge proposed in the embodiment will be described in detail below. The following content only provides implementation details for easy understanding, and is not necessary for implementing the present solution.

[0022] In industrial scenarios, long-horizon and structurally complex tasks are common, but traditional rule-based segmentation methods often lack robustness and accuracy. To address these limitations, the invention proposes a multi-dimensional decomposition method that utilizes visual features, global optical flow, and local motion cues. This method integrates information from multiple dimensions, including visual features, global optical flow, and local perturbations. Finally, an aggregation mechanism is employed to generate task boundaries that are both temporally consistent and reflect the underlying task structure, enabling accurate identification of subtasks in long-horizon tasks.

[0023] In the prior art, multi-robot systems with autonomous perception, decision-making, and control capabilities are increasingly being applied in real-world industrial scenarios. With the development of robotic technology, long-horizon tasks (long duration, complex subtask dependencies) pose higher requirements for perception and coordination, especially in multi-robot systems. Existing methods typically rely on single-source data or predefined task structures, making it difficult to adapt to task complexity and deployment heterogeneity, thereby limiting the transition from single-robot demonstrations to multi-robot collaboration.

[0024] To address this issue, the present application proposes a multi-robot collaboration method that is based on long-horizon task demonstrations and systematically addresses task segmentation, subtask modeling, and deployment perception execution, thereby enabling multi-robot collaboration. Specifically, the invention integrates visual features, global optical flow, and local motion information to achieve accurate subtask boundary detection. Subtasks are represented through structured prerequisites and impacts, enabling the automatic construction of a task dependency graph for sequential or parallel execution. Trajectory adaptation strategies adjust the orientation of the robot through coordinate transformation to adapt to different deployments.

[0025] Further, the application proposes a multi-robot arm cooperation method for long-view tasks, which realizes the systematic transition from single-robot arm demonstration to cooperative execution. The method supports three key stages in the multi-robot arm system, namely task segmentation, sub-task allocation and cooperative execution, aiming to bridge the gap between demonstration and cooperative execution through targeted module design. First, to improve the accuracy and robustness of task segmentation, the application proposes a multi-dimensional segmentation method that integrates visual feature segmentation algorithms, global optical flow segmentation algorithms and local motion information segmentation algorithms, and performs clustering in the joint feature space to obtain stable and accurate sub-task boundaries. Second, to explore the implicit temporal and logical dependency between sub-tasks, the application converts each sub-task segment into a structured representation containing preconditions and effects, and builds a sub-task dependency graph based on this, enabling the multi-robot arm system to distinguish between sequential and parallel execution relationships. Finally, to solve the execution failure problem in the multi-robot arm system caused by position differences, the application proposes a trajectory adaptation strategy based on deployment, which adjusts the demonstration trajectory based on the coordinate transformation of the position of each robot arm, aligns the direction consistency between different robot arms while keeping the position of the end effector unchanged, and ensures the cooperative and efficient execution of the multi-robot arm system.

[0026] The specific process of the multi-robot arm cooperative long-view task analysis method for industrial edges proposed in this embodiment can be as shown in Figure 1 The specific process of the multi-robot arm cooperative long-view task analysis method for industrial edges proposed in this embodiment can be as shown in Step 101, based on the fused visual feature segmentation algorithm, global optical flow segmentation algorithm and local motion information segmentation algorithm, the original perception input data is multi-dimensionally decomposed to obtain the corresponding sub-tasks, and the density-based spatial clustering algorithm is used in the joint feature space to cluster all sub-tasks to obtain sub-task segments with sub-task boundaries.

[0027] Specifically, the multi-robot arm cooperative long-view task analysis method for industrial edges.

[0028] In a specific implementation, the execution process of step S101 can be represented as follows: S1011, a visual embedding encoder is used to extract the visual state vector of the original perception input data, and clustering analysis is used to determine all potential task boundaries, and a first candidate set is obtained based on the summary of all task boundaries; In an embodiment of the present application, a visual embedding encoder is used to extract the visual state vector of the original perception input data, and clustering analysis is used to determine all potential task boundaries, and the execution process of obtaining the first candidate set based on the summary of all task boundaries can include the following: A time series embedding is constructed according to the embedding of each frame of the original perception input data, wherein the time series embedding is the visual state vector extracted by the visual embedding encoder; A hierarchical clustering algorithm is used to divide temporally adjacent frames with similar motion features into the same cluster, and candidate segmentation points are determined based on the cluster boundaries; The first candidate set is determined based on the boundaries between adjacent clusters.

[0029] Specifically, to achieve coarse-grained task segmentation based on visual dynamics, this invention employs a visual embedding encoder to extract visual state vectors and determines potential task boundaries through cluster analysis. First, the embedding representation of each frame is denoted as... e t This forms a time series embedding E = { e 1, e 2, ..., e T}, which is the visual state vector. Secondly, a hierarchical clustering algorithm is used to group temporally adjacent frames with similar motion features into the same cluster, and the boundaries of these clusters are calculated as candidate segmentation points; finally, the boundaries between adjacent clusters are defined as... z k = max( C k ) = min( C k+1 ),in, C k Indicates the first k Clusters, C k+1 Indicates the first k +1 clusters, and aggregate all boundaries to form the first candidate set. .

[0030] Clearly, this method can achieve coarse-grained action segmentation based on the aggregation of temporal features, but its accuracy in capturing motion changes during continuous actions is limited.

[0031] S1012. Calculate the optical flow tensor of adjacent frames of the original sensor input data; S1013. Calculate the pixel-level magnitude of each optical flow tensor; S1014. Perform time-sliding processing on the pixel-level amplitude to obtain a pixel-level amplitude sequence, and select the local maxima in the pixel-level amplitude sequence as candidate segmentation points. Based on all the candidate segmentation points after aggregation, obtain the second candidate set. Since the motion changes during task execution are mainly driven by physical changes such as displacement and operation actions, this application introduces optical flow amplitude as another criterion for segmentation determination. This segmentation method is completed by the processor through steps S1012-S1014.

[0032] The above steps can specifically include: First, for any adjacent frames ( o t, o t+1 Calculate its optical flow tensor F t→ t+1 Secondly, for each optical flow tensor F Calculate pixel-level amplitude M x,y = || F x,y | |, where ( x, y ( ) represents the image plane coordinates. The average optical flow magnitude of a frame is defined as... ,in, H and W These represent the height and width of the frame, respectively. Finally, a sequence is constructed by time sliding. }, and select local maxima as candidate split points, finally summarizing them into a second candidate set. Clearly, this method is highly sensitive to large-amplitude changes in motion (such as reaching or grasping), and is suitable for detecting the boundaries of highly manipulative actions.

[0033] Understandably, in certain task phases, background motion may obscure subtle but crucial action transitions. To address this issue, this invention introduces a target region-based masking process: S1015. Obtain the target segmentation mask of the original perceptual input data through the pre-trained target segmentation model, perform mask processing on the frame image with the target segmentation mask, and obtain the mask frame focused on the target object. S1016. Based on the optical flow perturbation within the mask frame region, multiple segmentation points are obtained, and a third candidate set is obtained based on each segmentation point.

[0034] For example, a target segmentation mask is obtained through a pre-trained target segmentation model. The target frame is obtained by masking the frame image. : . in, Indicates at time Focus on the mask frame of the target object. The background area is filled. By analyzing the optical flow perturbations within the target area, the processor can precisely locate fine-grained operations between the robotic arm and the interactive object, thereby obtaining more accurate segmentation points and summarizing them into a candidate set. .

[0035] It should be noted that long-field-of-view and structurally complex tasks are common in industrial scenarios, but traditional rule-based segmentation methods often suffer from shortcomings in robustness and accuracy. To address these limitations, this invention proposes a multi-dimensional decomposition method that integrates information from multiple dimensions, including visual features, global optical flow, and local motion cues. Finally, an aggregation mechanism is employed to generate task boundaries that are both temporally consistent and reflect the underlying task structure, thereby achieving accurate identification of subtasks within long-field-of-view tasks.

[0036] Step 102: Based on the structured semantic model, perform structured representation of each subtask fragment, and construct the dependency graph of the task to be executed based on the structured representation.

[0037] In one embodiment of this application, the structured representation includes an action name, preconditions, and execution effect; the execution process of step S102 can be represented as follows: S1021. Obtain the natural language descriptions corresponding to each subtask segment; S1022. Use a large language model to convert the natural language description into a structured representation that includes action name, preconditions and execution effect; S1023. Construct a dependent action relationship based on the logical relationship between preconditions and execution effects. If the execution effect of action A includes the precondition of action B, then action B depends on action A, and A→B represents the directed edge from action A to action B. S1024. If the preconditions and effects of two actions do not overlap and there is no conflicting dependency, then the two actions are determined to be parallel actions. S1025. Based on directed edges and parallel action relationships, determine the task dependency graph.

[0038] Specifically, in long-field-of-view robotic arm operations, accurately identifying the temporal and logical dependencies between subtasks is crucial for efficient execution. To this end, this invention proposes a structured semantic model to describe the preconditions and execution effects of each subtask. This model can identify the sequential and parallel relationships between subtasks and supports the construction of an executable task dependency graph, serving as the foundation for multi-robotic arm collaboration, such as... Figure 2 As shown.

[0039] After performing time segmentation on a long-view task, this invention associates each subtask segment with a natural language description, which can be obtained through manual annotation or an automatically generated model.

[0040] Based on the above description, this application utilizes a Large Language Model (LLM) to convert each subtask into a structured action representation. The structure is as follows: .

[0041] Furthermore, to ensure output consistency, this application designs a prompt template containing multiple examples and provides contextual examples to guide LLM in generating action structure definitions with a uniform format and clear logic for subsequent analysis.

[0042] Furthermore, after obtaining the structured definitions of all subtasks, this application analyzes the... and Derive the task dependency graph by considering the logical relationships between sets. Specifically, if actions... The effect set is that a certain predicate appears in the action If the preconditions are set, then the action is considered. Dependent on action and with directed edges Conversely, if there is no overlap between the preconditions and effects of two actions, and there is no conflicting dependency, then they are determined to be able to be executed in parallel.

[0043] The resulting task dependency graph supports topological sorting to determine the execution order and enables multi-arm task allocation based on dependencies. Parallelizable subtasks are assigned to different robotic arms for concurrent execution, while actions with sequential dependencies are assigned to the same or collaborative robotic arms and executed sequentially to ensure task correctness.

[0044] Clearly, the structured representation is a hierarchical structure. This hierarchical structure relies on modeling methods to form the core structural foundation for multi-robot collaboration and provides the necessary support for subtask allocation.

[0045] Step 103: Assign subtasks to at least one robotic arm based on the dependency graph of the task to be executed, and control at least one robotic arm to collaboratively execute each subtask according to the demonstration trajectory. Each demonstration trajectory is obtained from the trajectory sharing library, and each subtask is a subset of the long field of view task.

[0046] Specifically, based on the task dependency graph, subtasks are assigned to multiple robotic arms for execution. Each robotic arm retrieves a motion template corresponding to its assigned subtask from a shared library. The motion template includes a trajectory obtained from the demonstration, which is represented by a sequence of position and orientation quaternions of the end effector.

[0047] Since the actual deployment environment of each robotic arm may differ from the original demonstration environment, directly executing the same template trajectory on all robotic arms may lead to spatial inconsistencies. Especially when the base orientations of the robotic arms differ, even if the manipulated object remains stationary in the global coordinate system, the resolution of the end effector orientation may still be biased. To address this issue, a deployment-based quaternion adjustment method is proposed, which effectively adjusts the orientation information while keeping the end effector position unchanged.

[0048] Specifically, for each subtask, a time step is set. The direction quaternion of the gripper in the global coordinate system at any given time is Introducing the base orientation quaternion of the target robotic arm And calculate its inverse quaternion. The adjusted directional quaternion is obtained by left multiplication, as shown in the following formula:

[0049] in, To represent quaternion multiplication, This represents the adjusted orientation quaternion in the local coordinate system of the target robotic arm. The inverse of the unit quaternion is obtained by conjugate calculation. For the unit quaternion... Its inverse quaternion representation is:

[0050] Using the above method, the positional information of the trajectory remains unchanged, and only the orientation information is adjusted through geometric transformation. This method is computationally simple and highly efficient, and is suitable for task scenarios where the manipulated object remains fixed in global space. Therefore, each robotic arm can execute its assigned sub-tasks based on the adjusted trajectory, thereby achieving coordinated and efficient execution of long-field-of-view tasks.

[0051] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0052] In summary, this invention proposes a multi-robot collaborative method for long-field-of-view tasks, achieving a systematic transition from single-robot demonstration to collaborative execution. This method supports three key stages in multi-robot systems: task segmentation, subtask allocation, and collaborative execution, aiming to bridge the gap between demonstration and collaborative execution through targeted module design. First, to improve the accuracy and robustness of task segmentation, this invention proposes a multi-dimensional segmentation method that integrates visual features, global optical flow, and local motion information, and performs clustering in a joint feature space to obtain stable and accurate subtask boundaries. Second, to uncover the implicit temporal and logical dependencies between subtasks, this invention transforms each subtask fragment into a structured representation containing preconditions and effects, and constructs a subtask dependency graph based on this, enabling the multi-robot system to distinguish between sequential and parallel execution relationships. Finally, to address the execution failure problem caused by positional differences in multi-robot systems, this invention proposes a deployment-based trajectory adaptation strategy. This strategy adjusts the demonstration trajectory based on the coordinate transformation of each robotic arm's orientation while keeping the end effector position unchanged, thereby achieving directional alignment between different robotic arms and ensuring the coordinated and efficient execution of the multi-robotic arm system.

[0053] Based on the above embodiments, this application also provides a multi-robotic arm collaborative long-field-of-view task parsing device for industrial edge computing, including a task decomposition module, a task dependency modeling module, and a collaborative execution module. The task decomposition module performs multi-dimensional decomposition on the original perceptual input data based on a fused visual representation segmentation algorithm, a global optical flow segmentation algorithm, and a local motion information segmentation algorithm to obtain corresponding sub-tasks. In the joint feature space, a density-based spatial clustering algorithm is used to cluster all sub-tasks to obtain sub-task fragments with sub-task boundaries. The task dependency modeling module performs structured representation of each sub-task fragment based on a structured semantic model and constructs a dependency graph of the task to be executed based on the structured representation. The collaborative execution module allocates sub-tasks to at least one robotic arm according to the dependency graph of the task to be executed and controls at least one robotic arm to collaboratively execute each sub-task according to a demonstration trajectory. Each demonstration trajectory is obtained from a trajectory sharing library, and each sub-task is a subset of long-field-of-view tasks.

[0054] It is not difficult to see that this embodiment is a device embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still effective in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0055] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units do not exist in this embodiment.

[0056] Another embodiment of this application proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge computing described in the above method embodiments.

[0057] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0058] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0059] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0060] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disks, or optical disks.

[0061] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for parsing multi-robotic arm collaborative long-field-of-view tasks for industrial edge computing, characterized in that, include: After multidimensional decomposition of the original perceptual input data based on the fused visual representation segmentation algorithm, global optical flow segmentation algorithm and local motion information segmentation algorithm, the corresponding sub-tasks are obtained. In the joint feature space, the density-based spatial clustering algorithm is used to cluster all sub-tasks to obtain sub-task fragments with sub-task boundaries. The structured semantic model is used to represent each subtask fragment in a structured way, and the dependency graph of the task to be executed is constructed based on the structured representation. At least one robotic arm is assigned a subtask based on the dependency graph of the task to be executed. At least one robotic arm is controlled to collaboratively execute each subtask based on the demonstration trajectory. Each demonstration trajectory is obtained from the trajectory sharing library, and each subtask is a subset of the long field of view task.

2. The multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge computing as described in claim 1, characterized in that, The algorithm based on the fused visual representation segmentation algorithm, the global optical flow segmentation algorithm, and the local motion information segmentation algorithm performs multidimensional decomposition of the original perceptual input data, including: A visual state vector of the original perceptual input data is extracted using a visual embedding encoder, and all potential task boundaries are determined through cluster analysis. The first candidate set is obtained by summarizing all task boundaries. Calculate the optical flow tensor of adjacent frames of the original sensor input data; Calculate the pixel-level magnitude of each optical flow tensor; The pixel-level amplitudes are processed by time-sliding to obtain a pixel-level amplitude sequence. Local maxima in the pixel-level amplitude sequence are selected as candidate segmentation points. Based on all the candidate segmentation points after aggregation, a second candidate set is obtained. The target segmentation mask of the original perceptual input data is obtained by pre-training the target segmentation model. The frame image with the target segmentation mask is then processed to obtain the mask frame focused on the target object. Based on the optical flow perturbation within the mask frame region, multiple segmentation points are obtained, and a third candidate set is obtained based on each segmentation point.

3. The multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge computing as described in claim 2, characterized in that, The structured representation includes the action name, preconditions, and execution effect; The step of constructing the dependency graph of the task to be executed based on the structured representation includes: Obtain the natural language descriptions corresponding to each subtask segment; The natural language description is transformed into a structured representation containing action name, preconditions, and execution effect using a large language model; Dependent action relationships are constructed based on the logical relationship between preconditions and execution effects. If the execution effect of action A includes the precondition of action B, then action B depends on action A, and A→B represents the directed edge from action A to action B. If the preconditions and effects of two actions do not overlap and there is no conflicting dependency, then the two actions are determined to be parallel actions. Based on directed edges and parallel action relationships, the task dependency graph is determined.

4. The multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge computing as described in claim 1, characterized in that, The demonstration trajectory includes a position sequence and an orientation quaternion sequence representation of the end effector of the robotic arm: The step of allocating sub-tasks to at least one robotic arm based on the dependency graph of the task to be executed, and controlling at least one robotic arm to collaboratively execute each sub-task according to the demonstration trajectory, includes: Obtain the orientation quaternion of the robotic arm's gripper in the global coordinate system at each time step; Calculate the base direction quaternion and the base direction inverse quaternion of the target robotic arm while keeping the position sequence of the end effector unchanged; By left-multiplying the base direction quaternion by the base direction inverse quaternion, the base direction quaternion in the local coordinate system of each target robot arm is obtained.

5. The multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge computing as described in claim 2, characterized in that, The target segmentation mask obtained from the original perceptual input data through the pre-trained target segmentation model, and the masking process performed on the frame image with the target segmentation mask to obtain the expression of the mask frame focused on the target object, include: in, Indicates at time Focus on the mask frame of the target object. Fill the background area. Represents the target segmentation mask. Indicates time The original frame.

6. The multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge computing as described in claim 2, characterized in that, The process involves using a visual embedded encoder to extract the visual state vector from the original perceptual input data, and then determining all potential task boundaries through cluster analysis. Based on the aggregation of all task boundaries, a first candidate set is obtained, including: A time-series embedding is constructed based on the embedding of each frame of the original perceptual input data, where the time-series embedding is a visual state vector extracted by the visual embedding encoder. A hierarchical clustering algorithm is used to divide temporally adjacent frames with similar motion features into the same cluster, and candidate segmentation points are determined based on the cluster boundaries; The first candidate set is determined based on the boundaries between adjacent clusters.

7. The multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge computing as described in claim 6, characterized in that, The expression for determining the first candidate set based on the boundary between adjacent clusters is: z k = max( C k ) = min( C k+1 ): in, C k Indicates the first k Clusters, C k+1 Indicates the first k +1 cluster.

8. A multi-robotic arm collaborative long-field-of-view task analysis device for industrial edge applications, characterized in that, include: The task decomposition module is used to perform multidimensional decomposition of the original perceptual input data based on the fused visual representation segmentation algorithm, global optical flow segmentation algorithm and local motion information segmentation algorithm to obtain the corresponding sub-tasks. In the joint feature space, the density-based spatial clustering algorithm is used to cluster all sub-tasks to obtain sub-task fragments with sub-task boundaries. The task dependency modeling module is used to represent each subtask fragment in a structured manner based on a structured semantic model, and to construct a dependency graph of the task to be executed based on the structured representation. The collaborative execution module is used to allocate subtasks to at least one robotic arm based on the dependency graph of the task to be executed, and to control at least one robotic arm to collaboratively execute each subtask according to the demonstration trajectory. Each demonstration trajectory is obtained from the trajectory sharing library, and each subtask is a subset of the long field of view task.

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can realize the multi-robotic arm collaborative long-field-of-view task parsing method for industrial edge as described in any one of claims 1 to 7.

Images