Mechanical arm heterogeneous task cooperative control method based on graph neural network
By using graph neural networks and spectral domain modulation methods, a task relationship graph is constructed and feature decomposition is performed to generate a collaborative control representation. This solves the stability problem of collaborative control of robotic arms in heterogeneous task environments and enables efficient and stable collaborative execution of robotic arms under complex tasks.
Patent Information
- Application Number
- CN202511738997.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-01-09
AI Technical Summary
Existing robotic arm control methods struggle to effectively unify the modeling of temporal constraints, resource competition, and collaborative relationships among heterogeneous tasks, resulting in insufficient collaborative control capabilities. Furthermore, they lack frequency domain analysis and adaptive update capabilities, making it difficult to achieve stable collaborative control in complex task environments.
By employing graph neural networks and spectral domain modulation, a task relationship graph is constructed, and Laplacian matrix eigenvalue decomposition is performed to generate a spatial domain task embedding representation. An improved neural Laplacian operator is used for frequency dimension weighting and adjustment. Combined with frequency domain energy modulation and inverse spectral domain transformation, a time domain collaborative control representation is generated, and finally, the robotic arm motion control quantity is output. The control strategy is then optimized through adaptive updates.
It achieves stable collaborative control of the robotic arm in complex and heterogeneous task environments, with continuous movements, smooth posture changes, reduced conflict behaviors, improved execution quality, and meets the requirements of high flexibility and high reliability in intelligent manufacturing.
Smart Images
Figure CN121290435A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent control technology, and in particular to a method for collaborative control of heterogeneous tasks of a robotic arm based on graph neural networks. Background Technology
[0002] As the application of robotic arms in industrial manufacturing, service robots, and intelligent production lines continues to expand, robotic arms need to handle multiple heterogeneous tasks simultaneously within the same execution cycle, such as grasping, assembling, and transporting different objects, as well as complex operations with timing requirements. Existing technologies typically rely on task scheduling methods based on fixed rules or control models with single task inputs, making it difficult to uniformly model the timing constraints, resource competition relationships, and collaborative relationships between tasks. This results in an inability to accurately represent the coupling structure between heterogeneous tasks, thus limiting the collaborative control capabilities of robotic arms.
[0003] Existing robotic arm control methods also have significant shortcomings in task association modeling, graph structure processing, and motion generation. Traditional graph structure processing methods cannot perform effective frequency domain analysis of task relationship graphs, and lack mechanisms for enhancing inter-task dependencies and suppressing conflicts using Laplace eigenvalue sets and Laplace eigenvector sets, making it difficult to reflect the cooperative behavior of heterogeneous tasks in control variable inference. Furthermore, existing methods lack adaptive update capabilities based on frequency domain errors during motion control variable generation, making it difficult for robotic arms to achieve stable cooperative control outputs in complex heterogeneous task environments.
[0004] Therefore, how to provide a collaborative control method for heterogeneous tasks of robotic arms based on graph neural networks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose a collaborative control method for heterogeneous tasks of a robotic arm based on graph neural networks. This invention uses graph neural networks and spectral domain modulation to achieve collaborative control of heterogeneous tasks of a robotic arm, which has the advantage of high control stability.
[0006] A method for heterogeneous task collaborative control of a robotic arm based on a graph neural network according to an embodiment of the present invention includes the following steps:
[0007] Acquire multiple heterogeneous tasks during the execution process of the robotic arm, and construct a task relationship graph based on the temporal constraints, resource competition relationships, and cooperation relationships between the tasks;
[0008] Calculate the graph Laplacian matrix based on the task relationship graph, perform eigenvalue decomposition, and obtain the set of Laplacian eigenvalues and the set of Laplacian eigenvectors;
[0009] The task feature vectors in the task relationship graph are encoded, graph structure propagation processing is performed, and spatial domain task embedding representations are generated.
[0010] By using the set of Laplacian eigenvectors, the spatial domain task embedding representation is projected onto the spectral domain to generate a spectral domain task representation;
[0011] An improved neural Laplace operator is performed based on the Laplace eigenvalue set and the spectral domain task representation. The spectral domain task representation is weighted and adjusted in the frequency dimension to obtain the spectral domain response representation.
[0012] Based on the task dependencies and task conflicts in the task relationship graph, frequency domain energy modulation is applied to the spectral domain response representation to obtain the spectral domain control representation;
[0013] The inverse spectral domain transformation of the spectral domain control representation is performed using the set of Laplace eigenvectors to generate a time-domain cooperative control representation.
[0014] The temporal-domain collaborative control representation is converted into the motion control quantities of the multi-joint robotic arm and control commands are output.
[0015] The frequency domain error is constructed based on the feedback from the robotic arm's execution, and the parameters of the improved neural Laplacian operator are adaptively updated to continuously optimize the collaborative control strategy.
[0016] Optionally, the construction of the task relationship graph includes:
[0017] When acquiring multiple heterogeneous tasks during the execution of the robotic arm, task type information, task target information, task execution window information, task operation space information, task object parameter information, and task action constraint information are collected for each heterogeneous task, and the initial task data of the task node is constructed with the information.
[0018] Generate timing constraint information based on the execution sequence relationship between multiple heterogeneous tasks, generate resource competition relationship information based on the space occupation conflict between multiple heterogeneous tasks, and generate collaboration relationship information based on the action coordination requirements between multiple heterogeneous tasks.
[0019] The timing constraint information, resource competition relationship information, and cooperation relationship information are respectively established as connection relationships between task nodes;
[0020] Based on the initial task data, a task feature vector is generated. Based on the temporal constraint information, resource competition relationship information, and cooperation relationship information, a set of nodes and a set of edges of the task relationship graph are generated. The task feature vector, the set of nodes, and the set of edges are combined to form the task relationship graph.
[0021] Optionally, the generation of the Laplacian eigenvalue set and the Laplacian eigenvector set includes:
[0022] Read the node set from the task relationship graph and arrange all task nodes in the node set in a fixed order;
[0023] Read the set of edges in the task relationship graph, and process the connection relationships between all task nodes one by one according to the order of the nodes to form connection information;
[0024] Write the number of connections for each task node in the node set to the diagonal position of the graph structure matrix, and write the connection relationship of each item in the edge set to the non-diagonal position of the graph structure matrix to construct the graph structure matrix.
[0025] Construct a graph Laplace matrix based on the arrangement of the diagonal and off-diagonal parts of the graph structure matrix. Combine the diagonal and off-diagonal parts of the graph structure matrix in a fixed manner to form the graph Laplace matrix.
[0026] Eigenvalue decomposition is performed on the graph Laplacian matrix to extract multiple eigenvalues, which are then arranged in a fixed order to form a set of Laplacian eigenvalues.
[0027] The graph Laplacian matrix is subjected to eigenvector decomposition, which extracts multiple eigenvectors from the graph Laplacian matrix. The eigenvectors are then arranged in the same order as the eigenvalues to form a set of Laplacian eigenvectors.
[0028] Optionally, the generation of the spatial domain task embedding representation includes:
[0029] The task feature vectors in the task relationship graph are encoded once, and each task feature vector is converted into an encoded feature component for the graph neural network input through a fixed encoding transformation.
[0030] Read the encoded feature components corresponding to each task node from the task relationship graph, and arrange them sequentially according to the order of the task nodes in the task relationship graph to form a sequence of encoded feature components;
[0031] The sequence of encoded feature components is input into the graph neural network, and the propagation order between the encoded feature components is determined based on the set of edges in the task relationship graph.
[0032] In graph neural networks, based on the mutual propagation order, the encoded feature components of each task node are sequentially subjected to multiple rounds of propagation superposition processing to form intermediate embedding representations, and the intermediate embedding representations are preserved according to the order of the task nodes in the task relationship graph.
[0033] The intermediate embedding representations are combined according to the order of the task nodes to form the spatial domain task embedding representation.
[0034] Optionally, the generation of the spectral domain task representation includes:
[0035] Based on the spatial domain task embedding representation, all spatial domain task embedding data are read sequentially according to the arrangement order of task nodes in the task relationship graph to form a spatial domain task embedding sequence.
[0036] Based on the set of Laplacian feature vectors, all Laplacian feature vectors are read sequentially in a manner consistent with the arrangement order of task nodes, and all feature vectors are arranged to form a Laplacian feature vector sequence.
[0037] The spatial domain task embedding sequence and the Laplacian eigenvector sequence are processed in a fixed order, and each spatial domain task embedding data is projected along the corresponding Laplacian eigenvector direction to generate an initial spectral domain component sequence.
[0038] The initial spectral domain component sequence is recombined according to the arrangement of task nodes in the task relationship graph, and the order of each spectral domain component is adjusted according to the arrangement structure of the feature vector sequence to form a set of spectral domain task node components arranged in a fixed order.
[0039] The spectral domain task node component sets are merged according to the task node arrangement order to generate a complete spectral domain task representation.
[0040] Optionally, the generation of the spectral domain response representation includes:
[0041] Based on the spectral domain task representation, all spectral domain task data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain task data are arranged sequentially to form a spectral domain task sequence.
[0042] Based on the Laplace eigenvalue set, all eigenvalues are read sequentially in the same order as the task nodes, and the eigenvalues are arranged sequentially to form a Laplace eigenvalue sequence.
[0043] The spectral domain task sequence and the Laplace eigenvalue sequence are processed in a fixed order. Each spectral domain task data in the spectral domain task representation is weighted and adjusted in the frequency dimension, and all weighted and adjusted spectral domain task components are arranged in sequence to form a spectral domain task component sequence.
[0044] The weighted and adjusted spectral domain task component sequences are combined according to the arrangement order of task nodes in the task relationship graph to form a spectral domain response representation.
[0045] Optionally, the generation of the spectral domain control representation includes:
[0046] Based on the spectral domain response representation, all spectral domain response data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain response data are arranged sequentially to form a spectral domain response sequence.
[0047] Read the task dependency relationships and task conflict relationships from the task relationship graph, and form the task dependency sequence and task conflict sequence according to the arrangement order of the task nodes in the task relationship graph;
[0048] The spectral domain response sequence, task dependency sequence, and task conflict sequence are combined and processed in a fixed order. Frequency domain energy modulation is performed on each spectral domain response data in the spectral domain response representation to form a frequency domain energy modulated spectral domain task component.
[0049] All frequency-domain energy-modulated spectral domain task components are combined according to the task node arrangement order to form a spectral domain control representation.
[0050] Optionally, the generation of the time-domain cooperative control representation includes:
[0051] Based on the spectral domain control representation, all spectral domain control data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain control data are arranged sequentially to form a spectral domain control sequence.
[0052] Based on the set of Laplacian feature vectors, all Laplacian feature vectors are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the feature vectors are arranged sequentially to form a feature vector sequence.
[0053] The spectral domain control sequence and the feature vector sequence are combined in a fixed order, and an inverse spectral domain transformation is performed on each spectral domain control data to form a time-domain collaborative control representation.
[0054] Optionally, the generation of the robotic arm motion control quantities and control commands includes:
[0055] Based on the temporal collaborative control representation, all temporal collaborative control data are read sequentially according to the arrangement order of each joint of the robotic arm in the robotic arm structure, and the temporal collaborative control data are arranged sequentially to form a temporal collaborative control sequence.
[0056] The temporal collaborative control sequence is processed joint by joint according to the arrangement order of the multiple joints of the robotic arm. For each joint, the temporal collaborative control data related to that joint is read sequentially, and the data is transformed to form the intermediate control component of that joint. All intermediate control components are arranged sequentially according to the arrangement order of the multiple joints of the robotic arm to form an intermediate control component sequence.
[0057] The intermediate control component sequence is merged according to the arrangement order of each joint of the robotic arm in the structure of the robotic arm. Each intermediate control component is combined sequentially once to form a sequence of robotic arm motion control quantities.
[0058] The sequence of robotic arm motion control quantities is combined according to the arrangement order of the robotic arm's multiple joints to form the final robotic arm motion control quantities, and the robotic arm motion control quantities are output as control commands.
[0059] Optionally, the continuous optimization of the adaptive update and cooperative control strategy includes:
[0060] Read the robotic arm execution feedback during the execution process, and extract all feedback data in sequence according to the arrangement order of the multiple joints of the robotic arm in the robotic arm structure. Arrange the feedback data in sequence to form a robotic arm execution feedback sequence.
[0061] The robotic arm execution feedback sequence and spectral domain control representation are processed according to the arrangement order of task nodes in the task relationship diagram. For each feedback data and the corresponding spectral domain control representation, a sequential difference processing is performed to generate frequency domain error. All frequency domain errors are arranged sequentially to form a frequency domain error sequence.
[0062] The frequency domain error sequence is updated item by item according to the arrangement order of task nodes in the task relationship graph. A proportional adjustment operation is performed on each frequency domain error, and the result of the proportional adjustment is applied to the corresponding parameter position to form the parameter set of the improved neural Laplace operator that has been adaptively updated, and the collaborative control strategy is continuously optimized.
[0063] The beneficial effects of this invention are:
[0064] This invention constructs a task relationship graph, expressing the temporal constraints, resource competition, and collaborative relationships during robotic arm execution in a unified structure. This allows for the accurate characterization of previously dispersed, heterogeneous tasks within a single graph structure, overcoming the limitations of traditional methods that cannot provide a holistic understanding of complex coupling relationships between tasks. By performing eigenvalue decomposition on the graph's Laplacian matrix, this invention utilizes the set of Laplacian eigenvalues and Laplacian eigenvectors to construct a spectral domain representation of task relationships. This distinguishes collaborative and conflict trends between tasks in the frequency dimension, thereby providing clear directionality for subsequent adjustments. Combined with the spatial domain task embedding representation generated by a graph neural network, this invention enables each task node to simultaneously contain its own features and structural features propagated from associated tasks, providing a robust input foundation for spectral domain projection and resulting in a more complete expression of task relationships.
[0065] In the spectral domain control stage, this invention employs an improved neural Laplace operator to weight and adjust the spectral domain task representation. It strengthens dependent tasks and suppresses conflicting tasks through frequency domain methods, enabling the spectral domain response representation to naturally reflect collaborative control characteristics. Furthermore, by utilizing frequency domain energy modulation based on dependencies and conflict relationships in the task relationship graph, the spectral domain control representation accurately reflects the collaborative needs between heterogeneous tasks. This ensures that the robot arm's motion generation is no longer an isolated response to a single task, but a coordinated output oriented towards the overall task structure. Finally, a time-domain collaborative control representation is obtained through inverse spectral domain transformation and converted into multi-joint motion control quantities for the robot arm. This results in continuous motion, smooth posture changes, and reduced conflicting behaviors during robot arm execution. In addition, this invention constructs frequency domain errors based on robot arm execution feedback and performs adaptive updates, allowing the control strategy to be dynamically strengthened according to the operating environment, maintaining long-term stability and collaboration. Overall, this invention significantly improves the execution quality of the robot arm in complex heterogeneous task environments, making collaborative control more stable and task transitions smoother, meeting the high flexibility and high reliability requirements of intelligent manufacturing. Attached Figure Description
[0066] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0067] Figure 1 This is an overall flowchart of a heterogeneous task collaborative control method for robotic arms based on graph neural networks proposed in this invention.
[0068] Figure 2 This is a schematic diagram illustrating the generation of a time-domain collaborative control representation by performing an inverse spectral domain transformation using a set of Laplace eigenvectors in a heterogeneous task collaborative control method for robotic arms based on graph neural networks proposed in this invention. Detailed Implementation
[0069] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0070] refer to Figure 1 and Figure 2 A method for collaborative control of heterogeneous tasks of a robotic arm based on graph neural networks includes the following steps:
[0071] Multiple heterogeneous tasks during the execution process of a robotic arm are acquired, and a task relationship graph is constructed based on the temporal constraints, resource competition relationships, and cooperation relationships between the tasks. The task relationship graph contains a defined set of nodes and a set of edges.
[0072] Calculate the graph Laplacian matrix based on the task relationship graph, and perform eigenvalue decomposition on the graph Laplacian matrix to obtain the set of Laplacian eigenvalues and the set of Laplacian eigenvectors;
[0073] The task feature vectors in the task relationship graph are encoded, and graph structure propagation processing is performed based on the graph neural network to generate spatial domain task embedding representations.
[0074] By using the set of Laplacian eigenvectors, the spatial domain task embedding representation is projected onto the spectral domain to generate a spectral domain task representation;
[0075] An improved neural Laplace operator is performed based on the Laplace eigenvalue set and the spectral domain task representation. The spectral domain task representation is weighted and adjusted in the frequency dimension to obtain the spectral domain response representation.
[0076] Based on the task dependency and task conflict relationships in the task relationship diagram, the spectral domain response representation is frequency domain energy modulated to obtain the spectral domain control representation, which can reflect the collaborative control requirements between heterogeneous tasks.
[0077] The inverse spectral domain transformation of the spectral domain control representation is performed using the set of Laplace eigenvectors to generate a time-domain cooperative control representation.
[0078] The temporal-domain collaborative control representation is converted into the robotic arm motion control quantity of the multi-joint robotic arm, so that the robotic arm motion control quantity can be used to drive the robotic arm to perform collaborative task actions and output control commands.
[0079] Frequency domain error is constructed based on the feedback from the robotic arm's execution. The frequency domain error is then used to adaptively update the parameters of the improved neural Laplacian operator, continuously optimizing the collaborative control strategy.
[0080] In this embodiment, the construction of the task relationship graph includes:
[0081] When acquiring multiple heterogeneous tasks during the execution of the robotic arm, task type information, task target information, task execution window information, task operation space information, task object parameter information, and task action constraint information are collected for each heterogeneous task, and the initial task data of the task node is constructed with the information.
[0082] Generate timing constraint information based on the execution sequence relationship between multiple heterogeneous tasks, generate resource competition relationship information based on the space occupation conflict between multiple heterogeneous tasks, and generate collaboration relationship information based on the action coordination requirements between multiple heterogeneous tasks.
[0083] The timing constraint information, resource competition relationship information, and cooperation relationship information are respectively established as connection relationships between task nodes;
[0084] Based on the initial task data, a task feature vector is generated. Based on the temporal constraint information, resource competition relationship information, and cooperation relationship information, a set of nodes and a set of edges of the task relationship graph are generated. The task feature vector, the set of nodes, and the set of edges are combined to form the task relationship graph.
[0085] In this embodiment, the generation of the Laplacian eigenvalue set and the Laplacian eigenvector set includes:
[0086] Read the node set from the task relationship graph and arrange all task nodes in the node set in a fixed order;
[0087] Read the set of edges in the task relationship graph, and process the connection relationships between all task nodes one by one according to the order of the nodes to form the connection information used to generate the graph structure matrix;
[0088] Write the number of connections for each task node in the node set to the diagonal position of the graph structure matrix, and write the connection relationship of each item in the edge set to the non-diagonal position of the graph structure matrix to construct the graph structure matrix.
[0089] A graph Laplacian matrix is constructed based on the arrangement of the diagonal and off-diagonal parts of the graph structure matrix. The diagonal and off-diagonal parts of the graph structure matrix are combined in a fixed manner to form a graph Laplacian matrix for eigenvalue decomposition.
[0090] Eigenvalue decomposition is performed on the graph Laplacian matrix to extract multiple eigenvalues, which are then arranged in a fixed order to form a set of Laplacian eigenvalues.
[0091] The graph Laplacian matrix is subjected to eigenvector decomposition, which extracts multiple eigenvectors from the graph Laplacian matrix. The eigenvectors are then arranged in the same order as the eigenvalues to form a set of Laplacian eigenvectors.
[0092] In this embodiment, the generation of the spatial domain task embedding representation includes:
[0093] The task feature vectors in the task relationship graph are encoded once, and each task feature vector is converted into an encoded feature component for the graph neural network input through a fixed encoding transformation.
[0094] The encoded feature components corresponding to each task node are read from the task relationship graph and arranged sequentially according to the arrangement order of the task nodes in the task relationship graph to form a sequence of encoded feature components, so that the sequence of encoded feature components can correspond to all task nodes in the node set of the task relationship graph.
[0095] The sequence of encoded feature components is input into the graph neural network, and the propagation order between the encoded feature components is determined according to the edge set in the task relationship graph, so that each encoded feature component propagates to the task node connected to it according to the connection structure of the task relationship graph inside the graph neural network.
[0096] In graph neural networks, based on the mutual propagation order, the encoded feature components of each task node are subjected to multiple rounds of propagation superposition processing. Through the superposition and propagation of the encoded feature components in multiple rounds, each task node forms an intermediate embedding representation that includes its own task feature information and the encoded feature information propagated to the node from the task nodes it is connected to in the task relationship graph. The intermediate embedding representation is then preserved according to the order of the task nodes in the task relationship graph.
[0097] The intermediate embedded representations are combined according to the order of the task nodes to form a spatial domain task embedded representation, so that the spatial domain task embedded representation can completely correspond to all task nodes in the task relationship graph.
[0098] In this embodiment, the generation of the spectral domain task representation includes:
[0099] Based on the spatial domain task embedding representation, all spatial domain task embedding data are read sequentially according to the arrangement order of task nodes in the task relationship graph to form a spatial domain task embedding sequence.
[0100] Based on the set of Laplacian feature vectors, all Laplacian feature vectors are read sequentially in a manner consistent with the arrangement order of task nodes, and all feature vectors are arranged to form a Laplacian feature vector sequence.
[0101] The spatial domain task embedding sequence and the Laplacian eigenvector sequence are processed in a fixed order, and each spatial domain task embedding data is projected along the corresponding Laplacian eigenvector direction to generate an initial spectral domain component sequence.
[0102] The initial spectral component sequence is recombined according to the arrangement of task nodes in the task relationship graph, and the order of each spectral component is adjusted according to the arrangement structure of the feature vector sequence to form a set of spectral task node components arranged in a fixed order, so that each spectral component and the corresponding task node maintain a one-to-one correspondence.
[0103] The spectral domain task node component sets are merged according to the task node arrangement order to generate a complete spectral domain task representation.
[0104] In this embodiment, the generation of the spectral domain response representation includes:
[0105] Based on the spectral domain task representation, all spectral domain task data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain task data are arranged sequentially to form a spectral domain task sequence.
[0106] Based on the Laplace eigenvalue set, all eigenvalues are read sequentially in the same order as the task nodes, and the eigenvalues are arranged sequentially to form a Laplace eigenvalue sequence.
[0107] The spectral domain task sequence and the Laplace eigenvalue sequence are processed in a fixed order. Each spectral domain task data in the spectral domain task representation is weighted and adjusted in the frequency dimension, and all weighted and adjusted spectral domain task components are arranged in sequence to form a spectral domain task component sequence.
[0108] The generation of the weighted and adjusted spectral domain task components specifically includes: performing a proportional amplification operation on the spectral domain task data and the corresponding feature value, wherein proportional amplification means multiplying the spectral domain task data by the corresponding feature value, adding the proportionally amplified result to the original spectral domain task data, so that each spectral domain task data forms a combination result containing the original component and the component amplified by the feature value, and scaling the combination result proportionally through a fixed adjustment coefficient, so that the combination result forms the weighted and adjusted spectral domain task components under the combined action of the adjustment coefficient and the feature value;
[0109] The weighted and adjusted spectral domain task component sequences are combined according to the arrangement order of task nodes in the task relationship graph to form a spectral domain response representation.
[0110] In this embodiment, the generation of the spectral domain control representation includes:
[0111] Based on the spectral domain response representation, all spectral domain response data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain response data are arranged sequentially to form a spectral domain response sequence.
[0112] Read the task dependency relationships and task conflict relationships from the task relationship graph, and form the task dependency sequence and task conflict sequence according to the arrangement order of the task nodes in the task relationship graph;
[0113] The spectral domain response sequence, task dependency sequence, and task conflict sequence are combined and processed in a fixed order. Frequency domain energy modulation is performed on each spectral domain response data in the spectral domain response representation to form a frequency domain energy modulated spectral domain task component.
[0114] The formation of the frequency-domain energy-modulated spectral domain task component specifically includes: performing a proportional amplification process on each spectral domain response data according to the task dependency sequence, multiplying the spectral domain response data by the corresponding dependency strength proportionally to form a dependency enhancement component for enhancing the cooperation strength of the task node; performing a proportional reduction process on the spectral domain response data according to the task conflict sequence, multiplying the spectral domain response data by the corresponding conflict strength proportionally to form a conflict reduction component for weakening the conflict impact of the task node; and adding the dependency enhancement component and the conflict reduction component in a fixed order to form the frequency-domain energy-modulated spectral domain task component.
[0115] All frequency-domain energy-modulated spectral domain task components are combined according to the task node arrangement order to form a spectral domain control representation, which can reflect the collaborative control requirements between heterogeneous tasks.
[0116] In this embodiment, the generation of the time-domain cooperative control representation includes:
[0117] Based on the spectral domain control representation, all spectral domain control data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain control data are arranged sequentially to form a spectral domain control sequence.
[0118] Based on the set of Laplacian eigenvectors, all Laplacian eigenvectors are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the eigenvectors are arranged sequentially to form a eigenvector sequence for inverse spectral domain transformation processing;
[0119] The spectral domain control sequence and the feature vector sequence are combined in a fixed order, and an inverse spectral domain transformation is performed on each spectral domain control data to form a time-domain collaborative control representation.
[0120] The formation of the time-domain collaborative control representation specifically includes: performing a proportional expansion process on the spectral domain control data along the direction of the corresponding feature vector, so that the spectral domain control data forms an expansion component in the direction of the feature vector; performing a sequential addition process on all the expansion components in all directions according to the order of the task nodes to form the time-domain task component of the corresponding task node; and combining all the time-domain task components according to the order of the task nodes to form the time-domain collaborative control representation.
[0121] In this embodiment, the generation of the robotic arm motion control quantities and control commands includes:
[0122] Based on the temporal collaborative control representation, all temporal collaborative control data are read sequentially according to the arrangement order of each joint of the robotic arm in the robotic arm structure, and the temporal collaborative control data are arranged sequentially to form a temporal collaborative control sequence.
[0123] The temporal collaborative control sequence is processed joint by joint according to the arrangement order of the multiple joints of the robotic arm. For each joint, the temporal collaborative control data related to that joint is read sequentially, and the data is transformed to form the intermediate control component of that joint. All intermediate control components are arranged sequentially according to the arrangement order of the multiple joints of the robotic arm to form an intermediate control component sequence.
[0124] The generation of the intermediate control component specifically includes: performing a scaling process on the time-domain collaborative control data, multiplying the time-domain collaborative control data by a fixed scaling factor, and adding the scaled result with other control data of the same joint in a fixed order to form the intermediate control component of the joint.
[0125] The intermediate control component sequence is merged according to the arrangement order of each joint of the robotic arm in the structure of the robotic arm. Each intermediate control component is combined sequentially once to form a sequence of robotic arm motion control quantities.
[0126] The sequence of robotic arm motion control quantities is combined according to the arrangement order of the robotic arm's multiple joints to form the final robotic arm motion control quantities, and the robotic arm motion control quantities are output as control commands to drive the robotic arm to perform actions.
[0127] In this embodiment, the continuous optimization of the adaptive update and cooperative control strategy includes:
[0128] Read the robotic arm execution feedback during the execution process, and extract all feedback data in sequence according to the arrangement order of the multiple joints of the robotic arm in the robotic arm structure. Arrange the feedback data in sequence to form a robotic arm execution feedback sequence.
[0129] The robotic arm execution feedback sequence and spectral domain control representation are processed according to the arrangement order of task nodes in the task relationship diagram. For each feedback data and the corresponding spectral domain control representation, a sequential difference processing is performed. The sequential difference processing includes subtracting the feedback data from the spectral domain control representation proportionally to generate a frequency domain error, and arranging all frequency domain errors sequentially to form a frequency domain error sequence.
[0130] The frequency domain error sequence is updated item by item according to the arrangement order of task nodes in the task relationship graph. A proportional adjustment operation is performed on each frequency domain error, and the result of the proportional adjustment is applied to the corresponding parameter position to form the parameter set of the improved neural Laplace operator that has been adaptively updated, and the collaborative control strategy is continuously optimized.
[0131] Example 1:
[0132] To verify the feasibility of this invention in practice, it was applied to the intelligent flexible operation environment of a smart manufacturing center. This manufacturing center is equipped with a six-degree-of-freedom industrial robotic arm, along with a multi-station parts supply device, a vision inspection device, and various actuators, to perform heterogeneous tasks such as grasping, moving, positioning, assembly, and inspection. These tasks arrive at different times in actual production, their operating spaces overlap, and the relationships between tasks are complex. Some tasks must be executed strictly in sequence, some tasks conflict with each other, and some tasks require the robotic arm to quickly coordinate between different workstations. Traditional control methods typically rely on generating control variables for individual tasks, lacking an understanding of the overall structure of multiple tasks. This leads to problems such as abrupt posture switching, unstable path planning, delayed conflict handling, and frequent pauses when the robotic arm performs continuous actions. The heterogeneous task collaborative control method for robotic arms based on graph neural networks proposed in this invention is designed to address these problems.
[0133] Before the system officially goes into operation, engineers first collect information on various tasks throughout the entire workflow, including task objectives, action requirements, execution time intervals, associated workstations, operating space ranges, collaboration needs, and mutual exclusion requirements. Using this information, the system automatically constructs a task relationship diagram, clearly presenting the temporal constraints, resource conflicts, and collaborative relationships between tasks in a structured form. In the on-site example at this manufacturing center, the robotic arm needs to sequentially pick up parts from different material channels and deliver them to different assembly stations, while simultaneously identifying and calibrating the parts' angles during the operation. These tasks exhibit significant dependencies and spatial overlap. The task relationship diagram constructed by this invention allows these complex relationships to be fully represented, providing fundamental structural information for subsequent control processes.
[0134] After the task relationship graph is constructed, the system further calculates the graph Laplacian matrix and performs eigenvalue decomposition to generate a set of Laplacian eigenvalues and a set of Laplacian eigenvectors, enabling the task relationships to be transformed from a spatial structure to a spectral structure. The spectral structure can separate cooperative trends from conflict trends, allowing characteristics that were previously difficult to observe directly from the task relationship graph to be presented more clearly in the frequency domain. In the system interface at the manufacturing center, engineers can see that the low-frequency components reflect the cooperative relationships between tasks, while the high-frequency components reflect the conflict relationships, making subsequent adjustments to the impact on tasks more explicit.
[0135] Subsequently, this invention utilizes a graph neural network to propagate the task feature vectors, enabling each task node to retain not only its own features but also structural information propagated from its neighboring tasks, thus forming a spatial domain task embedding representation. In practical applications, this embedding representation can typically capture complex task dependency structures, allowing the system to understand implicit relationships between certain tasks. For example, the completion of a grasping task may affect subsequent assembly tasks, or a detection task may interfere with certain spatial poses and handling actions. Since these relationships have been absorbed by the model, the spatial domain task embedding representation provides a more stable foundation for subsequent spectral domain projection.
[0136] After obtaining the spatial domain task embedding representation, the system projects it onto the spectral domain based on the Laplacian eigenvector set, forming a spectral domain task representation. This allows for the separation of different types of task relationships along the frequency dimension, facilitating further control. After applying the spectral domain processing of this invention, the system can establish a control mechanism in the spectral domain that "enhances cooperation and suppresses conflict." In this implementation scenario, the spectral domain task representation is weighted and adjusted using an improved neural Laplacian operator, amplifying low-frequency components related to cooperation and effectively suppressing high-frequency components related to conflict. This step ensures that the spectral domain response representation truly reflects the needs of the production site, making subsequent control more aligned with task relationships.
[0137] To further improve control performance, the system performs frequency domain energy modulation on the spectral domain response representation based on the dependencies and conflicts in the task relationship graph. This ensures that the generated spectral domain control representation clearly reflects the strengthening relationships between dependent tasks and the weakening relationships between conflicting tasks. Engineers observed in the test data visualization interface that after frequency domain energy modulation, the relationships between tasks were more clearly expressed, and the entire control structure was more coordinated.
[0138] After obtaining the spectral domain control representation, the system performs an inverse spectral domain transformation using the Laplace eigenvector set to restore the control information to the time domain, thereby generating a time-domain cooperative control representation. During field operation, the robotic arm's movements became smoother, with a significant reduction in sudden stops, jitter, and invalid swings in the motion trajectory. When performing multiple interconnected tasks, the robotic arm exhibited more coherent motion transitions, more natural posture adjustments, and a marked improvement in overall operational rhythm.
[0139] Ultimately, this invention converts the time-domain collaborative control representation into multi-joint motion control quantities for the robotic arm and outputs control commands, enabling the robotic arm to execute tasks according to the collaborative planning results. Throughout the entire operating cycle, the system also constructs frequency-domain errors based on the robotic arm's execution feedback and updates the parameters of the improved neural Laplace operator in real time, allowing the control strategy to continuously adapt to changes in production pace and environment. After several days of continuous operation, the robotic arm's motion stability and task seamlessness have significantly improved, fully verifying the effectiveness of this invention in heterogeneous task collaborative control.
[0140] Table 1. Experimental data comparing the performance of the method of this invention with that of traditional methods.
[0141] Evaluation indicators Traditional rules and laws Traditional single-task control Method of the present invention Increase Movement stability (amplitude of joint vibration) 12.5 9.8 4.1 -67.2% Task connection delay (milliseconds) 185 132 68 -63.2% Number of conflict triggers (per hour) 14 11 3 -78.6% Attitude transition smoothness (normalized value) 0.42 0.51 0.83 +62.7% Overall operational rhythm continuity (score) 6.1 7.3 9.2 +30.9%
[0142] As shown in Table 1, the method of this invention exhibits significant advantages in several key performance indicators. Firstly, regarding motion stability, the joint jitter amplitude of traditional rule-based methods and traditional single-task control methods is at a relatively high level, while that of this invention is only 4.1, a reduction of over 60%. This is mainly due to the fact that this invention effectively weakens high-frequency interference components in the spectral domain through an improved neural Laplace operator, enabling the robotic arm to maintain stability even under complex task transitions.
[0143] Regarding task transition latency, this invention significantly reduces the high latency of traditional rule-based methods, making task switching for robotic arms smoother. This is thanks to the invention's use of task relationship graphs to uniformly express temporal constraints, dependencies, and conflict relationships, and the generation of spatial domain task embedding representations that reflect the overall structure through graph neural networks, thus eliminating fragmented motion planning.
[0144] This invention offers a significant advantage in terms of the number of conflict triggers. Traditional methods generate numerous conflict events per hour, while this invention enhances task dependence and actively weakens conflict relationships through frequency domain energy modulation, reducing the number of conflict triggers to three. This demonstrates that this invention can resolve potential conflicts in advance during the control generation phase, rather than passively avoiding them when the robotic arm is executing.
[0145] In terms of posture transition smoothness, this invention significantly improves it to 0.83, indicating that the temporal collaborative control representation generated by inverse spectral domain transformation effectively reduces discontinuities in the robotic arm's posture change process, making the motion trajectory more natural. Meanwhile, the overall running rhythm continuity score reaches 9.2, a significant improvement over traditional methods, reflecting the system's ability to maintain a high level of collaboration during long-term operation.
[0146] In summary, this invention, by combining spectral domain modulation with structured task modeling, fundamentally solves problems such as unstable motion, disjointed task switching, and difficulty in suppressing conflicts, enabling the robotic arm to exhibit stronger reliability and coordinated execution capabilities when handling complex and heterogeneous tasks.
[0147] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for heterogeneous task collaborative control of a robotic arm based on graph neural networks, characterized in that, Includes the following steps: Acquire multiple heterogeneous tasks during the execution process of the robotic arm, and construct a task relationship graph based on the temporal constraints, resource competition relationships, and cooperation relationships between the tasks; Calculate the graph Laplacian matrix based on the task relationship graph, perform eigenvalue decomposition, and obtain the set of Laplacian eigenvalues and the set of Laplacian eigenvectors; The task feature vectors in the task relationship graph are encoded, graph structure propagation processing is performed, and spatial domain task embedding representations are generated. By using the set of Laplacian eigenvectors, the spatial domain task embedding representation is projected onto the spectral domain to generate a spectral domain task representation; An improved neural Laplace operator is performed based on the Laplace eigenvalue set and the spectral domain task representation. The spectral domain task representation is weighted and adjusted in the frequency dimension to obtain the spectral domain response representation. Based on the task dependencies and task conflicts in the task relationship graph, frequency domain energy modulation is applied to the spectral domain response representation to obtain the spectral domain control representation; The inverse spectral domain transformation of the spectral domain control representation is performed using the set of Laplace eigenvectors to generate a time-domain cooperative control representation. The temporal-domain collaborative control representation is converted into the motion control quantities of the multi-joint robotic arm and control commands are output. The frequency domain error is constructed based on the feedback from the robotic arm's execution, and the parameters of the improved neural Laplacian operator are adaptively updated to continuously optimize the collaborative control strategy.
2. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The construction of the task relationship graph includes: When acquiring multiple heterogeneous tasks during the execution of the robotic arm, task type information, task target information, task execution window information, task operation space information, task object parameter information, and task action constraint information are collected for each heterogeneous task, and the initial task data of the task node is constructed with the information. Generate timing constraint information based on the execution sequence relationship between multiple heterogeneous tasks, generate resource competition relationship information based on the space occupation conflict between multiple heterogeneous tasks, and generate collaboration relationship information based on the action coordination requirements between multiple heterogeneous tasks. The timing constraint information, resource competition relationship information, and cooperation relationship information are respectively established as connection relationships between task nodes; Based on the initial task data, a task feature vector is generated. Based on the temporal constraint information, resource competition relationship information, and cooperation relationship information, a set of nodes and a set of edges of the task relationship graph are generated. The task feature vector, the set of nodes, and the set of edges are combined to form the task relationship graph.
3. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The generation of the Laplacian eigenvalue set and the Laplacian eigenvector set includes: Read the node set from the task relationship graph and arrange all task nodes in the node set in a fixed order; Read the set of edges in the task relationship graph, and process the connection relationships between all task nodes one by one according to the order of the nodes to form connection information; Write the number of connections for each task node in the node set to the diagonal position of the graph structure matrix, and write the connection relationship of each item in the edge set to the non-diagonal position of the graph structure matrix to construct the graph structure matrix. Construct a graph Laplace matrix based on the arrangement of the diagonal and off-diagonal parts of the graph structure matrix. Combine the diagonal and off-diagonal parts of the graph structure matrix in a fixed manner to form the graph Laplace matrix. Eigenvalue decomposition is performed on the graph Laplacian matrix to extract multiple eigenvalues, which are then arranged in a fixed order to form a set of Laplacian eigenvalues. The graph Laplacian matrix is subjected to eigenvector decomposition, which extracts multiple eigenvectors from the graph Laplacian matrix. The eigenvectors are then arranged in the same order as the eigenvalues to form a set of Laplacian eigenvectors.
4. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The generation of the spatial domain task embedding representation includes: The task feature vectors in the task relationship graph are encoded once, and each task feature vector is converted into an encoded feature component for the graph neural network input through a fixed encoding transformation. Read the encoded feature components corresponding to each task node from the task relationship graph, and arrange them sequentially according to the order of the task nodes in the task relationship graph to form a sequence of encoded feature components; The sequence of encoded feature components is input into the graph neural network, and the propagation order between the encoded feature components is determined based on the set of edges in the task relationship graph. In graph neural networks, based on the mutual propagation order, the encoded feature components of each task node are sequentially subjected to multiple rounds of propagation superposition processing to form intermediate embedding representations, and the intermediate embedding representations are preserved according to the order of the task nodes in the task relationship graph. The intermediate embedding representations are combined according to the order of the task nodes to form the spatial domain task embedding representation.
5. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The generation of the spectral domain task representation includes: Based on the spatial domain task embedding representation, all spatial domain task embedding data are read sequentially according to the arrangement order of task nodes in the task relationship graph to form a spatial domain task embedding sequence. Based on the set of Laplacian feature vectors, all Laplacian feature vectors are read sequentially in a manner consistent with the arrangement order of task nodes, and all feature vectors are arranged to form a Laplacian feature vector sequence. The spatial domain task embedding sequence and the Laplacian eigenvector sequence are processed in a fixed order, and each spatial domain task embedding data is projected along the corresponding Laplacian eigenvector direction to generate an initial spectral domain component sequence. The initial spectral domain component sequence is recombined according to the arrangement of task nodes in the task relationship graph, and the order of each spectral domain component is adjusted according to the arrangement structure of the feature vector sequence to form a set of spectral domain task node components arranged in a fixed order. The spectral domain task node component sets are merged according to the task node arrangement order to generate a complete spectral domain task representation.
6. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The generation of the spectral domain response representation includes: Based on the spectral domain task representation, all spectral domain task data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain task data are arranged sequentially to form a spectral domain task sequence. Based on the Laplace eigenvalue set, all eigenvalues are read sequentially in the same order as the task nodes, and the eigenvalues are arranged sequentially to form a Laplace eigenvalue sequence. The spectral domain task sequence and the Laplace eigenvalue sequence are processed in a fixed order. Each spectral domain task data in the spectral domain task representation is weighted and adjusted in the frequency dimension, and all weighted and adjusted spectral domain task components are arranged in sequence to form a spectral domain task component sequence. The weighted and adjusted spectral domain task component sequences are combined according to the arrangement order of task nodes in the task relationship graph to form a spectral domain response representation.
7. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The generation of the spectral domain control representation includes: Based on the spectral domain response representation, all spectral domain response data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain response data are arranged sequentially to form a spectral domain response sequence. Read the task dependency relationships and task conflict relationships from the task relationship graph, and form the task dependency sequence and task conflict sequence according to the arrangement order of the task nodes in the task relationship graph; The spectral domain response sequence, task dependency sequence, and task conflict sequence are combined and processed in a fixed order. Frequency domain energy modulation is performed on each spectral domain response data in the spectral domain response representation to form a frequency domain energy modulated spectral domain task component. All frequency-domain energy-modulated spectral domain task components are combined according to the task node arrangement order to form a spectral domain control representation.
8. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The generation of the time-domain cooperative control representation includes: Based on the spectral domain control representation, all spectral domain control data are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the spectral domain control data are arranged sequentially to form a spectral domain control sequence. Based on the set of Laplacian feature vectors, all Laplacian feature vectors are read sequentially according to the arrangement order of task nodes in the task relationship graph, and the feature vectors are arranged sequentially to form a feature vector sequence. The spectral domain control sequence and the feature vector sequence are combined in a fixed order, and an inverse spectral domain transformation is performed on each spectral domain control data to form a time-domain collaborative control representation.
9. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The generation of the robotic arm motion control quantities and control commands includes: Based on the temporal collaborative control representation, all temporal collaborative control data are read sequentially according to the arrangement order of each joint of the robotic arm in the robotic arm structure, and the temporal collaborative control data are arranged sequentially to form a temporal collaborative control sequence. The temporal collaborative control sequence is processed joint by joint according to the arrangement order of the multiple joints of the robotic arm. For each joint, the temporal collaborative control data related to that joint is read sequentially, and the data is transformed to form the intermediate control component of that joint. All intermediate control components are arranged sequentially according to the arrangement order of the multiple joints of the robotic arm to form an intermediate control component sequence. The intermediate control component sequence is merged according to the arrangement order of each joint of the robotic arm in the structure of the robotic arm. Each intermediate control component is combined sequentially once to form a sequence of robotic arm motion control quantities. The sequence of robotic arm motion control quantities is combined according to the arrangement order of the robotic arm's multiple joints to form the final robotic arm motion control quantities, and the robotic arm motion control quantities are output as control commands.
10. The method for heterogeneous task collaborative control of a robotic arm based on graph neural networks according to claim 1, characterized in that, The continuous optimization of the adaptive update and cooperative control strategy includes: Read the robotic arm execution feedback during the execution process, and extract all feedback data in sequence according to the arrangement order of the multiple joints of the robotic arm in the robotic arm structure. Arrange the feedback data in sequence to form a robotic arm execution feedback sequence. The robotic arm execution feedback sequence and spectral domain control representation are processed according to the arrangement order of task nodes in the task relationship diagram. For each feedback data and the corresponding spectral domain control representation, a sequential difference processing is performed to generate frequency domain error. All frequency domain errors are arranged sequentially to form a frequency domain error sequence. The frequency domain error sequence is updated item by item according to the arrangement order of task nodes in the task relationship graph. A proportional adjustment operation is performed on each frequency domain error, and the result of the proportional adjustment is applied to the corresponding parameter position to form the parameter set of the improved neural Laplace operator that has been adaptively updated, and the collaborative control strategy is continuously optimized.