An industrial robot intelligent control method based on a knowledge graph

Abstract

The application discloses an industrial robot intelligent control method based on a knowledge graph, and relates to the technical field of industrial robot control.The method constructs a dynamic subgraph according to a work order identifier, obtains a graph embedding vector through multi-layer message passing of node features by using a graph neural network, and maps the graph embedding vector into a forced term weight coefficient and a target position parameter of a dynamic motion primitive to generate a robot candidate control instruction.Before the instruction is issued, a safety specification, a human skeleton key point and a robot link bounding box are expressed as physical space-time edges for safety checking, and a replacement control instruction is generated when it is determined that there is an interference risk.The checked control instruction is issued to a servo driver for execution.The application shortens the control link delay, reduces the collision risk in a human-machine integration scene, and is suitable for an intelligent manufacturing scene.

Description

Technical Field

[0001] This invention relates to the field of industrial robot control technology, specifically to an intelligent control method for industrial robots based on knowledge graphs. Background Technology

[0002] Knowledge graphs, in the form of structured semantic networks, represent entities and their relationships in industrial scenarios and have been applied to the field of robot task planning. Existing robot control schemes based on knowledge graphs typically first obtain scene description information through sensor or instruction parsing, then match it with a pre-built static knowledge graph, and obtain high-level task symbol instructions through predefined rule reasoning. Subsequently, the symbol instructions are converted into robot joint angle sequences or end-effector pose sequences by a manually pre-written translation rule base, and finally the controller drives the servo system to execute the corresponding actions.

[0003] In the above scheme, the reasoning results of the knowledge graph need to go through multiple serial conversion processes such as rule matching, program calling and trajectory interpolation between the robot's underlying continuous control signals. This results in high system response latency, making it difficult to meet the requirements of millisecond-level real-time control in industrial scenarios. At the same time, this type of graph remains static after the system is deployed and is no longer updated. When new tools or parts that are not defined in the graph appear on the production site, the system often fails to retrieve the corresponding entity, leading to task planning failure.

[0004] Furthermore, the lack of a calculable real-time verification relationship between existing control strategies and industrial safety standards poses a safety hazard of physical collisions when personnel enter the robot's workspace.

[0005] Therefore, how to provide an intelligent control method for industrial robots that can shorten the conversion delay between graph inference and low-level control, has online autonomous evolution capabilities, and has an embedded security verification mechanism has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The technical problem to be solved by this invention is that in existing knowledge graph-based industrial robot control methods, the conversion between knowledge graph reasoning results and robot underlying continuous control signals relies on manually pre-written translation rules, resulting in high system response latency. Static knowledge graphs are no longer updated after system deployment, making it difficult to cope with task planning failures caused by new objects on site. Furthermore, there is a lack of a computable real-time verification mechanism between control strategies and industrial safety standards, posing a potential physical collision hazard in human-machine collaborative scenarios.

[0007] To address the aforementioned technical problems, this invention provides a knowledge graph-based intelligent control method for industrial robots, comprising:

[0008] Based on the current work order identifier, obtain the set of entities and relationships associated with the work order identifier from the static process knowledge graph in the cloud, and construct a dynamic sub-graph by combining the scene object status information collected in real time by the sensor;

[0009] The graph embedding vector of the task target node is obtained by performing multi-layer message passing aggregation on the node features in the dynamic subgraph through a graph neural network.

[0010] The graph is embedded into a vector input to a pre-trained mapping network to obtain the forced term weight coefficients and target position parameters of the dynamic motion primitives, and candidate control commands for the robot are generated accordingly.

[0011] Before issuing the candidate control command, the safety distance requirements in the industrial safety specifications, the key point model of the human skeleton and the bounding box of the robot link kinematics are expressed as the physical spatiotemporal edges in the dynamic sub-graph. Based on the candidate control command, the spatial position trajectory of each link bounding box of the robot in the future preset time window is predicted and the safety distance is calculated. When it is determined that there is an interference risk, an alternative control command is generated to obtain the verified control command.

[0012] The verified control command is sent to the robot servo driver to drive the robot to perform actions.

[0013] This invention also provides an online self-growing optimization implementation of the knowledge graph corresponding to the above method, as well as a corresponding intelligent control system for industrial robots.

[0014] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0015] First, by using graph neural networks to extract features from dynamic subgraphs and directly map them to dynamic motion primitive parameters, the manual translation rule base required for converting discrete symbolic instructions into continuous control signals is eliminated, shortening the conversion link from graph inference results to underlying control signals, and meeting the real-time requirements of millisecond-level control cycles in industrial scenarios.

[0016] Secondly, in a further embodiment of the present invention, when encountering an unknown object, its features are injected into a dynamic subgraph and combined with reinforcement learning for trial manipulation. The successful operation trajectory obtained from the trial is encoded as a physical operation token and fed back to the graph node and edge attributes, so that the graph structure has the ability to evolve autonomously with the on-site task, thereby improving the system's adaptability in flexible manufacturing scenarios with multiple varieties and small batches.

[0017] Third, by unifying industrial safety standards, key points of the human skeleton, and the bounding box of robot linkage kinematics into a computable physical spatiotemporal boundary in the graph, real-time interference prediction is performed before the control command is issued, and alternative control commands are output. This reduces the risk of physical collisions in human-machine collaborative scenarios and improves the overall safety and reliability of the system. Attached Figure Description

[0018] Figure 1 This is a diagram showing the overall architecture of the intelligent control system for industrial robots of this invention.

[0019] Figure 2 This is a flowchart of the knowledge graph-based intelligent control method for industrial robots according to the present invention.

[0020] Figure 3 This is a flowchart of the online self-growth mechanism of the knowledge graph of this invention. Detailed Implementation

[0021] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0022] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] I. System Overall Architecture

[0024] The intelligent control method for industrial robots provided in this invention is based on a three-tier architecture: a cloud-based static process knowledge graph, an edge-based dynamic sub-graph, and the robot's underlying execution layer. The cloud server stores a plant-wide static process knowledge graph, including process relationships, process parameters, and equipment topology information, and provides graph version management and sub-graph trimming and distribution interfaces. The edge controller can be implemented using an industrial computing unit with graphics processing capabilities. Based on the current work order identifier, it retrieves relevant local sub-graphs from the cloud to construct a lightweight dynamic sub-graph, and performs graph neural network inference, dynamic motion primitive parameter generation, security verification, and other functions at the edge. For real-time computing tasks such as line learning, the robot's underlying servo system only receives verified control commands from the edge and executes corresponding actions. At the same time, it transmits data collected by joint encoders, force sensors, and vision sensors back to the edge controller. Data interaction between the cloud and the edge controller can be achieved through industrial Ethernet or a private wireless network. The edge controller and the robot servo drive communicate using a fieldbus that meets real-time requirements. The decoupled design of the above three-level architecture allows graph storage and updating, graph inference and mapping calculation, and underlying servo execution to be completed on different computing nodes, thus balancing the system's real-time performance and scalability.

[0025] II. Construction of Dynamic Subgraphs

[0026] The edge controller retrieves the set of entities and relationships associated with the current work order from the static process knowledge graph in the cloud based on the current work order identifier, and combines this with the scene object state information collected in real time by the vision and force sensors to construct a dynamic sub-graph. This dynamic sub-graph is denoted as... ,in For a set of nodes, For the set of edges, each node in the dynamic subgraph corresponds to an entity in the scene, such as a workpiece, tool, robot joint, or person. Each node carries category attributes, geometric pose attributes, and physical attributes. The physical attributes include mass and friction coefficient. The edges between nodes represent semantic relationships between entities, such as grasping reachability, contact, and proximity relationships. Each edge is accompanied by a relationship type label and confidence weight.

[0027] For each node Assign initial feature vectors The vector is composed of the one-hot encoding of the category of the entity corresponding to the node, the pose coordinates, and the physical attributes.

[0028] III. Graph Neural Network Inference and Graph Embedding Vector Generation

[0029] For the first graph neural network layer( For each node Aggregate its neighbor node set In the The feature vector of the layer is updated according to the following formula to obtain the first layer. Feature vectors of the layer:

[0030] ;

[0031] in, Represents a node In the The feature vector of the layer, The neighbor feature aggregation function is represented in this embodiment using a weighted summation method. In other embodiments, it can also be implemented using average aggregation or attention-weighted aggregation. For the first The learnable weight matrix of the layer, For the first The layer's bias vector, Since it is a non-linear activation function, this embodiment uses a linear rectified function to implement it. Represents a node In the figure The set of first-order neighbor nodes in the [theory / system].

[0032] After a total After the message passing iteration, the task target node (i.e., the node corresponding to the current workpiece to be operated) is taken at the [number]th [time]. eigenvectors of the layer As the graph embedding vector for this task, denoted as ,in As the dimension of the graph embedding vector, the weight matrix and bias vector of the graph neural network can be obtained in advance through offline supervised training based on the graph structure and corresponding control trajectory samples recorded during historical operations.

[0033] IV. Generation of Dynamic Motion Element Parameters and Generation of Candidate Control Commands

[0034] Embedding a graph into a vector Input pre-trained mapping network Output the forced term weight coefficient matrix of the dynamic motion primitive. With target position parameters , represented as:

[0035] ;

[0036] in, for The weight coefficients corresponding to each basis function. For the target position parameters of the dynamic motion primitive, For a pre-trained fully connected mapping network, Its network parameters.

[0037] weight coefficient matrix Each weight coefficient in the equation is weighted and summed with its corresponding Gaussian function, and then multiplied by the phase variable describing the motion progress to generate the continuous trajectory forcing term:

[0038] ;

[0039] in, For the forced term function value, As a phase variable, it monotonically converges from 1 to 0 over time and is used to describe the progress of motion. For the first Gaussian functions, whose expressions are as follows: , For the first The central parameters of the basis functions Its width parameter, both of which are preset values. For the first The weight coefficients corresponding to the basis functions, i.e., the aforementioned The amount.

[0040] Compulsory items With target position parameters Substituting the second-order differential equation system into the dynamic motion primitives, the continuous position and velocity commands of the robot's end effector or joints are obtained by real-time integration:

[0041] ;

[0042] ;

[0043] in, This represents the robot's current trajectory position. For trajectory velocity, For the target position parameters obtained above, The elastic coefficient, The damping coefficient is... This is a time scale constant used to adjust the trajectory execution speed. , They are respectively , The first derivative with respect to time.

[0044] The position and velocity commands obtained by the above integral solution constitute candidate control commands, which are continuously output in a control cycle. In this embodiment, the control cycle is 1 millisecond to 4 milliseconds.

[0045] V. Spatiotemporal Security Guardian Verification

[0046] Before the candidate control command is sent to the robot servo driver, the edge controller performs a safety check on the command. Specifically, the safety distance requirements in the ISO industrial safety standard, the set of three-dimensional coordinates of key points of the human skeleton estimated in real time by the vision sensor, and the kinematic bounding boxes of each link of the robot are uniformly expressed as physical spatiotemporal edges in the dynamic subgraph. Each edge is accompanied by the spatial coordinates, velocity vector and safety distance threshold attributes of both parties at the current moment.

[0047] Based on the target position in the candidate control command With trajectory Predicting future preset time windows The spatial position trajectory of each link bounding box of the internal robot is calculated, and the real-time safe distance between this trajectory and key points of the human skeleton and other obstacle nodes is calculated:

[0048] ;

[0049] in, Let be the three-dimensional coordinate vector of the center point of the robot link bounding box. The three-dimensional coordinate vectors of key points on the human body. , These are the equivalent radii of the robot link enclosure and the human safety envelope, respectively. This is an additional safety margin set according to ISO safety standards. This represents the Euclidean norm.

[0050] A graph search algorithm is used to search for the existence of a feature in the spatiotemporal safety graph. Path nodes with a value less than zero: If no such path node exists, the candidate control command is deemed safe and is used as the verified control command. If such a path node exists, it is deemed a potential physical interference. Based on the current interference direction and the preset safety level rules, a detour trajectory or a deceleration and stop command is generated as an alternative control command to replace the trajectory segment in the corresponding time window of the candidate control command, thus obtaining the verified control command.

[0051] The verified control command is sent to the robot servo driver for execution, and the safety verification result of this cycle is recorded and fed back to the reward calculation stage of the reinforcement learning module described later.

[0052] VI. Online Self-Growing Knowledge Graph

[0053] In multi-variety, small-batch manufacturing scenarios, to enable the system to autonomously learn and cope with unknown objects, this embodiment also sets up an online self-growing mechanism for the knowledge graph. Visual and force sensors continuously collect image features and contact force features of objects within the current workspace, inputting them into a pre-trained feature extraction network to obtain object feature vectors. The similarity between these object feature vectors and the feature vectors of existing nodes in the dynamic sub-graph is calculated one by one. The similarity is maintained until all similarity values ​​are below a preset threshold. In the case of an unknown object, determine that the object is an unknown object; otherwise, match it to an existing node.

[0054] Use the feature vector of the unknown object as a new node Inject a dynamic subgraph and establish initial candidate edges with existing nodes in the graph based on its spatial location and geometric contact relationship. The relationship type of the candidate edge is temporarily marked as pending confirmation.

[0055] With new nodes The state of the local subgraph is used as the observation state of the reinforcement learning environment. Invoke the pre-trained policy network Output probing action That is, the candidate fine-tuning trajectory of the robot's end effector:

[0056] ;

[0057] in, For at any time Output candidate actions, For the corresponding graph state observation, For policy networks, Its network parameters.

[0058] Robot performs actions Then, a comprehensive reward value is calculated based on task completion, safety, and energy consumption:

[0059] ;

[0060] in, The reward for task completion is calculated based on whether the target contact state or pose error has been achieved. As a security penalty item, it is determined by the risk score output from the aforementioned security verification process. This is an energy consumption penalty term, calculated based on the integral of joint torque. , , These are the corresponding weighting coefficients.

[0061] After repeating the above trial action output and reward value calculation process for a preset number of trial rounds, sort the interaction trajectories according to the cumulative reward value, and select the interaction trajectory that is ranked first as the successful operation trajectory of the unknown object.

[0062] The successful operation trajectory is encoded as a physical operation token, which specifically includes the joint angle sequence, force feedback sequence, and the finally confirmed edge relationship type during the operation. This token is then written to a new node. The attribute fields of the candidate edges are updated, and the relationship type of the candidate edges is updated from the pending confirmation state to the actual relationship type confirmed according to the successful operation trajectory, thus completing the graph solidification update.

[0063] The updated local subgraph structure is periodically synchronized to the cloud static process graph, and after review, it is incorporated into the global knowledge graph for reuse in subsequent similar tasks, realizing knowledge sharing across workstations and batches.

[0064] VII. Specific Application Examples

[0065] Taking a mechanical parts assembly scenario as an example, after the robot receives a work order identifier, the edge controller retrieves the workpiece nodes, tool nodes, and workstation nodes associated with the work order from the cloud. Combined with the actual material pose identified by the visual sensor, a dynamic sub-graph is constructed, containing the aforementioned nodes, their proximity relationships, and grasping reachability relationships. After a three-layer message passing process, the graph neural network takes the feature vector of the workpiece node to be grasped as the graph embedding vector, inputs it into the mapping network to obtain the weight coefficients of the dynamic motion primitives and the target position parameters. Then, it integrates to obtain the continuous position and velocity commands of the robot's end effector on the assembly path. Before issuing these commands, the edge controller uses real-time collected human bone data... The system sets the coordinates of key points and determines that the safe distance between the trajectory and the operator is greater than zero within a preset time window. If the command is deemed safe, it is directly issued for execution. When the operator briefly approaches the assembly area, causing the safe distance to become less than zero at a path node, the edge controller generates a deceleration and stop command based on the interference direction, replaces the trajectory segment within the corresponding time window, and issues it for execution. After the operator leaves, the execution of the original trajectory is resumed. When a new type of fixture not registered in the map appears at the workstation, the system autonomously completes feature extraction, new node injection, and trial control based on the online self-growth mechanism of this invention, and solidifies the confirmed operational experience into the map for direct reuse in subsequent assembly tasks of similar new fixtures.

[0066] VIII. Alternative Implementation Methods

[0067] In other embodiments of the present invention, the trajectory parameterization method of the dynamic motion primitive can also be replaced by a trajectory parameterization method based on spline curves, that is, mapping the graph embedding vector to the control point coordinates of the spline curve, and generating a continuous trajectory through spline interpolation. This can also realize the end-to-end conversion of the graph inference results to continuous control signals, and can serve as an effective supplement to the dynamic motion primitive scheme.

[0068] In other embodiments of the present invention, the reinforcement learning trial manipulation in the online self-growth mechanism of the knowledge graph can be replaced by a method based on imitation learning, that is, providing a demonstration trajectory for new objects through remote teaching, and directly encoding the demonstration trajectory as a physical operation token and solidifying it into the graph, saving multiple rounds of trial processes, which is suitable for scenarios with high security requirements or high trial costs.

[0069] IX. Implementation Results Description

[0070] In the specific application scenario of this embodiment, after the robot receives the work order, the edge controller completes the complete closed-loop processing of dynamic sub-graph construction, graph neural network inference, dynamic motion primitive parameter generation, safety verification and instruction issuance within a millisecond-level control cycle. There is no need to manually pre-write the symbol translation rule base, which can meet the real-time control needs of industrial sites. When new tools or new parts appear in the workspace, the system can autonomously complete feature learning and experience solidification through an online self-growth mechanism without stopping the machine for manual reconfiguration of the graph. When personnel enter the robot's workspace, the spatiotemporal safety guard mechanism can output alternative control instructions in time before the instruction is issued, reducing the risk of physical collision and improving the safety of operation in human-machine collaborative scenarios.

[0071] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A knowledge graph-based intelligent control method for industrial robots, characterized in that, include: Based on the current work order identifier, the entity and relationship set associated with the work order identifier are obtained from the static process knowledge graph in the cloud. Combined with the scene object status information collected in real time by the sensor, a dynamic subgraph is constructed. The dynamic subgraph includes a node set and an edge set. Each node in the node set corresponds to an entity in the scene and carries category attributes, pose attributes and physical attributes. Each edge in the edge set represents the semantic relationship between nodes. The graph embedding vector of the task target node is obtained by performing multi-layer message passing aggregation on the node features in the dynamic subgraph through a graph neural network. The graph is embedded into a vector and input into a pre-trained mapping network to obtain the forced term weight coefficients and target position parameters of the dynamic motion primitives. The forced terms of the dynamic motion primitive are generated according to the forced term weight coefficients. The forced terms and the target position parameters are substituted into the second-order differential equation system of the dynamic motion primitive for integration and solution to obtain the candidate control commands of the robot end effector or joint. The candidate control commands include continuous position commands and velocity commands. Before issuing the candidate control command, a safety verification is performed on the candidate control command to obtain a verified control command. Specifically, this includes: expressing the safety distance requirements in the industrial safety specifications, the key point model of the human skeleton, and the bounding box of the robot's linkage kinematics as physical spatiotemporal edges in the dynamic sub-graph; predicting the spatial position trajectory of each link bounding box of the robot within a future preset time window based on the candidate control command; calculating the safety distance between the spatial position trajectory and the key points of the human skeleton and obstacle nodes; when it is determined that there is a situation where the safety distance is less than zero, generating an alternative control command to replace the trajectory segment in the corresponding time window of the candidate control command to obtain the verified control command; when it is determined that there is no situation where the safety distance is less than zero, using the candidate control command as the verified control command. The verified control command is sent to the robot servo driver to drive the robot to perform actions.

2. The method according to claim 1, characterized in that, The step of performing multi-layer message passing aggregation on node features in the dynamic sub-graph using a graph neural network includes: Each node in the dynamic sub-graph is assigned an initial feature vector, which is formed by concatenating the category one-hot encoding, pose coordinates and physical attributes of the entity corresponding to the node; For each layer of the graph neural network, the feature vectors of the neighboring nodes in the current layer are aggregated for each node. The aggregation result is linearly transformed by a learnable weight matrix and a bias vector, and then processed by a non-linear activation function to obtain the feature vector of the node in the next layer. After a preset number of message passing iterations, the feature vector of the task target node is determined as the graph embedding vector.

3. The method according to claim 1, characterized in that, The step of generating the forced terms of the dynamic motion primitives based on the forced term weight coefficients includes: The weight coefficients of the forced term are weighted and summed with the corresponding Gaussian functions, and then multiplied with the phase variable describing the motion progress to obtain the forced term. The center parameter and width parameter of the Gaussian function are preset values.

4. The method according to claim 1, characterized in that, The physical spacetime boundary includes the spatial coordinates, velocity vectors, and safety distance threshold attributes of both parties. The safety distance is calculated by subtracting the sum of the equivalent radius of the robot link enclosure, the equivalent radius of the human safety envelope, and the preset safety margin from the Euclidean distance between the center point coordinates of the robot link enclosure and the key point coordinates of the human skeleton.

5. The method according to claim 4, characterized in that, The alternative control command is generated based on the current interference direction and the preset safety level rules, and the alternative control command includes at least one of detour trajectory and deceleration stop command.

6. The method according to claim 1, characterized in that, Also includes: The object image features and contact force features acquired by the sensor are input into a pre-trained feature extraction network to obtain the object feature vector; The similarity between the object feature vector and the feature vector of the existing nodes in the dynamic subgraph is calculated one by one. If the similarity is determined to be lower than the preset threshold, the corresponding object is determined to be an unknown object. The feature vector of the unknown object is injected into the dynamic subgraph as a new node. Candidate edges are established with existing nodes based on the spatial position and geometric contact relationship of the new node. The relationship type of the candidate edges is marked as pending confirmation. Using the local subgraph state where the new node is located as the observation state of the reinforcement learning environment, the pre-trained policy network is invoked to output a trial action, the robot is controlled to execute the trial action, and the reward value is calculated based on the execution result. After repeatedly executing the output trial action and calculating the reward value, and after setting the number of trial rounds, sort the interaction trajectories according to the cumulative reward value, and determine the interaction trajectory that is sorted first as the successful operation trajectory. The successful operation trajectory is encoded as a physical operation token, written into the attribute field of the new node, and the relationship type of the candidate edge is updated from pending confirmation to the relationship type confirmed according to the successful operation trajectory.

7. The method according to claim 6, characterized in that, The reward value is calculated by weighting the task completion rate, safety penalty items, and energy consumption penalty items. The safety penalty items are determined based on the risk score obtained from the safety verification, and the energy consumption penalty items are calculated based on the robot joint torque integral.

8. The method according to claim 6, characterized in that, Also includes: The updated local subgraph structure of the dynamic subgraph is synchronized to the cloud static process knowledge graph, and after review, it is incorporated into the cloud static process knowledge graph for reuse in subsequent tasks.

9. A knowledge graph-based intelligent control system for industrial robots, characterized in that, This includes a cloud-based knowledge graph storage module, an edge controller, and a robot execution module; The cloud-based knowledge graph storage module is used to store the plant-wide static process knowledge graph. The edge controller includes a dynamic subgraph construction unit, a graph neural network inference unit, a mapping unit, a security verification unit, and an online learning unit, for implementing the method of any one of claims 1 to 8; The robot execution module includes a servo driver, which is used to receive verified control commands issued by the edge controller and execute actions.

Images