An industrial robot cluster control method and system combining physical constraints, vPLC and multi-agent reinforcement learning

Abstract

The application provides a kind of industrial robot cluster control method and system of fusion physical constraint vPLC and multi-agent reinforcement learning, it is related to industrial automation and intelligent manufacturing technical field, this method includes: constructing virtualization control layer;Define agent, and respectively construct depth reinforcement learning model for each agent;All depth reinforcement learning models are trained centrally, and the loss function of each model is used to constrain the action instruction output by policy network, so that it meets the preset robot kinematics limit and / or dynamics limit;Each agent generates control strategy using depth reinforcement learning model according to local observation information;According to the control strategy, the control instruction is issued to the physical industrial robot through the corresponding vPLC instance.The application introduces kinematics constraint penalty term, dynamics constraint penalty term and workspace boundary constraint penalty term in the loss function, so that the action instruction output by the depth reinforcement learning model meets the physical feasibility requirement from the source.

Description

Technical Field

[0001] This application relates to the field of industrial automation and intelligent manufacturing technology, specifically to an industrial robot swarm control method and system that integrates physically constrained vPLC and multi-agent reinforcement learning. Background Technology

[0002] Industrial robot swarms undertake a variety of complex tasks in modern intelligent manufacturing, such as assembly, welding, handling, and sorting. The performance of their control systems directly affects the efficiency, flexibility, and reliability of production lines. With the development of the Industrial Internet and software-defined manufacturing technologies, industrial robot control systems are evolving from traditional dedicated hardware control towards virtualization and intelligence, giving rise to virtual programmable logic controllers (vPLCs). vPLCs decouple control logic from dedicated hardware by running the control functions of traditional hardware PLCs in software on a general-purpose computing platform, providing a technological foundation for the flexible deployment and dynamic reconfiguration of industrial robot control systems. Meanwhile, reinforcement learning, as an important branch of artificial intelligence, has shown promising application prospects in single-robot intelligent control. However, applying reinforcement learning to industrial robot swarm control still faces the challenge of multi-agent collaborative decision-making.

[0003] To address the aforementioned challenges in collaborative control, current industrial robot swarm control systems generally employ a centralized architecture. This involves a central controller that collects state information from all robots, centrally calculates control commands, and issues them for execution. Building upon this, some research has incorporated reinforcement learning into this architecture to enhance system intelligence. Simultaneously, to overcome the inherent limitations of centralized architectures in terms of scalability, some research has begun exploring distributed reinforcement learning methods. By distributing the decision-making and learning processes across robot nodes, these methods aim to achieve collaborative control and autonomous decision-making among multiple robots, thereby improving the system's computational parallelism and fault tolerance.

[0004] Although distributed reinforcement learning has alleviated the pressure on centralized architecture to some extent, existing distributed training processes generally do not take into account the actual physical constraints of industrial robots (such as joint angle limits, speed limits, torque limits, etc.). This may result in the generated control strategy outputting physically infeasible instructions when executed by real equipment. This could affect the accuracy of the operation or even cause equipment overload or mechanical collisions, resulting in serious safety hazards. Summary of the Invention

[0005] In view of this, this application provides an industrial robot swarm control method and system that integrates vPLC with physical constraints and multi-agent reinforcement learning, in order to solve the problem that existing distributed reinforcement learning methods do not fully consider the physical constraints of the equipment in multi-robot collaborative training, which may lead to physical infeasibility such as joint over-limit and speed / torque over-limit during actual execution, and even cause safety hazards such as equipment damage.

[0006] In a first aspect, embodiments of this application provide an industrial robot swarm control method that integrates physically constrained vPLC with multi-agent reinforcement learning, including: Construct a virtualized control layer based on a virtual programmable logic controller (vPLC); The vPLC instance in the virtualization control layer and the physical industrial robot it controls are defined as intelligent agents, and a deep reinforcement learning model is constructed for each intelligent agent. All deep reinforcement learning models are trained in a centralized manner, and the loss function of the deep reinforcement learning model is used to constrain the action instructions output by the policy network to conform to preset robot kinematic constraints and / or dynamic constraints. Each agent generates a control strategy based on local observation information through a trained deep reinforcement learning model; Based on the control strategy, control commands are issued to the physical industrial robot through the corresponding vPLC instance.

[0007] In one possible implementation, all deep reinforcement learning models are trained centrally, wherein the loss function of the deep reinforcement learning models is used to constrain the action commands output by the policy network to conform to preset robot kinematic and / or dynamic constraints, including: The deep reinforcement learning model is trained using global state information, with the goal of maximizing the preset global cumulative reward, and the policy network parameters of each agent are optimized. The preset global cumulative reward is defined as the sum of the discounted cumulative rewards of all agents over the entire task cycle; The loss function includes kinematic constraint penalties for constraining the motion command to meet joint angle range limits and end-effector pose reachability space limits, dynamic constraint penalties for constraining the motion command to meet joint velocity limits, joint acceleration limits and joint torque limits, and workspace boundary constraint penalties for constraining the end-effector to be within a preset safe workspace range.

[0008] In one possible implementation, adaptive communication based on an attention mechanism is introduced between the agents, where the first agent's attention weight is assigned to the communication information sent by the second agent. Calculate using the following formula: in, and For the hidden state vector, and It is a learnable linear transformation matrix. Let be the dimension of the hidden state vector; This indicates that the softmax normalization operation is performed on the dimension containing the second agent; T is the transpose. When focusing on weight Greater than or equal to the preset communication threshold At that time, the first intelligent agent receives and processes communication information from the second intelligent agent.

[0009] In one possible implementation, the construction of the virtualized control layer based on a virtual programmable logic controller (vPLC) includes: The vPLC instances are deployed on edge computing nodes. Each vPLC instance establishes a deterministic communication link with the physical industrial robot through the real-time industrial Ethernet protocol and collects the operating status data of the physical industrial robot. The vPLC instance collects environmental perception data through visual sensors and lidar sensors deployed in the work area. Establish a mapping relationship between vPLC instances and physical industrial robots.

[0010] In one possible implementation, the mapping relationship between the vPLC instance and the physical industrial robot includes: Each vPLC instance controls one physical industrial robot; or, Each vPLC instance controls multiple physical industrial robots; this vPLC instance is defined as a multi-armed intelligent agent; or, Multiple vPLC instances collaboratively control the same physical industrial robot, and these multiple vPLC instances are combined into a single logical intelligent agent.

[0011] In one possible implementation, the deep reinforcement learning model includes a policy network and a value network. The policy network takes the agent's local observation information as input and outputs the conditional probability distribution of each action instruction of the agent. The value network takes the global state information and the joint actions of all agents as input and outputs an estimate of the state-action value.

[0012] In one possible implementation, during the centralized training of all deep reinforcement learning models, the parameters of each policy network are updated using a policy gradient algorithm, and the update direction is determined according to the following gradient formula: in, Indicates policy network parameters The gradient; Indicates the sampling trajectory Seeking expectations; This indicates that the policy network provides local observation information. Select Action Command The logarithm of the probability of the policy network parameters The gradient; For the first The advantage function of each agent.

[0013] In one possible implementation, the advantage function is calculated using a counterfactual baseline method: in, For the first Each agent has global state information and joint actions The value of the following; Indicates the first An agent observes information locally. Next, select action instructions from its policy network. ; For the first Each agent has global state information and joint actions The value of time indicates the value of the first time. The action commands of each agent are replaced with Action commands from other intelligent agents Value when it remains unchanged.

[0014] One possible implementation also includes: Collect operational status feedback data of industrial robot clusters, construct state transition experience data and store it in the experience replay buffer; Experience data is sampled from the experience replay buffer, and incremental training is performed on each deep reinforcement learning model to update the parameters of each policy network. The updated parameters will be synchronously deployed to each vPLC instance after security verification.

[0015] Secondly, embodiments of this application provide an industrial robot swarm control system that integrates a physically constrained vPLC with multi-agent reinforcement learning. The control system is used to implement the control method described in the first aspect, including: The virtualization control module, deployed on edge computing nodes, is used to build a virtualization control layer based on virtual programmable logic controllers (vPLCs). The multi-agent reinforcement learning decision-making module includes a centralized training unit and a distributed inference unit. The centralized training unit is used to define the vPLC instance in the virtualization control layer and the physical industrial robot it controls as an agent, and to construct a deep reinforcement learning model containing a policy network for each agent; the centralized training unit is also used to perform centralized training on the deep reinforcement learning model of each agent, and the loss function of the deep reinforcement learning model is used to constrain the action instructions output by the policy network to conform to preset robot kinematic constraints and / or dynamic constraints. The distributed inference unit is deployed in each vPLC instance to enable each agent to generate control strategies based on local observation information through a trained deep reinforcement learning model. The execution module is used to convert the control strategy into control instructions through the corresponding vPLC instance and issue them for execution.

[0016] The beneficial effects of this application are as follows: 1. This application deeply integrates virtual programmable logic controller technology with multi-agent reinforcement learning algorithms to construct a distributed intelligent control architecture of "centralized training in the cloud and decentralized execution at the edge", which realizes efficient collaborative control of industrial robot clusters.

[0017] 2. This application migrates the PLC control functions that traditionally rely on dedicated hardware to a general computing platform through containerized deployment of vPLC instances and software definition of control logic. This enables on-demand allocation, dynamic reconfiguration, and flexible expansion of control resources, significantly reducing the deployment cost and operational complexity of industrial robot cluster control systems, while improving the system's rapid response capability in the face of production line adjustments and process changes.

[0018] 3. This application introduces a multi-agent reinforcement learning algorithm, enabling each industrial robot to make autonomous decisions based on local observation information. It can achieve cluster collaborative operation without relying on the unified scheduling of a central controller, effectively avoiding the single point of failure risk and communication bottleneck problem existing in the traditional centralized control architecture.

[0019] 4. This application embeds physical information regularization constraints during reinforcement learning training, enabling the generated cooperative control strategy to automatically meet the kinematic constraints, dynamic constraints, and workspace boundary constraints of the industrial robot, thus ensuring the physical feasibility and execution safety of the control strategy at the algorithm level.

[0020] 5. This application enables each agent to dynamically select communication objects and communication content through an adaptive communication module based on an attention mechanism, thereby significantly reducing the communication bandwidth consumption between multiple robots while ensuring collaborative efficiency.

[0021] 6. This application constructs a closed-loop optimization mechanism of experience playback and incremental training, enabling the control system to continuously utilize newly generated operating data for strategy iteration during actual operation, thereby continuously improving control performance and possessing adaptive optimization capabilities for complex dynamic working conditions. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of the deployment environment for an industrial robot swarm control method that integrates physically constrained vPLC and multi-agent reinforcement learning, provided in an embodiment of this application; Figure 2 A flowchart illustrating an industrial robot swarm control method that integrates physically constrained vPLC with multi-agent reinforcement learning, provided for embodiments of this application; Figure 3 A flowchart illustrating the optimization steps of an industrial robot swarm control method that integrates physically constrained vPLC and multi-agent reinforcement learning, provided in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of an industrial robot swarm control system that integrates a vPLC with physical constraints and multi-agent reinforcement learning, provided as an embodiment of this application. Detailed Implementation

[0024] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0025] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0026] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims should also be understood to include the plural forms unless the context clearly indicates otherwise.

[0027] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0028] This embodiment provides an industrial robot swarm control method that integrates a vPLC with physical constraints and multi-agent reinforcement learning. This method combines virtualized programmable logic controller technology with multi-agent reinforcement learning algorithms to achieve distributed intelligent control of industrial robot swarms.

[0029] The deployment environment of this embodiment is as follows: Figure 1 As shown, it mainly includes three layers: cloud server layer (i.e., cloud layer), edge computing node layer (i.e., edge layer), and industrial robot cluster execution layer (i.e., device layer).

[0030] The cloud server layer is configured with a high-performance computing cluster (in this embodiment, 8 NVIDIA A100 GPUs are used as centralized training units) to perform centralized training tasks for multi-agent reinforcement learning models and to distribute control policies to the edge computing node layer.

[0031] The edge computing node layer is deployed in the industrial field and runs multiple vPLC instances (i.e., vPLC instance 1, vPLC instance 2, ..., vPLC instance N). Each vPLC instance establishes a deterministic communication connection with one or more physical industrial robots (i.e., robot 1, robot 2, ..., robot N) through the real-time industrial Ethernet protocol.

[0032] The industrial robot cluster execution layer consists of several six-degree-of-freedom serial physical industrial robots. Each robot is equipped with joint angle sensors, joint torque sensors, end-effector pose sensors, and a vision perception module to feed back operational status data to the edge computing node layer.

[0033] In this embodiment, the EtherCAT protocol is selected as the real-time industrial Ethernet protocol, and the communication period is set to 1ms to ensure deterministic data exchange between the vPLC instance and the physical industrial robot.

[0034] like Figure 2 As shown, the detailed solution of this embodiment is as follows.

[0035] S101 constructs a virtualized control layer based on a virtual programmable logic controller (vPLC).

[0036] This embodiment deploys multiple vPLC instances on edge computing nodes. Each vPLC instance is implemented using a lightweight software encapsulation method, that is, the ladder diagram logic program, motion control algorithm, and safety interlock logic in the traditional PLC are converted into a structured text program for encapsulation. Each vPLC instance establishes a deterministic communication link through the EtherCAT master station protocol stack to collect the following operating status data in real time: Joint angle ,in , representing the first physical industrial robot The joint angles of each joint; Joint angular velocity ,in , representing the first physical industrial robot The joint angular velocity of each joint; Joint angular acceleration ,in , representing the first physical industrial robot Joint angular acceleration of each joint; End effector pose ,in, The three-dimensional position coordinates of the end effector tool center point TCP in the robot base coordinate system or world coordinate system; The attitude angle of the end effector relative to the same reference coordinate system, which is the same as the reference coordinate system used for the three-dimensional position coordinates. This embodiment uses ZYX Euler angles, where... This represents the yaw angle about the Z-axis. Indicates the pitch angle about the Y-axis. This indicates the roll angle about the X-axis.

[0037] The transpose symbol indicates It is a column vector.

[0038] Joint torque ,in , representing the first physical industrial robot Joint torque of each joint.

[0039] This embodiment collects environmental perception data, including visual image data and lidar point cloud data, by deploying visual sensors and lidar sensors in the work area.

[0040] A mapping relationship is established between vPLC instances and physical industrial robots. In this embodiment, the mapping relationship between vPLC instances and physical industrial robots adopts a one-to-one correspondence, that is, each vPLC instance controls one physical industrial robot. It should be noted that in other embodiments of this application, the following mapping relationship can also be adopted: each vPLC instance controls multiple physical industrial robots, or multiple vPLC instances collaboratively control the same physical industrial robot.

[0041] S102 defines the vPLC instance in the virtualization control layer and the physical industrial robot it controls as an intelligent agent.

[0042] In this embodiment, each vPLC instance in the virtualization control layer and the physical industrial robot it controls are collectively defined as an independent intelligent agent; let the total number of intelligent agents in the system be... In this embodiment, we take This corresponds to a cluster of six industrial robots.

[0043] In other embodiments of this application, when each vPLC instance controls multiple physical industrial robots, the vPLC instance and the multiple physical industrial robots it controls are defined as a multi-armed intelligent agent, whose action space is expanded to a joint representation of the action spaces of each robot; when multiple vPLC instances collaboratively control the same physical industrial robot, these vPLC instances and the physical industrial robots they control are combined into a logical intelligent agent, whose joint action space is the Cartesian product of the action spaces of each vPLC instance.

[0044] Construct deep reinforcement learning models for each agent, wherein the deep reinforcement learning models include: a policy network. and value network .

[0045] The policy network With the first Local observation information of an agent Input, output action command The conditional probability distribution, with parameters as .

[0046] For the An intelligent agent, its local observation information include: in, Indicates the first The joint angles of the 1st to 6th joints of a physical industrial robot controlled by an intelligent agent; Indicates the first The joint angular velocities of the 1st to 6th joints of a physical industrial robot controlled by an intelligent agent; Corresponding to the pose of the end effector of a physical industrial robot, where, The three-dimensional position coordinates of the end effector tool center point TCP in the robot's base coordinate system or world coordinate system. The attitude angle of the end effector relative to the same reference coordinate system; For the first The minimum distance between the end effector of a physical industrial robot controlled by an agent and the nearest obstacle or other physical industrial robot; T represents transpose.

[0047] Action commands Defined as the incremental displacement command for the end effector of a physical industrial robot: in, , , These represent the normalized components of the position increment of the end effector along the X, Y, and Z axes of the reference coordinate system, respectively. , , These represent the normalized components of the attitude increments of the end effector about the X, Y, and Z axes of the reference coordinate system; the value range of each component is normalized to an interval. ;T indicates transpose.

[0048] The reference coordinate system can be the robot base coordinate system, the world coordinate system, or the end-effector coordinate system. In this embodiment, to ensure consistency with the inverse kinematics solution, the reference coordinate system is the robot base coordinate system.

[0049] The action command Output by the policy network. Specifically, the policy network outputs the original action vector. Compressed to using the tanh activation function The interval is used to obtain action instructions. : During the training phase, the Gaussian distribution of the policy network output can be sampled and compressed using the tanh activation function. During the distributed execution phase, the mean of the policy network output can be directly compressed using the tanh activation function and used as deterministic action instructions to reduce inference jitter.

[0050] Furthermore, the action command Mapped to actual physical increments via physical increment scaling factor : in, It is the physical increment scale matrix, and it is a diagonal matrix. Indicates the diagonal; These represent the maximum permissible position increment of the end effector along the X, Y, and Z axes, respectively. These represent the maximum permissible attitude increments of the end effector per step around the X, Y, and Z axes, respectively.

[0051] These are collectively referred to as physical increment scale factors, which can be determined based on the robot model, control cycle, and safe speed limit. For example, the control cycle is... When the maximum permissible linear velocity at the end is ,but If the maximum permissible angular velocity at the end is ,but The values ​​above are merely examples, and this application is not limited to them.

[0052] Therefore, the actual physical increment can be expanded as follows: After obtaining the actual physical increment, the target pose is calculated from the current end effector pose. Let the current end effector pose be... for: in, The current three-dimensional position coordinates of the end effector tool center point TCP in the robot's base coordinate system or world coordinate system; This represents the current attitude angle of the end effector relative to the same reference coordinate system.

[0053] Its location part is The attitude part corresponds to the rotation matrix. Then the target location for: The target pose is obtained by rotation increment synthesis: in, Let this be the attitude rotation vector; This represents the antisymmetric matrix formed by the rotation vector; The exponential mapping can be represented and calculated using the Rodrigues formula; Let be the target rotation matrix.

[0054] Subsequently, by the target rotation matrix Extract the target Euler angles or target quaternions to obtain the target pose. .

[0055] This embodiment uses rotation matrices or quaternions for attitude increment synthesis, which can effectively avoid the errors introduced by directly adding Euler angles near large angles or singular configurations.

[0056] Then, the target pose is calculated using inverse kinematics. Convert to joint angle command : in, This is the current joint angle vector. This represents the inverse kinematics solution function. The inverse kinematics solution can be calculated using analytical or numerical methods.

[0057] For a six-DOF serial industrial robot, the analytical method is preferred for solving the problem. After obtaining multiple sets of inverse solutions, the solution is selected based on the current joint angle vector. The set of solutions that minimizes the Euclidean distance and satisfies the joint constraint condition is used as the joint angle command. .

[0058] If the analytical solution does not exist or falls into a singular configuration, then a numerical iteration method is used to solve it, specifically: in, For the first The joint angle of the next iteration; For the first The joint angle of the next iteration; Here is the Jacobian matrix for the robot; T is the transpose. Let the positive kinematics function of the robot be denoted as . This is the damping coefficient, with a value ranging from 0.01 to 0.1, used to suppress excessive joint velocities near singular configurations; It is the identity matrix; The target pose.

[0059] The iteration terminates when the pose error is less than a preset threshold. Preset threshold The range of values ​​is The iterations may fail until the maximum number of iterations is reached. If the iterations do not converge, the inverse kinematics solution is considered to have failed.

[0060] In another implementation, the Jacobi pseudo-inverse can also be used directly for incremental inverse solution: in, This is the pseudo-inverse of the Jacobian matrix or the damped least-squares pseudo-inverse. This method has a relatively low computational cost and is suitable for high real-time scenarios with a control period of 1 ms.

[0061] After receiving the joint angle command Afterwards, safety constraint verification and amplitude limiting processing are required. Specifically, this includes: Joint angle limit verification: Joint speed limit verification: Joint acceleration limiting verification: in, For the first Minimum joint angle limit for each joint; For the first Joint angle commands for each joint; For the first Maximum joint angle limit for each joint; For the first The current joint angle of each joint; To control the period, for example, 1 ms; This is the upper limit of the joint angular velocity; This is the upper limit of joint angular acceleration; For the first The current joint angular velocity of each joint; To perform absolute value operations.

[0062] Joint torque verification: Estimate the joint torque required for the target motion based on the robot's dynamics model, and determine whether it exceeds the limit. , This represents the upper limit of the joint torque.

[0063] If any of the above constraints are not met, the joint increments are scaled proportionally: in, This is a safety scaling factor; This is the final joint angle command issued; Indicates angular deviation; Indicates angular deviation No. Each element.

[0064] If the safety constraints cannot be met after scaling, or the inverse kinematics calculation fails, the joint angle command of the previous control cycle is retained, or braking is performed at the preset maximum deceleration, and the abnormal status is reported to the upper-level vPLC instance.

[0065] Furthermore, to improve the smoothness of the movement, the final joint angle command can be adjusted. Perform first-order low-pass filtering: in, These are the filter coefficients; For the first The joint angle command after a first-order low-pass filter in each control cycle; For the first The joint angle command after a first-order low-pass filter in each control cycle; For the first The joint angle command in each control cycle was not processed by a first-order low-pass filter.

[0066] In other embodiments, the actual physical increment can also be used. Perform a moving average or low-pass filter before performing inverse kinematics calculation.

[0067] Ultimately, the vPLC instance will The commands are converted into joint angle control commands and sent to the servo drive of the physical industrial robot in the next control cycle via real-time industrial Ethernet protocols such as EtherCAT. The control cycle can be 1 ms, meaning the inference engine receives local observation information at a frequency of 1 kHz. Action instructions are obtained through forward reasoning via the policy network. After the above-mentioned normalization mapping, target pose synthesis, inverse kinematics calculation, safety verification, and smoothing processing, joint angle commands are generated and issued. This realizes a complete closed loop from multi-agent reinforcement learning action commands to joint-level control commands for physical industrial robots.

[0068] The value network With global state information Joint actions of all intelligent agents The input is a state-action value estimate, and the output is a state-action value estimate with parameters. .

[0069] The policy network adopts a three-layer fully connected neural network structure with hidden layer dimensions of 256 and 128, and the activation function is the ReLU function; the value network adopts a four-layer fully connected neural network structure with hidden layer dimensions of 512, 256 and 128, and the activation function is also the ReLU function.

[0070] The policy network includes an input layer, two hidden layers, and an output layer; the two hidden layers are fully connected layers with dimensions of 256 and 128 respectively, and the activation function for both is the ReLU function; the output layer is a fully connected layer, and its output dimension is determined according to the action space, used to output the conditional probability distribution parameters of each action instruction of the agent.

[0071] The value network comprises an input layer, three hidden layers, and an output layer; the three hidden layers are all fully connected layers with dimensions of 512, 256, and 128, respectively, and the activation function is ReLU; the output layer is a fully connected layer used to output the estimated state-action value.

[0072] S103 performs centralized training on all deep reinforcement learning models, wherein the loss function of the deep reinforcement learning model is used to constrain the action commands output by the policy network to conform to preset robot kinematic and / or dynamic constraints.

[0073] Among them, policy network loss Defined as: in, For policy entropy; The entropy regularization coefficient is used in this embodiment. ; To calculate the expected value; To perform logarithmic operations.

[0074] The strategy entropy can be defined as: Indicates the first Under the policy network of individual agents, Seeking expectations.

[0075] Advantage function Calculated using the counterfactual baseline method: in, For the first Each agent has global state information and joint actions The value of the following; Indicates the first An agent observes information locally. Next, select action instructions from its policy network. ; For the first Each agent has global state information and joint actions The value of time indicates the value of the first time. The action commands of each agent are replaced with Action commands from other intelligent agents Value when it remains unchanged.

[0076] Advantage function The value equals the value of the current joint action minus the expected value after replacing the agent's action with other possible actions. For a discrete action space, the above formula represents the value of the action instruction. The expectation can be simplified to summation; for continuous action spaces, the expected value can be approximated by Monte Carlo sampling or numerical integration.

[0077] Value network loss Defined as: in, For the first Each agent has global state information and joint actions The value of the following; For the first The target value of each agent; It indicates a desire for the expected value.

[0078] No. The target value of each agent Determine by the following formula: in, For the first Instant rewards for each agent; This is the discount factor, which ranges from 0 to 1. In this embodiment, it is set to... ; This provides global state information for the next time step. For the first Each agent has global state information and joint actions The value of what is below.

[0079] Target Value Network parameters Update using the soft update method: in, For soft update coefficients, in this embodiment, we take... .

[0080] In one implementation, the value network input includes joint actions. Therefore, the physical information regularization term can be calculated based on this joint action. This is to guide the value network to give a lower value estimate for joint actions that violate kinematic or dynamic constraints, at which point the total loss of the value network is reduced. for: In another implementation, the physical information regularization term Only the policy network loss is added, while the value network loss is only used. ,Right now: Training termination conditions include at least one of the following: reaching the maximum training iteration count of 10,000 epochs; the global cumulative reward improving by less than a preset threshold over several consecutive epochs; or the change in policy network parameters being less than a preset threshold, where the preset threshold ranges from 0.01 to 0.1. After training, the policy network parameters... The security and stability are verified in the simulation environment by the security verification gateway. After the verification is passed, the system is synchronously deployed to the lightweight model inference engine of each vPLC instance.

[0081] During the intensive training phase, this embodiment utilizes global state information. The deep reinforcement learning model is trained to optimize the policy network parameters of each agent with the goal of maximizing the preset global cumulative reward. .

[0082] The preset global cumulative reward Defined as the sum of the discounted cumulative rewards for all agents over the entire task cycle: in, The total number of agents; In this embodiment, the maximum number of decision steps per task cycle is... ; This is the discount factor, with a value ranging from 0 to 1. In this example, we take... ; For the first An intelligent agent in Instant rewards for every moment.

[0083] The first An intelligent agent in Instant rewards for moments It is determined by the weighted sum of the following four components: in, , , and These are the weighting coefficients. In this embodiment, the values ​​for each weighting coefficient are: , , , The aforementioned weighting coefficients are pre-configured using a grid search method; for Real-time task completion rewards; for Energy consumption penalty at any given moment; for Collision risk penalties at any time; for Consistency penalty at any given moment.

[0084] In the above formula, each term is defined as follows: The task completion reward items The computational model used to guide the robot's end effector towards the target position is shown below: in, This is a task completion reward item, a scalar quantity, with a value range of [value range missing]. The closer the end effector is to the target position, the larger its value becomes; it reaches its maximum value of 0 when it has fully reached the target position. Indicates the current time. The three-dimensional position coordinates of the center point TCP of the end effector tool of a physical industrial robot controlled by an intelligent agent in a reference coordinate system. In one embodiment, ,in These are the position coordinates of the center point TCP of the end effector tool of the physical industrial robot along the X, Y, and Z axes of the reference coordinate system, in meters; For the first The target position of the end effector of a physical industrial robot controlled by an intelligent agent. In one implementation, ,in, These are the position coordinates of the target point along the X, Y, and Z axes of the reference coordinate system, respectively.

[0085] is the L2 norm of the vector, i.e., the Euclidean distance, used to measure the spatial distance between the current endpoint position and the target position.

[0086] The energy consumption penalty item The calculation model used to reduce energy consumption during robot movement is shown below: in, For the first physical industrial robot Joint torques of each joint; For the first physical industrial robot The joint angular velocity of each joint; To perform absolute value operations.

[0087] The collision risk penalty item Used to prevent robots from colliding with obstacles or other robots, hour, ,otherwise, .

[0088] in, For the first The actual minimum distance between the end effector of a physical industrial robot controlled by an intelligent agent and the nearest obstacle or other physical industrial robot; A preset safe distance threshold is used for... Compare; when At that time, the judgment of the first The physical industrial robot controlled by the intelligent agent poses a collision risk and triggers a collision risk penalty item. .

[0089] The consistency penalty item The computational model used to promote the consistency of motion in multi-physics industrial robots is shown below: in, The consistency penalty term is a scalar with a value range of (-∞, 0]; The closer the speed of each agent is to the average speed of the cluster, the lower the consistency penalty. The larger the value, the greater the maximum value is 0; For the first The velocity vector of a physical industrial robot controlled by an intelligent agent; L2 norm is used to measure the L2 norm of the first-order numerator. The deviation between the speed of an individual industrial robot and the average speed of the cluster.

[0090] The average velocity vector of a cluster of physical industrial robots is calculated using the following formula: in, The total number of agents. For the first The velocity vector of a physical industrial robot controlled by an intelligent agent.

[0091] To achieve dimensional uniformity for each item, all items were normalized before weighted summation, mapping their respective numerical ranges to the interval [0, 1].

[0092] To address the issue that control strategies generated by multi-agent reinforcement learning in existing technologies may violate robot physical constraints, this embodiment introduces a physical information regularization term into the loss function during the intensive training phase. : in, , , The preset regularization weight coefficient is taken in this embodiment. , , ; This is a kinematic constraint penalty term; This is a penalty term for dynamic constraints; This is a workspace boundary constraint penalty term. In one implementation, the constraint penalty term takes the form of linear over-limit cumulative.

[0093] The definitions of each item are as follows: The kinematic constraint penalty item The calculation model used to constrain the action commands output by the intelligent agent to meet the joint angle range limitations and end-effector pose reachability space limitations of the industrial robot is shown below: in, and The first The upper and lower limits of the joint angle of each joint; For the first physical industrial robot The joint angles of each joint; This is for calculating the maximum value.

[0094] The dynamic constraint penalty term The calculation model for constraining the action commands output by the intelligent agent to meet the joint velocity limits, joint acceleration limits, and joint torque limits of the industrial robot is shown below: in, , , The first physical industrial robot The upper limit of joint angular velocity, the upper limit of joint angular acceleration, and the upper limit of joint torque for each joint; The first physical industrial robot The absolute value of the joint angular velocity of each joint; The first physical industrial robot The absolute value of the joint angular acceleration of each joint; The first physical industrial robot The absolute value of the joint torque of each joint; This is for calculating the maximum value.

[0095] The workspace boundary constraint penalty item The calculation model used to constrain the end effector to keep it within a preset safe working space is shown below: in, The spatial coordinates of the center point TCP of the robot's end effector tool; The center of the safe workspace; The radius of the safe workspace; This indicates the calculation of the L2 norm; This is for calculating the maximum value.

[0096] The safe working space radius It can be determined by human setting, geometric calculation, simulation calibration or online dynamic adjustment based on at least one of the following: robot model, end effector size, joint limit, accessible workspace, obstacle distribution, multi-robot cooperative safety distance, sensor measurement error, control cycle and safety level.

[0097] In one implementation, the safe workspace radius The working space radius that can be reached by the end effector of a physical industrial robot Subtract safety margin To obtain, that is: in, It is determined based on at least one of the following: end effector size, maximum position deviation within the control cycle, and minimum safe distance from obstacles.

[0098] In another implementation, the safe workspace radius Set it to a fixed value, for example, 0.85m.

[0099] In yet another implementation, the safe workspace radius It can be dynamically adjusted according to the robot's current speed.

[0100] In another implementation, the constraint penalty term adopts the form of over-limit squared penalty, that is, when the limit is exceeded, the penalty is calculated based on the square of the excess amount, thereby imposing a stronger penalty on actions that significantly exceed the limit.

[0101] In this embodiment, the kinematic constraint penalty term The calculation model is as follows: in, Let be the joint angle of the k-th joint of the physical industrial robot, which is obtained by inverse kinematics calculation from the motion command output by the policy network. , The first The upper and lower limits of the angle of each joint; To perform the maximum value operation, This indicates a truncation operation that takes a non-negative value. When the joint angle is within the limit range, this item is 0. When it exceeds the limit, the penalty is calculated based on the square of the excess amount.

[0102] The dynamic constraint penalty term The calculation model is as follows: in, , The first The joint angular velocity and joint angular acceleration of each joint are obtained by differential or inverse kinematics calculation based on the motion command; For the first The predicted torque of each joint is estimated based on the robot dynamics model; , , The first The upper limits of angular velocity, angular acceleration, and torque for each joint.

[0103] The workspace boundary constraint penalty item The calculation model is as follows: in, The spatial coordinates of the center point TCP of the robot's end effector tool are obtained through forward kinematics calculation based on the motion command. The center of the safe workspace; The radius of the safe workspace; This represents the L2 norm.

[0104] Workspace boundary constraint penalty when the end effector is within the safe workspace If the value is 0, and it exceeds the limit, the penalty is calculated as the square of the excess amount.

[0105] In calculating the physical information regularization term In this case, the mean of the action distribution output by the policy network can be taken, or the action instructions obtained by reparameterized sampling can be used. The joint angles are obtained after inverse kinematics calculation and forward kinematics calculation. Joint angle Joint angular acceleration Joint torque Spatial coordinates of the center point TCP of the robot's end effector tool. Thus, the kinematic constraint penalty term is calculated. Dynamic constraint penalty term Workspace boundary constraint penalty And enable the physical information regularization term to be backpropagated to the policy network parameters. If the inverse kinematics solution is not differentiable, the Jacobi pseudo-inverse or differentiable inverse kinematics approximation can be used to ensure that the gradient can propagate in the back.

[0106] The total loss of the policy network for: in, The policy network loss is determined using the policy gradient algorithm: in, For the first An agent observes information locally. Below, its policy network selects action instructions. The probability of; For policy network parameters; For the first The advantage function of each agent; For entropy regularization coefficients, in this embodiment, we take... .

[0107] The policy entropy is defined as: The advantage function Calculated using the counterfactual baseline method: in, For the first Each agent has global state information and joint actions The value of the following; Indicates the first An agent observes information locally. Next, select action instructions from its policy network. ; For the first Each agent has global state information and joint actions The value of time indicates the value of the first time. The action commands of each agent are replaced with Action commands from other intelligent agents Value when it remains unchanged.

[0108] For a continuous action space, the action instructions in the above formula... The expected value can be approximated using Monte Carlo sampling or numerical integration.

[0109] The total loss of the value network for: in, The calculation model for value network loss is as follows: in, For the first Each agent has global state information and joint actions The value below indicates the first The value network of each agent in the global state and joint actions The output is the state-action value estimate; For the first Value network parameters of each agent; For the target value: in, For the first Instant rewards for each agent; As the discount factor, in this embodiment, we take... ; This provides global state information for the next time step. For the first Each agent has global state information and joint actions The value of time.

[0110] Indicates the first The target value network of an intelligent agent, its structure and value network Same, parameters are The target value network parameters are updated using a soft update method: in, For soft update coefficients, in this embodiment, we take... .

[0111] In another implementation, the total loss function of the value network can also include a physical penalty term based on the value network output. ,Right now: in, Based on the construction of physical constraint violations in value network outputs and joint actions, the value network is guided to give lower value estimates for joint actions that violate physical constraints. In one implementation, It can be represented as: in, The preset upper limit of security value, This is an indicator function that indicates a joint action violates physical constraints. It takes the value 1 when the joint action violates kinematic, dynamic, or workspace boundary constraints, and 0 otherwise.

[0112] Therefore, during centralized training, under the constraint of physical information regularization terms, the policy network can reduce the probability that the generated cooperative control strategy violates the kinematic constraints, dynamic constraints and workspace boundary constraints of the industrial robot, thereby improving the physical feasibility and execution safety of the control strategy from the algorithm level.

[0113] During the centralized training of all deep reinforcement learning models, the parameters of each policy network are updated using the policy gradient algorithm, and the update direction is determined according to the following gradient formula: in, Indicates the sampling trajectory Seeking expectations; For the first The advantage function of each agent; The objective function of the policy network is represented by Regarding policy network parameters The gradient is the direction of the policy network parameter update. Indicates local observation information Next, the policy network selects action instructions. The logarithm of the probability with respect to the policy network parameters The gradient.

[0114] To address the credit allocation problem in a multi-agent environment, this embodiment employs the counterfactual baseline method to calculate the dominance function: When calculating the dominance function using the counterfactual baseline method, the value network requires corresponding parameter updates. The value network loss, target value, and target value network soft update method are the same as described above. During intensive training, the policy network and value network can be updated alternately or jointly. In one implementation, in each training step, a batch of empirical data is sampled from the empirical replay buffer, and the value network loss is calculated. And update the value network parameters Then, the advantage function is calculated based on the updated value network. Then calculate the policy network loss. And update the policy network parameters. .

[0115] In another implementation, the value network loss can also be calculated simultaneously. and strategy network loss The Adam optimizer was used to update the value network parameters respectively. and policy network parameters Through synchronous updates of the value network, the counterfactual baseline... The estimation accuracy gradually increases, thus providing the policy network with more accurate credit allocation signals. During the distributed execution phase, only the trained policy network parameters are used. In the lightweight model inference engine distributed to each vPLC instance, the value network does not participate in the inference computation during the distributed execution phase.

[0116] in, For the first Each agent has global state information and joint actions The value of the following; Indicates the first An agent observes information locally. Next, select action instructions from its policy network. ; For the first Each agent has global state information and joint actions The value of time indicates the value of the first time. The action commands of each agent are replaced with Action commands from other intelligent agents Value when it remains unchanged.

[0117] The Adam optimizer was used during training, and the learning rate was set to [value missing]. The batch size is 64, and the training iteration rounds are 10,000.

[0118] S104 introduces an adaptive communication module based on an attention mechanism during centralized training and distributed execution.

[0119] To reduce the communication bandwidth consumption between multiple agents, this embodiment introduces an adaptive communication module based on an attention mechanism during centralized training and distributed execution; The first agent on the first The attention weight of the communication information sent by each agent Calculate using the following formula: in, , The first The first intelligent agent and the first The hidden state vectors of each agent are extracted from the intermediate layer output of the policy network of each agent. , Let be a learnable linear transformation matrix, with dimensions of 1. In this embodiment , ; Let be the dimension of the hidden state vector; Indicates the first The softmax normalization operation is performed on the dimension of each agent; T is the transpose.

[0120] When focusing on weight Greater than or equal to the preset communication threshold At that time, the first The agent receives and processes data from the first intelligent agent. The communication information of each agent is used; otherwise, the communication information is ignored. In this embodiment, the preset communication threshold... The value is 0.1.

[0121] This mechanism enables each agent to dynamically select communication targets and communication content, significantly reducing the bandwidth consumption of communication between multiple robots while ensuring collaborative efficiency.

[0122] After S105 completes intensive training, the trained policy network parameters will be... A lightweight model inference engine deployed in each vPLC instance.

[0123] Each agent relies solely on its own locally perceived observation information. It independently generates collaborative control strategies through a policy network, without relying on global state information or communicating with other intelligent agents in real time, thus achieving distributed intelligent control.

[0124] The lightweight model inference engine deployed in each vPLC instance of S106 receives local observation information of the corresponding agent in real time at a frequency of 1 kHz. Call the trained policy network Perform forward reasoning and output the action command at the current moment. .

[0125] Action commands The inverse kinematics calculations convert the data into joint angle commands, which are then sent to the servo drive of the physical industrial robot via the EtherCAT deterministic communication link.

[0126] like Figure 3 As shown, in another embodiment of this application, the control method further includes the following optimization steps: S201, in the edge computing node, collects state transition experience data generated by each agent during operation according to a preset time period (every 10 minutes in this embodiment) or event triggering conditions (such as collision warning or task completion). Each piece of experience data is represented by a quadruple ( The data is stored in the form of an experience replay buffer, where: for Global state information at any given moment, including the joint states, end-effector poses, and sensor readings of all physical industrial robots; for The coordinated actions at all times, including all physical industrial robot motion commands; for The global reward at a given moment is defined as the sum of the instantaneous rewards of all agents at that moment: in, The total number of agents; For the first An intelligent agent in Instant rewards for every moment.

[0127] for The global state at any given moment.

[0128] The maximum capacity of the experience replay buffer is set to The empirical data is managed using a first-in, first-out (FIFO) queue structure.

[0129] S202 randomly samples a batch of state transition experience data from the experience replay buffer to perform offline incremental training on the deep reinforcement learning model. In this embodiment, the batch size is 128.

[0130] The training process uses the same policy gradient algorithm and loss function (including a physical information regularization term) as the initial training set to update the policy network parameters of each agent. and value network parameters The learning rate for incremental training is set to one-tenth of the initial training learning rate, i.e. This is to ensure stable optimization of the model based on existing strategies.

[0131] After S203 training is completed, the updated policy network parameters are verified through the security verification gateway. Safety and stability were verified in a simulation environment.

[0132] This embodiment uses the Gazebo robot simulation platform to build a verification environment, accurately simulating the kinematic and dynamic characteristics of an actual industrial robot. The verification content includes: Single robot motion range safety verification: Execute 1000 random motion commands continuously and check whether the joint angles are always within the limit range; Multi-robot collaborative safety verification: Simulate multi-robot collaborative operation scenarios and check for collisions; Task completion rate verification: Evaluate the strategy's task success rate and average completion time on standard test tasks.

[0133] After successful verification, the updated policy network parameters will be applied. The system is synchronously deployed to the inference engine of each vPLC instance, replacing the original model parameters and completing the closed-loop strategy update. If the verification fails, the original strategy parameters remain unchanged, and the verification failure log is recorded for manual analysis.

[0134] Corresponding to the above embodiments, such as Figure 4 As shown, this application also provides an industrial robot swarm control system 30 that integrates a vPLC with physical constraints and multi-agent reinforcement learning, for implementing the above-mentioned industrial robot swarm control method. The control system includes: The virtualization control module 301 is deployed on an edge computing node to run multiple vPLC instances. Each vPLC instance establishes a deterministic communication connection with at least one physical industrial robot via the real-time industrial Ethernet protocol (EtherCAT protocol is used in this embodiment). The main functions of this module include: Data acquisition function: Collects operating status data (joint angle, joint angular velocity, joint angular acceleration, end effector pose, joint torque) and environmental perception data (visual image data, LiDAR point cloud data, millimeter-wave radar data, infrared sensor data) of each industrial robot at a frequency of 1 kHz.

[0135] Control logic software-defined encapsulation function: Convert the control program in the traditional PLC into a standardized software module for encapsulation, supporting online modification, version management and hot update of the control logic.

[0136] Resource management function: The vPLC instance is uniformly scheduled through the orchestration platform to realize the dynamic allocation of control resources and fault migration.

[0137] The multi-agent reinforcement learning decision module 302 includes a centralized training unit 3021, a distributed reasoning unit 3022, and an agent communication unit 3023.

[0138] The centralized training unit 3021 is deployed on a cloud server and is used to perform the following operations: define each vPLC instance and the physical industrial robot it controls as an independent intelligent agent; construct a deep reinforcement learning model containing a policy network and a value network for each intelligent agent; train the deep reinforcement learning model using global state information during the centralized training phase, with the goal of maximizing the preset global cumulative reward, and optimize the policy network parameters of each intelligent agent; and introduce the physical information regularization term into the loss function during the training process.

[0139] The distributed inference unit 3022 is deployed in each vPLC instance and uses a lightweight inference engine to achieve efficient deployment of the policy network. This unit is used to enable each agent to independently generate a cooperative control policy based on its own perceived local observation information through the trained policy network during the distributed execution phase.

[0140] The intelligent agent communication unit 3023 adopts an adaptive communication topology based on an attention mechanism, and its attention weight is calculated in the same way as the method described above. The first agent on the first The attention weight of the communication information sent by each agent is calculated according to the following formula: in, , The first The first intelligent agent and the first The hidden state vectors of each agent are extracted from the intermediate layer output of the policy network of each agent. , Let be a learnable linear transformation matrix with dimensions respectively. In this embodiment , ; Let be the dimension of the hidden state vector; Indicates the first The softmax normalization operation is performed on the dimension of each agent; T is the transpose.

[0141] Only when focusing on weight Greater than or equal to the preset communication threshold At that time, the first The agent receives and processes data from the first intelligent agent. The communication information of the agent is used; otherwise, the communication information is ignored.

[0142] The execution module 303 is used to convert the collaborative control strategies generated by each agent into control instructions for the physical industrial robot through the corresponding vPLC instance, and send them to the physical industrial robot for execution. This module is also used to continuously collect the actual operating status feedback data of the industrial robot cluster, build state transition experience data and store it in the experience replay buffer, triggering the incremental training process. After the updated policy network parameters are verified for security, they are synchronized to the distributed inference unit of each vPLC instance to form a closed-loop optimization control loop.

[0143] To further illustrate the actual effect of the physical information regularization term in this embodiment, a comparative test was conducted, quantitatively comparing the constraint processing method of this embodiment with two common constraint processing methods in the prior art. It should be noted that this comparative test was completed in a simulation environment, and its purpose is to assist in illustrating the technical effects of this application, and is not intended to limit the scope of protection of this application.

[0144] (a) Test objects and test methods 1. Test Object The test object is a six-degree-of-freedom serial industrial robot (a simulation model was established based on the kinematic and dynamic parameters of the KUKA KR6 R900 robot), and its main physical constraint parameters are shown in Table 1.

[0145] Table 1. Schematic diagram of main physical constraint parameters The constraint handling methods for the comparison are as follows: Method 1: Employ an execution-layer amplitude limiting strategy. After the policy network outputs action commands, the joint angle commands are amplitude-limited, truncating values ​​exceeding joint limits to boundary values. Method 1 does not alter the loss and reward functions during training.

[0146] Method 2: Employing a reward function penalty strategy. A constraint penalty term is added to the reward function. When the output motion command causes joint angles, velocities, accelerations, or torques to exceed limits, a negative reward is given. The weights of each penalty term are determined after grid search optimization.

[0147] Method 3: This embodiment employs a physical information regularization method, which involves introducing physical information regularization terms into the loss of the policy network and the value network in accordance with the aforementioned method described in this embodiment. .

[0148] All three approaches use the same network structure (the policy network uses three fully connected layers with hidden layer dimensions of 256 and 128 respectively; the value network uses four fully connected layers with hidden layer dimensions of 512, 256 and 128 respectively), and the same optimizer (Adam, learning rate). The same training rounds (10,000 rounds) and the same random seed.

[0149] The test task is as follows: the robot end effector moves from the starting position to the target position, which is randomly generated within the safe workspace. 100 target points are randomly generated for each test, and the performance under each method is statistically analyzed.

[0150] Evaluation metrics include overall constraint violation rate, task completion rate, average single-step movement variation, and training process reward value fluctuation.

[0151] Overall constraint violation rate: The percentage of time steps in which any physical constraint (joint angle, joint velocity, joint acceleration, joint torque, workspace boundary) is violated during the execution of a complete task.

[0152] Task completion rate: The percentage of times the end effector successfully reaches the target position (position error less than 0.01 m) out of the total number of tests.

[0153] Average single-step action change: The average Euclidean distance between action commands in adjacent decision steps. The average single-step action change is used to measure the smoothness of the policy output; the smaller the value, the smoother the action.

[0154] Training reward fluctuation: The standard deviation of the cumulative reward over the last 1000 rounds of training. This training reward fluctuation measures training stability; a smaller value indicates more stable training.

[0155] Following the above testing method, each method was run 5 times independently under the same conditions, and the average value was taken as the final result. The specific data is shown in Table 2.

[0156] Table 2 Comparison of Test Results for Three Constraint Processing Methods As can be seen from Table 2, although Method 1 applied output-side limiting to the joint angles, the overall violation rate still reached 3.6% because the limiting operation only covered the angle constraints and not the velocity, acceleration, and torque constraints. At the same time, the limiting operation disrupted the consistency between the policy network output and the actual executed action instructions. During the training process, the estimation bias of the value network gradually accumulated, resulting in a relatively low task completion rate (78%) and poor action smoothness.

[0157] Method two introduces a constraint penalty term, enabling the policy network to learn to avoid illegal action instructions through reward signals during training. However, actual testing revealed that adjusting the penalty weights is quite sensitive; too low a penalty results in insufficient constraint effectiveness, while too high a penalty leads to a conservative policy. After adjusting multiple sets of weights, the overall constraint violation rate decreased to 3.8%, and the task completion rate increased to 81%, but there is still significant room for improvement in the constraint effectiveness.

[0158] Method 3 employs the physical information regularization method of this embodiment. Under the same training conditions, the overall constraint violation rate is reduced to 0.7%, the task completion rate is increased to 94%, and the smoothness of actions and training stability are also better than the first two methods. In actual testing, it was observed that the policy network of Method 3 exhibits a faster convergence speed in the early stage of training, and simulation interruptions caused by large fluctuations in output are rare during the training process.

[0159] To further explain (Kinematic constraint penalty) (Dynamic constraint penalty term) and The contributions of each of the (workspace boundary constraint penalty items) were considered. In this embodiment, an ablation test was conducted under the same conditions. After removing one of the items, the changes in overall performance were observed. The test results are shown in Table 3.

[0160] Table 3. Ablation test results of physical information regularization term As shown in Table 3, each of the three sub-constraints plays an irreplaceable role in reducing the overall constraint violation rate. (Removal of kinematic penalty term) Afterwards, the joint angle violation rate rose sharply from 0.4% to 7.8%, and the overall constraint violation rate rose to 3.1%; the dynamic constraint penalty item was removed. Subsequently, the overall violation rate for velocity, acceleration, and torque increased from 0.9% to 5.6%; the workspace boundary constraint penalty was removed. Subsequently, the workspace boundary violation rate increased from 0.3% to 6.2%. When the three sub-constraints work together, the overall constraint violation rate is controlled at 0.7%, and the absence of any one of them will cause the overall violation rate to rise to more than 2%.

[0161] Based on the results of the above comparative and ablation tests, the following conclusions can be drawn: First, under the same training conditions and testing tasks, the physical information regularization method (i.e., method three) in this embodiment has a comprehensive constraint violation rate of 0.7%, which is significantly lower than the execution layer limit (3.6%) and reward function penalty (3.8%). This shows that embedding physical constraints into the loss function in the form of regularization terms can continuously provide constraint guidance during the optimization process of the policy network, which is more effective than the two methods of output truncation and post-penalty.

[0162] Second, the physical information regularization method in this embodiment improves task completion rate (94%) and motion smoothness ( In terms of both physical constraints and training stability (18.5), this embodiment outperforms the other two methods, indicating that it does not sacrifice task execution efficiency and policy stability while ensuring that physical constraints are met.

[0163] Third, ablation tests show that the three sub-constraints in the physical information regularization term each play a role in different types of physical constraints, and the three work together to achieve the optimal constraint effect; none of them can be omitted.

[0164] The above comparative tests were conducted in the Gazebo simulation environment (version Gazebo 11, equipped with ROS Noetic, simulation control frequency of 1 kHz). The test results reflect relative performance differences. In actual industrial deployments, due to factors such as communication latency, sensor noise, and model uncertainty, the specific values ​​may vary, but the relative advantages and disadvantages of each method are expected to be consistent with the simulation test conclusions.

[0165] Furthermore, the results of this comparative test are only used to illustrate the technical effects of this application and should not be construed as limiting the scope of protection of this application in any way. Any adaptive adjustments made to the parameters by those skilled in the art, inspired by the concept of this application and tailored to specific application scenarios, are all within the scope of protection of this application.

[0166] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0167] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0168] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0169] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An industrial robot cluster control method fusing physical constraint vPLC and multi-agent reinforcement learning, characterized in that, include: Construct a virtualized control layer based on a virtual programmable logic controller (vPLC); The vPLC instance in the virtualization control layer and the physical industrial robot it controls are defined as intelligent agents, and a deep reinforcement learning model is constructed for each intelligent agent. All deep reinforcement learning models are trained in a centralized manner, and the loss function of the deep reinforcement learning model is used to constrain the action instructions output by the policy network to conform to preset robot kinematic constraints and / or dynamic constraints; Each agent generates a control strategy based on local observation information through a trained deep reinforcement learning model; Based on the control strategy, control commands are issued to the physical industrial robot through the corresponding vPLC instance.

2. The method according to claim 1, wherein, All deep reinforcement learning models are trained centrally. The loss function of the deep reinforcement learning model is used to constrain the action commands output by the policy network to conform to preset robot kinematic and / or dynamic constraints, including: The deep reinforcement learning model is trained using global state information, with the goal of maximizing the preset global cumulative reward, and the policy network parameters of each agent are optimized. The preset global cumulative reward is defined as the sum of the discounted cumulative rewards of all agents over the entire task cycle; The loss function includes kinematic constraint penalties for constraining the motion command to meet joint angle range limits and end-effector pose reachability space limits, dynamic constraint penalties for constraining the motion command to meet joint velocity limits, joint acceleration limits and joint torque limits, and workspace boundary constraint penalties for constraining the end-effector to be within a preset safe workspace range.

3. The industrial robot swarm control method integrating physically constrained vPLC and multi-agent reinforcement learning as described in claim 1, characterized in that, Introducing adaptive communication based on attention mechanism between agents, the first agent pays attention to the communication information sent by the second agent The following formula is calculated: in, and For the hidden state vector, and It is a learnable linear transformation matrix. Let be the dimension of the hidden state vector; This indicates that the softmax normalization operation is performed on the dimension containing the second agent; T is the transpose. When focusing on weight Greater than or equal to the preset communication threshold At that time, the first intelligent agent receives and processes communication information from the second intelligent agent.

4. The industrial robot swarm control method integrating physically constrained vPLC and multi-agent reinforcement learning as described in claim 1, characterized in that, The construction of the virtualized control layer based on the virtual programmable logic controller (vPLC) includes: The vPLC instances are deployed on edge computing nodes. Each vPLC instance establishes a deterministic communication link with the physical industrial robot through the real-time industrial Ethernet protocol and collects the operating status data of the physical industrial robot. The vPLC instance collects environmental perception data through visual sensors and lidar sensors deployed in the work area. Establish a mapping relationship between vPLC instances and physical industrial robots.

5. The industrial robot swarm control method integrating physically constrained vPLC and multi-agent reinforcement learning according to claim 4, characterized in that, The mapping relationship between the vPLC instance and the physical industrial robot includes: Each vPLC instance controls one physical industrial robot; or, Each vPLC instance controls multiple physical industrial robots; this vPLC instance is defined as a multi-armed intelligent agent; or, Multiple vPLC instances collaboratively control the same physical industrial robot, and these multiple vPLC instances are combined into a single logical intelligent agent.

6. The industrial robot swarm control method integrating physically constrained vPLC and multi-agent reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning model includes a policy network and a value network. The policy network takes the agent's local observation information as input and outputs the conditional probability distribution of each action instruction of the agent. The value network takes the global state information and the joint actions of all agents as input and outputs the estimated value of the state-action value.

7. The industrial robot swarm control method integrating physically constrained vPLC and multi-agent reinforcement learning according to claim 6, characterized in that, During the centralized training of all deep reinforcement learning models, the parameters of each policy network are updated using the policy gradient algorithm, and the update direction is determined according to the following gradient formula: in, Indicates policy network parameters The gradient; Indicates the sampling trajectory Seeking expectations; This indicates that the policy network provides local observation information. Select Action Command The logarithm of the probability of the policy network parameters The gradient; For the first The advantage function of each agent.

8. The industrial robot swarm control method integrating physically constrained vPLC and multi-agent reinforcement learning according to claim 7, characterized in that, The advantage function is calculated using the counterfactual baseline method: in, For the first Each agent has global state information and joint actions The value of the following; Indicates the first An agent observes information locally. Next, select action instructions from its policy network. ; For the first Each agent has global state information and joint actions The value of time indicates the value of the first time. The action commands of each agent are replaced with Action commands from other intelligent agents Value when it remains unchanged.

9. The industrial robot swarm control method integrating physically constrained vPLC and multi-agent reinforcement learning according to claim 1, characterized in that, Also includes: Collect operational status feedback data of industrial robot clusters, construct state transition experience data and store it in the experience replay buffer; Experience data is sampled from the experience replay buffer, and incremental training is performed on each deep reinforcement learning model to update the parameters of each policy network. The updated parameters will be synchronously deployed to each vPLC instance after security verification.

10. An industrial robot swarm control system integrating physically constrained vPLC and multi-agent reinforcement learning, the control system being used to implement the control method as described in any one of claims 1-9, characterized in that, include: The virtualization control module, deployed on edge computing nodes, is used to build a virtualization control layer based on virtual programmable logic controllers (vPLCs). The multi-agent reinforcement learning decision-making module includes a centralized training unit and a distributed inference unit. The centralized training unit is used to define the vPLC instance in the virtualization control layer and the physical industrial robot it controls as an agent, and to construct a deep reinforcement learning model containing a policy network for each agent; the centralized training unit is also used to perform centralized training on the deep reinforcement learning model of each agent, and the loss function of the deep reinforcement learning model is used to constrain the action instructions output by the policy network to conform to preset robot kinematic constraints and / or dynamic constraints. The distributed inference unit is deployed in each vPLC instance to enable each agent to generate control strategies based on local observation information through a trained deep reinforcement learning model. The execution module is used to convert the control strategy into control instructions through the corresponding vPLC instance and issue them for execution.

Images