Coal gangue lateral dial-off control method based on MAGS-PPO

Abstract

The invention discloses a coal gangue lateral shifting control method based on MAGS-PPO, and the method comprises the steps: collecting the joint angle of a mechanical arm, the speed of a conveyor belt and the position of coal gangue given by a camera in real time, and constructing a state space after coordinate transformation and normalization; a joint angle increment is used as a continuous action space, and a composite reward function of approaching potential energy, dialling potential energy, sparse events and safety punishment is designed; establishing a multi-agent parallel simulation environment in Unity, updating a strategy network by adopting asynchronous sampling, GAE advantage estimation and PPO-clip, and introducing staged curriculum learning and randomization of physical parameters such as a friction coefficient and gangue quality; after training convergence, a strategy model is deployed to a real mechanical arm controller through ONNX, a lateral shifting action instruction is output on line, a mechanical arm is driven to shift moving gangues into a recovery area, and efficient and safe sorting under the condition of a high-speed conveying belt is achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot intelligent control and reinforcement learning, and particularly relates to a coal gangue lateral pushing away control method based on MAGS-PPO. BACKGROUND

[0002] Coal is the main energy source in China, and coal gangue is a byproduct in the process of coal mining and washing. Efficient separation of coal gangue is a key link to improve the utilization rate of coal and build a green mine. At present, the sorting operation of coal gangue mainly relies on manual selection, which has problems such as poor working environment, high labor intensity, and high safety risk. With the advancement of intelligent construction of coal mines, the use of coal gangue sorting robots to replace manual work has become an inevitable trend in the industry.

[0003] Most existing sorting robots use a control mode of "machine vision positioning + inverse kinematics solution + grasping and placing". However, the underground environment has highly unstructured characteristics, and the shape of coal gangue is extremely irregular, the size difference is large, and it is randomly stacked on the conveyor belt. The traditional grasping method has very high requirements for visual positioning accuracy and end effector pose planning. Once there is a slight deviation in calculation or the friction characteristics of the object surface change, it is easy to cause grasping failure or sliding. In addition, the operation process of "grasping-lifting-moving-placing" is complicated and the action rhythm is slow, which is difficult to meet the real-time sorting needs of high-speed large-capacity conveyors. In contrast, the lateral pushing away action is more in line with the characteristics of high-speed flow sorting, but it involves complex nonlinear contact dynamics, which is difficult to accurately plan through traditional mathematical models.

[0004] Although deep reinforcement learning provides an end-to-end solution to the difficulty of contact dynamics modeling, its application to real robotic sorting still faces great challenges. Reinforcement learning relies on a large amount of trial-and-error interaction data. If it is directly trained on a real physical prototype, not only is the time cost high, but the random actions in the initial exploration stage are also likely to cause the robot to collide and be damaged. Coal gangue sorting is a long sequence decision-making process, and the agent can only get positive feedback when the object successfully enters the recycling area. In the case of high-dimensional and continuous action space, this sparse reward mechanism will cause the policy network to fail to obtain effective guidance for a long time in the early training stage, causing training divergence or falling into a local optimum, and making it difficult to learn precise pushing away skills.

[0005] In the multi-agent parallel training architecture, how to balance the exploration of unknown states and the use of known high-value experiences is a difficult problem. Traditional proximal policy optimization algorithms often lack focus on key successful samples when dealing with sorting tasks, resulting in low learning efficiency, and fixed entropy regularization coefficients are difficult to adapt to different needs for policy certainty at different stages of training.

[0006] In summary, there is an urgent need for a robotic arm control method that can adapt to unstructured dynamic environments, has high robust transferability from simulation to reality, and can efficiently complete lateral pushing operations. SUMMARY

[0007] In view of the above technical deficiencies, the purpose of the present application is to provide a coal gangue lateral pushing control method based on MAGS-PPO. By constructing a multi-agent parallel training architecture and an improved PPO algorithm, combining a phased curriculum learning and domain randomization mechanism, efficient and compliant lateral pushing of irregular gangue in a complex dynamic coal flow environment is achieved.

[0008] To solve the above technical problems, the present application adopts the following technical solutions: the present application provides a coal gangue lateral pushing control method based on MAGS-PPO. The construction method comprises the following steps:

[0009] S1, real-time acquisition of current joint angle data of the robotic arm, running speed of the conveyor belt and position information of the target gangue in the camera coordinate system;

[0010] S2, normalization processing and relative coordinate conversion of the above raw data;

[0011] S3, establishing a reinforcement learning model based on Markov decision process, defining a state space S, constructing a continuous action space A based on joint angle increment, and designing a compound dense reward function R based on potential energy guidance;

[0012] S4, constructing a multi-agent parallel simulation environment, using the MAGS-PPO algorithm to iteratively train the policy network, including asynchronous sampling, GAE advantage estimation and PPO-clip update;

[0013] S5, using a phased curriculum learning strategy during training, sequentially performing contact guidance, dynamic pushing and generalization training;

[0014] S6, introducing a physical parameter randomization mechanism in the simulation environment, dynamically perturbing the environmental friction coefficient and object mass for training, and obtaining a converged policy network;

[0015] S7, deploying the trained policy network to a real robotic arm controller, outputting lateral pushing action instructions according to real-time state, and driving the robotic arm to perform lateral gangue pushing operation.

[0016] Optionally, the S1 specifically comprises the following contents:

[0017] S11, reading the current 6 joint angle values through the robotic arm controller bottom interface ;

[0018] S12, read the current conveyor line speed scalar through the conveyor encoder, and convert it to a velocity vector in the base coordinate system according to the running direction of the conveyor ;

[0019] S13, collect the conveyor image using the visual recognition system, and receive the original recognition result output by the visual recognition module, including the category, confidence and center point coordinates of the target gangue in the camera coordinate system .

[0020] Optionally, the S2 specifically includes the following contents:

[0021] S21, use the pre-calibrated homogeneous transformation matrix of the camera to the base of the robot arm to convert the coordinates of the target gangue from the camera coordinate system to the robot arm base coordinate system, and the calculation formula is: = , wherein = is the absolute position of the gangue in the robot arm base coordinate system

[0022] S22, use the D-H parameter model of the robot arm, and according to the current joint angle , calculate the current position of the robot arm end effector in the base coordinate system through forward kinematics = ;

[0023] S23, for the position vector g2t of the end effector relative to the gangue, that is:

[0024] ;

[0025] for the relative position vector t2g of the gangue to the center of the specified recycling area, that is:

[0026] ;

[0027] , wherein is a preset fixed coordinate point

[0028] S24, in order to eliminate the influence of different dimensions on the training of the neural network, the joint angle data in the state space is normalized and mapped to the interval [-1, 1], and the formula is:

[0029] ;

[0030] , wherein denotes the normalized i-th joint angle, which is used as the input feature of the agent, denotes the minimum joint angle allowed for the i-th joint ​represents the maximum joint angle allowed for the ith joint.

[0031] Optionally, the S3 specifically includes the following contents:

[0032] S31, construct a state space S, including normalized joint angles of the robot arm, a position vector g2t of the end effector relative to the gangue, a relative position vector t2g of the gangue relative to the center of the target recovery area, and a conveyor belt velocity vector ;

[0033] S32, construct a continuous action space A, define the action vector output by the policy network , where each component corresponds to the control instruction of one joint, and each action component is limited in the range of [-1, 1], and when actually executed, it is mapped to the joint angle increment , according to the reinforcement learning code written, the calculation formula is:

[0034] ;

[0035] where, represents the target angle increment of the ith joint actually output to the underlying controller; represents the original action control quantity of the ith joint output by the policy network; represents the preset joint reference angular velocity, is a single time step of the simulation environment, and this mechanism ensures the smoothness of the robot arm movement;

[0036] S33, in order to adapt to the phased course learning strategy, construct a reward function architecture containing an approach potential energy item, a push-off potential energy item, a sparse event item and a constraint penalty item, and the calculation formula is as follows:

[0037] ;

[0038] S34, potential energy guided reward item : use the difference between the distance at the last moment and the distance at the current moment to guide the action:

[0039] ;

[0040] where, represents the guided reward value calculated based on the distance potential energy difference; and respectively represent the Euclidean distance from the center of the target gangue to the center of the recovery area at the last moment t-1 and at the current moment t; and respectively represent the Euclidean distance from the center of the target gangue to the center of the recovery area at the last moment t-1 and at the current moment t; This represents the potential energy reward weighting coefficient during the disengagement phase. This represents the potential energy reward weighting coefficient for the approach phase. and As a phased adaptive adjustment factor, it is set during the contact-guided training phase. , 0, only guide the robotic arm to touch the target; during the dynamic manipulation training phase, set... , 1, of which This is a contact state indicator variable, indicating the state when the robotic arm comes into contact with the gangue. =1, otherwise 0;

[0041] S35, Sparse Event Rewards Used to provide a one-time strong incentive for key milestone events in a task, in order to reinforce successful behavior.

[0042] ;

[0043] in, It is the first contact indicator variable, which takes a value of 1 only when the end of the robotic arm first enters the contact radius of the target gangue and no contact has occurred before; otherwise, it is 0. This is the reward constant for the first contact; It is a task success indicator variable. It takes a value of 1 when the target gangue enters the recycling area and remains stable in the area for a certain period of time, and is 0 at other times. It is the reward constant for successful task completion;

[0044] S36, Constraints and Penalties Used to constrain invalid actions of intelligent agents and ensure operational safety.

[0045] ;

[0046] in, This is a penalty value for single-step time consumption, forcing the agent to complete the task in the fewest possible time steps; It is a safety collision indicator variable. It takes a value of 1 when a collision is detected between the robotic arm and the conveyor belt, except for the end gripper; otherwise, it takes a value of 0. It is the penalty constant for collisions in unsafe areas, used to suppress dangerous actions.

[0047] Optionally, S4 specifically includes the following:

[0048] S41. Create M independent robotic arm sorting scenarios in the simulation software Unity. Train the robotic arm using the ML-Agents plugin for reinforcement learning. Add an agent to each scenario, and all agents share the same policy network. and value networks ;

[0049] S42, a variant of the centralized training and decentralized execution architecture is adopted, each agent interacts with the environment in parallel in a respective randomized environment, outputs actions using a shared policy network, and independently collects experience data;

[0050] S43, in each round of update iteration, all agents perform T time step sampling in parallel, each agent uses a shared policy network according to the current state output action , interact with the environment to get reward and the next time state ;

[0051] S44, all trajectory data tuples generated by all agents are aggregated and stored in a global experience buffer to break the temporal correlation of single-agent sampling data;

[0052] S45, a shared value network is used to evaluate the value of each state and calculate the generalized advantage estimation value , the calculation formula is:

[0053] ;

[0054] ;

[0055] wherein, is the temporal difference error, is the discount factor, is the GAE smoothing factor;

[0056] S46, a weight coefficient is introduced:

[0057] ;

[0058] wherein, is a preset key sample adjustment constant;

[0059] S47, in order to balance exploration and utilization, a dynamic entropy coefficient is designed:

[0060] ;

[0061] wherein, is the proportion of task success in the current training batch; and are the maximum and minimum values of the entropy coefficient respectively; is the decay rate constant;

[0062] S48, construct a composite objective function containing an adaptive entropy regularization term and a key sample weighting term, update the policy network by maximizing the function:

[0063] ;

[0064] wherein, is the standard PPO clipping objective function; is the policy entropy; is the key sample weight coefficient, which is positively correlated with the absolute value of the advantage value , used to strengthen the learning of high-value experience; is the value function loss term, used to update the value network parameters; is the value loss coefficient; is the adaptive entropy coefficient, which is negatively correlated with the success rate of the current batch of tasks ; when is low, increase to encourage exploration, and when is high, reduce to speed up convergence.

[0065] Optionally, the S5 specifically includes the following contents:

[0066] S51, design a phased course learning strategy, first is the first phase: freeze the conveyor belt movement in the simulation environment, and set the target gangue as a kinematic fixed state;

[0067] S52, set the stage adjustment factor of the potential energy guidance reward term to =1, ; when the average contact success rate of the agent in the last N rounds exceeds the preset threshold , enter the next phase, which trains the agent to quickly identify and reach the target gangue in a static environment;

[0068] S53, design the second phase: restore the conveyor belt to run at a rated speed, remove the kinematic fixed state of the gangue, enable rigid body collision simulation of the physics engine, and randomly generate gangue positions in the entire workspace;

[0069] S54, set the stage adjustment factor to , and is dynamically related to the contact state, i.e. =(1- ), when the task success rate of the agent in the last N rounds exceeds the preset threshold , enter the next phase; this phase trains the agent to laterally push the moving gangue into the recycling area in a dynamic environment;

[0070] S55, design the third stage: on the basis of maintaining the dynamic environment of the second stage, superimpose the physical parameter domain randomization mechanism, at the beginning of each round, randomly sample and disturb the friction coefficient of the conveyor belt, the gangue mass, the illumination condition and the gangue pose;

[0071] S56, maintain the same reward weight configuration as the second stage, and introduce an adaptive entropy regularization mechanism, dynamically reduce the exploration noise according to the success rate, until the policy network converges completely.

[0072] Optionally, the S6 specifically includes the following contents:

[0073] S61, set the dynamic friction coefficient of the conveyor belt surface in Unity as a random variable to simulate the change of friction in real working conditions;

[0074] S62, set the mass of the target gangue as a random variable to simulate the inertia difference of gangues of different sizes and densities;

[0075] S63, set the light intensity and light source angle in Unity as random variables;

[0076] S64, at the beginning of each training round, randomly sample the initial generated position and rotation attitude of the target gangue on the conveyor belt;

[0077] S65, based on the above-mentioned constructed dynamic parameter space, perform adversarial training, and adjust the hyperparameters and other data until the average success rate of the policy network in the full parameter domain converges.

[0078] Optionally, the S7 specifically includes the following contents:

[0079] S71, after the policy network training is completed, first deploy the model in the simulation environment Unity, verify the effect of running, and ensure that the action can be successfully completed;

[0080] S72, export the trained model through the ONNX format, and deploy and apply it on the actual mechanical arm on the ROS side to drive the mechanical arm to perform operations.

[0081] The beneficial effects of the present application are:

[0082] Break through the limitations of unstructured grabbing: the present application discards the traditional "visual positioning-reverse solution-grabbing" mode, and adopts an end-to-end lateral pushing control strategy. Compared with precise grabbing, the lateral pushing action is not sensitive to visual errors, does not need complex grabbing point planning, can adapt to coal gangues of different shapes and stacking obstructions, and significantly improves the success rate and efficiency of sorting operations.

[0083] The convergence problem under sparse rewards is solved: by designing a compound dense reward function based on potential energy guidance, the present application converts long sequence sparse feedback into distance potential energy difference feedback at each step, effectively guiding the agent to quickly approach the target; combined with the key sample weighting mechanism, the learning of high-value successful experience is emphasized, which greatly shortens the training cycle.

[0084] The sample efficiency and exploration ability of the algorithm are improved: the proposed MAGS-PPO algorithm uses a multi-agent parallel architecture and a shared network mechanism to significantly improve sample throughput and break the time sequence correlation of samples; the adaptive entropy regularization mechanism introduced can dynamically adjust the exploration noise according to the task success rate, effectively avoiding local optimal trap in the early stage of training and strategy shock in the later stage.

[0085] The migration robustness from simulation to reality is improved: the cold start problem is solved by using a phased curriculum learning strategy, and the friction, mass and light fluctuation that may occur in the real environment are covered by using a physical parameter domain randomization mechanism. This makes the strategy trained in the simulation environment able to be directly migrated to the real physical system, overcoming the "reality gap" and having strong engineering practical value. BRIEF DESCRIPTION OF DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0087] Figure 1 is a flow chart of the coal gangue lateral pushing away control method based on MAGS-PPO provided by the embodiment of the present application;

[0088] Figure 2 is a mechanical arm ROC6 structure diagram provided by the embodiment of the present application;

[0089] Figure 3 is a multi-agent training scene screenshot provided by the embodiment of the present application;

[0090] Figure 4 is a comparison diagram of multi-agent and single-agent training results provided by the embodiment of the present application;

[0091] Figure 5 is a comparison diagram of experimental results of MAGS-PPO and ordinary PPO algorithm provided by the embodiment of the present application; DETAILED DESCRIPTION

[0092] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0093] Reference Figures 1-5 , the coal gangue lateral pushing away control method based on MAGS-PPO comprises the following steps:

[0094] S1, real-time acquisition of current joint angle data of the mechanical arm, running speed of the conveying belt and position information of the target gangue in the camera coordinate system;

[0095] S2, normalization processing and relative coordinate conversion of the position information;

[0096] S3, establishment of a reinforcement learning model based on Markov decision process, definition of a state space S, construction of a continuous action space A based on joint angle increments, and design of a compound dense reward function R based on potential energy guidance;

[0097] S4, construction of a multi-agent parallel simulation environment, iterative training of the strategy network using the MAGS-PPO algorithm, including asynchronous sampling, GAE advantage estimation and PPO-clip update;

[0098] S5, adoption of a phased curriculum learning strategy during training, and sequential performance of contact guidance, dynamic pushing and generalization training;

[0099] S6, introduction of a physical parameter randomization mechanism in the simulation environment, dynamic disturbance training of the environmental friction coefficient and object mass, and obtaining of a converged strategy network;

[0100] S7, deployment of the trained and converged strategy network to a real mechanical arm controller, output of lateral pushing away action instructions according to real-time states, and driving of the mechanical arm to perform lateral gangue pushing away operations.

[0101] In the present embodiment, S1 specifically comprises the following contents:

[0102] S11, as shown in Figure 2 , the present application performs experiments through an ROC6 mechanical arm, reads current 6 joint angle values through a mechanical arm controller bottom interface;

[0103] S12, reading of a current conveying belt linear velocity scalar through a conveying belt encoder, and conversion of the conveying belt linear velocity scalar into a velocity vector in the base coordinate system according to the conveying belt running direction ;

[0104] S13, acquiring a conveyor image by using a visual recognition system, receiving an original recognition result output by a visual recognition module, the original recognition result including a category of the target gangue, a confidence degree, and a center point coordinate in a camera coordinate system .

[0105] In this embodiment, S2 specifically includes the following content:

[0106] S21, using a pre-calibrated homogeneous transformation matrix of a camera to a robot base , converting the coordinates of the target gangue from the camera coordinate system to the robot base coordinate system, the calculation formula being: = , wherein = is the absolute position of the gangue in the robot base coordinate system;

[0107] S22, using a D-H parameter model of the robot arm, according to the current joint angle , calculating the current position of the robot arm end effector in the base coordinate system by forward kinematics = ;

[0108] S23, for the position vector g2t of the end effector relative to the gangue, that is:

[0109] ;

[0110] for the relative position vector t2g of the gangue to the center of the specified recycling area, that is:

[0111] ;

[0112] , wherein is a preset fixed coordinate point;

[0113] S24, in order to eliminate the influence of different dimensions on neural network training, the joint angle data in the state space is normalized and mapped to the interval [-1, 1], the formula being:

[0114] ;

[0115] , wherein represents the normalized i-th joint angle, which is used as the input feature of the agent, represents the minimum joint angle allowed by the i-th joint, represents the maximum joint angle allowed by the i-th joint.

[0116] In this embodiment, S3 specifically includes the following content: ​

[0117] S31, construct the state space S, including the normalized joint angle of the robot arm, the position vector g2t of the end effector relative to the gangue, the relative position vector t2g of the gangue relative to the center of the target recovery area, and the conveyor belt velocity vector ;

[0118] S32, construct the continuous action space A, define the action vector output by the policy network , where each component corresponds to the control instruction of one joint, and each action component is limited in the range of [-1, 1], and when actually executed, it is mapped to the joint angle increment , according to the reinforcement learning code written, the calculation formula is:

[0119] ;

[0120] , where represents the target angle increment of the i-th joint actually output to the underlying controller; represents the original action control amount of the i-th joint output by the policy network; represents the preset joint reference angular velocity rad / s, is a single time step of the simulation environment, and this mechanism ensures the smoothness of the robot arm movement;

[0121] S33, in order to adapt to the phased course learning strategy, construct a reward function architecture containing an approach potential energy item, a push-away potential energy item, a sparse event item, and a constraint penalty item, and the calculation formula is as follows:

[0122] ;

[0123] S34, potential energy guided reward item : use the difference between the distance at the last time and the distance at the current time to guide the action:

[0124] ;

[0125] , where represents the guided reward value calculated based on the distance potential energy difference; and respectively represent the Euclidean distance from the center of the target gangue to the center of the recovery area at the last time t-1 and at the current time t; and respectively represent the Euclidean distance from the center of the target gangue to the center of the recovery area at the last time t-1 and at the current time t; represents the potential energy reward weight coefficient of the push-away stage; represents the potential energy reward weight coefficient of the approach stage; and is a phase adaptive adjustment factor, set as , 0, only guiding the robot arm to touch the target; during the dynamic dialing training phase, set as , 1, wherein is a contact state indication variable, taking the value of 1 when the robot arm is in contact with the gangue, otherwise 0;

[0126] S35, sparse event reward term : used to give a one-time strong incentive to key node events in the task to strengthen successful behavior:

[0127] ;

[0128] wherein, is a first contact indication variable, taking the value of 1 only when the end of the robot arm first enters the contact radius of the target gangue and no contact has occurred before, and 0 otherwise; is the reward constant of the first contact; is a task success indication variable, taking the value of 1 when the target gangue enters the recycling area and remains stable in the area for a threshold time, and 0 otherwise; is the reward constant of task success;

[0129] S36, constraint penalty term : used to constrain the invalid action of the agent and ensure the safety of the work:

[0130] ;

[0131] wherein, is a single-step time consumption penalty value, forcing the agent to complete the task in the least number of time steps; is a safe collision indication variable, taking the value of 1 when a collision between the robot arm (except the end gripper) and the conveyor belt is detected, and 0 otherwise; is a penalty constant for non-safe area collision, used to suppress dangerous actions.

[0132] In the embodiment, the S4 specifically includes the following contents:

[0133] S41, as shown in Figure 3 , M independent robot arm sorting scenes are created in the simulation software Unity, and reinforcement learning training is performed through the ML-Agents plug-in, an agent is added in each created scene, and all agents share the same set of policy network and value network ;

[0134] ​S42, in the variant of centralized training and decentralized execution architecture, each agent interacts with the environment in parallel in its own randomized environment, outputs actions using a shared policy network, and independently collects experience data;

[0135] S43, in each round of update iteration, all agents perform T time step sampling in parallel, each agent uses a shared policy network to output actions according to the current state ; , interacts with the environment to get rewards and the next state ;

[0136] S44, all trajectory data tuples generated by all agents are aggregated and stored in the global experience buffer to break the temporal correlation of single-agent sampling data;

[0137] S45, using a shared value network to evaluate the value of each state and calculate the generalized advantage estimation value , the calculation formula is:

[0138] ;

[0139] ;

[0140] wherein, is the temporal difference error, is the discount factor, is the GAE smoothing factor;

[0141] S46, introduce weight coefficient :

[0142] ;

[0143] wherein, is a preset key sample adjustment constant;

[0144] S47, in order to balance exploration and utilization, a dynamic entropy coefficient is designed:

[0145] ;

[0146] wherein, is the proportion of task success in the current training batch; and are the maximum and minimum values of the entropy coefficient respectively; is the decay rate constant;

[0147] S48, construct a composite objective function containing an adaptive entropy regularization term and a key sample weighting term, update the policy network by maximizing the function:

[0148] ;

[0149] wherein, is a standard PPO clipping objective function; is the policy entropy; is the key sample weight coefficient, which is positively correlated with the absolute value of the advantage value , used to strengthen the learning of high-value experience; is a value function loss term, used to update the value network parameters; is the value loss coefficient; is an adaptive entropy coefficient, which is negatively correlated with the success rate of the current batch of tasks ; when is low, increase to encourage exploration, and when is high, reduce to speed up convergence.

[0150] In the embodiment, S5 specifically includes the following contents:

[0151] S51, design a phased course learning strategy, first, the first phase: freeze the conveyor belt movement in the simulation environment, and set the target gangue as a kinematic fixed state;

[0152] S52, set the stage adjustment factor of the potential energy guidance reward term to =1, ; when the average contact success rate of the agent in the last N rounds exceeds the preset threshold , enter the next phase, which trains the agent to quickly identify and reach the target gangue in a static environment;

[0153] S53, design the second phase: restore the conveyor belt to run at a rated speed, remove the kinematic fixed state of the gangue, enable rigid body collision simulation of the physics engine, and randomly generate gangue positions in the entire workspace;

[0154] S54, set the stage adjustment factor to , and is dynamically related to the contact state, i.e. =(1- ), when the task success rate of the agent in the last N rounds exceeds the preset threshold , enter the next phase; this phase trains the agent to laterally push the moving gangue into the recycling area in a dynamic environment;

[0155] S55, design the third stage: on the basis of maintaining the dynamic environment of the second stage, superimpose the physical parameter domain randomization mechanism, at the beginning of each round, randomly sample and disturb the friction coefficient of the conveyor belt, the gangue mass, the lighting conditions and the gangue pose;

[0156] S56, keep the same reward weight configuration as the second stage, and introduce an adaptive entropy regularization mechanism to dynamically reduce the exploration noise according to the success rate until the policy network converges completely.

[0157] In this embodiment, S6 specifically includes the following contents:

[0158] S61, set the dynamic friction coefficient of the conveyor belt surface in Unity as a random variable to simulate the change of friction in real working conditions;

[0159] S62, set the mass of the target gangue as a random variable to simulate the inertia difference of gangues of different sizes and densities;

[0160] S63, set the light intensity and light source angle in Unity as random variables;

[0161] S64, at the beginning of each training round, randomly sample the initial generated position and rotation attitude of the target gangue on the conveyor belt;

[0162] S65, perform adversarial training based on the above constructed dynamic parameter space, and adjust the hyperparameters and other data until the average success rate of the policy network in the full parameter domain converges;

[0163] As shown in Figure 4 , it can be seen that with the increase of the number of training steps, the cumulative rewards of single-agent and multi-agent both show an upward trend, but the multi-agent model trains the same number of steps as the single-agent model, and the multi-agent training time is much shorter than the single-agent;

[0164] As shown in Figure 5 , it can be seen that using the ordinary PPO algorithm for training, the model cannot get good convergence, while the improved MAGS-PPO algorithm can get good training effect.

[0165] In this embodiment, S7 specifically includes the following contents:

[0166] S71, after the policy network training is completed, the model is first deployed in the simulation environment Unity to verify the running effect and ensure that the action can be successfully completed;

[0167] S72, export the trained model through the ONNX format, and deploy it on the ROS side to apply it to the actual mechanical arm to drive the mechanical arm to perform operations.

[0168] Embodiments:

[0169] The application firstly acquires the normalized joint angle of the mechanical arm, the running speed of the conveyor belt and the relative position information of the target gangue and the end of the mechanical arm and the recycling area in real time, and constructs a high-dimensional state space. Then, aiming at the complex working conditions such as sparse reward, unstructured target grasping difficulty and low sample utilization rate in the coal gangue sorting task, the improved MAGS-PPO algorithm is introduced, the exploration efficiency of the strategy network in the complex dynamic environment is improved through the multi-agent parallel sampling architecture and the shared network mechanism. By introducing the key sample weighting mechanism and the adaptive entropy regularization mechanism, the exploration and utilization are dynamically balanced, the strategy update gradient is optimized, so that the model converges more rapidly and robustly in the long sequence decision-making task. By designing a compound dense reward function based on potential energy guidance, combined with a phased curriculum learning strategy, contact guidance, dynamic poking and generalization training are carried out in turn, effectively solving the cold start problem of end-to-end control. On the simulation side, the physical parameter domain randomization mechanism is introduced, the environmental friction coefficient, object mass and observation noise are dynamically disturbed, and the adversarial training is performed in the full parameter space to narrow the gap between simulation and reality. Finally, the strategy network trained to converge is deployed to the real controller, and the joint angle increment instruction is output according to the real-time state to drive the mechanical arm to perform the lateral poking operation. In summary, the application realizes the efficient and flexible lateral sorting of coal gangue in an unstructured environment by improving the reinforcement learning algorithm and combining curriculum learning and domain randomization technology, which has important application value in the fields of coal mines and intelligent manufacturing.

[0170] Obviously, those skilled in the art can make various modifications and variations to the application without departing from the spirit and scope of the application. Thus, if these modifications and variations of the application belong to the scope of the claims of the application and their equivalent technologies, the application also intends to include these modifications and variations. The above description is only the preferred embodiment of the application, but the protection scope of the application is not limited thereto, and any skilled person in the art can make equivalent substitutions or changes to the technical solutions and inventive concepts of the application within the technical scope disclosed by the application, which should be covered by the protection scope of the application.

Claims

1. A method for controlling the lateral separation of coal gangue based on MAGS-PPO, characterized in that, Includes the following steps: S1. Real-time acquisition of the current joint angle data of the robotic arm, the conveyor belt running speed, and the position information of the target gangue in the camera coordinate system; S2. Normalize the location information and perform relative coordinate transformation; S3. Establish a reinforcement learning model based on Markov decision process, define the state space S, construct a continuous action space A based on joint angle increment, and design a composite dense reward function R based on potential energy guidance. S4. Construct a multi-agent parallel simulation environment and use the MAGS-PPO algorithm to iteratively train the policy network, including asynchronous sampling, GAE advantage estimation and PPO-clip update; S5. During the training process, a phased course learning strategy is adopted, which includes contact guidance, dynamic manipulation and generalization training in sequence. S6. Introduce a physical parameter randomization mechanism in the simulation environment, and dynamically perturb the environmental friction coefficient and the mass of the object to obtain a converged policy network. S7. Deploy the converged policy network to the real robotic arm controller, and output lateral disengagement action commands according to the real-time status to drive the robotic arm to perform lateral rock-disengagement operations.

2. The method for lateral separation control of coal gangue based on MAGS-PPO according to claim 1, characterized in that, S1 specifically includes the following: S11. Read the current angle values ​​of the 6 joints through the underlying interface of the robotic arm controller. ; S12. Read the current linear velocity scalar of the conveyor belt through the conveyor belt encoder, and convert it into a velocity vector in the base coordinate system according to the direction of the conveyor belt's movement. ; S13. Acquire images of the conveyor belt using a visual recognition system, and receive the raw recognition results output by the visual recognition module, including the type of the target gangue, the confidence level, and the location information of the center point coordinates in the camera coordinate system. .

3. The method for lateral separation control of coal gangue based on MAGS-PPO according to claim 1, characterized in that, S2 specifically includes the following: S21. Utilize the pre-calibrated homogeneous transformation matrix from the camera to the robotic arm base. The coordinates of the target rock are transformed from the camera coordinate system to the robot arm base coordinate system. The calculation formula is as follows: = ,in, = This represents the absolute position of the gangue in the coordinate system of the robotic arm base; S22. Using the DH parameter model of the robotic arm, based on the current joint angle... The current position of the robotic arm's end effector in the base coordinate system is calculated using forward kinematics. = ; S23, For the position vector of the end effector relative to the gangue. ,Right now: ； For gangue to be disposed of at the designated recycling center relative position vector ,Right now: ； in, These are preset fixed coordinate points; S24. Normalize the joint angle data in the state space and map it to the interval [-1, 1] to eliminate the influence of different dimensions on neural network training: ； in, This represents the normalized i-th joint angle, used as an input feature for the agent. This represents the minimum allowed joint angle for the i-th joint. This represents the maximum allowed joint angle for the i-th joint.

4. The method for lateral separation control of coal gangue based on MAGS-PPO according to claim 1, characterized in that, S3 specifically includes the following: S31. Construct the state space S, including the normalized joint angles of the robotic arm, the position vector g2t of the end effector relative to the gangue, the relative position vector t2g of the gangue relative to the center of the target recycling area, and the conveyor belt speed vector. ; S32. Construct a continuous action space A, and define the action vectors output by the policy network. Each component corresponds to a control command for a joint, and each motion component... Constrained within the range of [-1, 1], it is actually mapped to joint angle increments during execution. Based on the reinforcement learning code, the calculation formula is: ； in, This represents the target angle increment that is actually output to the i-th joint of the underlying controller; This represents the original motion control quantity for the i-th joint output by the policy network; This indicates the preset joint reference angular velocity. This mechanism ensures the smoothness of the robotic arm's movements for a single time step in the simulation environment. S33. To adapt to the phased learning strategy, a reward function architecture is constructed that includes an approach potential energy term, a departure potential energy term, a sparse event term, and a constraint penalty term. The calculation formula is as follows: ； S34, Potential Energy Guiding Reward Item Use the difference between the distance at the previous time step and the distance at the current time step to guide the action: ； in, This represents the guided reward value calculated based on the distance potential energy difference; and These represent the Euclidean distances from the center of the target gangue to the center of the recovery area at the previous time t-1 and the current time t, respectively. and These represent the Euclidean distances from the end effector of the robotic arm to the center of the target gangue at the previous time t-1 and the current time t, respectively. This represents the potential energy reward weighting coefficient during the disengagement phase. This represents the potential energy reward weighting coefficient for the approach phase. and As a phased adaptive adjustment factor, it is set during the contact-guided training phase. , 0, only guide the robotic arm to touch the target; during the dynamic manipulation training phase, set... , 1, of which This is a contact state indicator variable, indicating the state when the robotic arm comes into contact with the gangue. =1, otherwise 0; S35, Sparse Event Rewards Used to provide a one-time strong incentive for key milestone events in a task, in order to reinforce successful behavior. ； in, It is the first contact indicator variable, which takes a value of 1 only when the end of the robotic arm first enters the contact radius of the target gangue and no contact has occurred before; otherwise, it is 0. This is the reward constant for the first contact; It is a task success indicator variable. It takes a value of 1 when the target gangue enters the recycling area and remains stable in the area for a certain period of time, and is 0 at other times. It is the reward constant for successful task completion; S36, Constraints and Penalties Used to constrain invalid actions of intelligent agents and ensure operational safety. ； in, This is a penalty value for single-step time consumption, forcing the agent to complete the task in the fewest possible time steps; It is a safety collision indicator variable. It takes a value of 1 when a collision is detected between the robotic arm and the conveyor belt, except for the end gripper; otherwise, it takes a value of 0. It is the penalty constant for collisions in unsafe areas, used to suppress dangerous actions.

5. The method for lateral separation control of coal gangue based on MAGS-PPO according to claim 1, characterized in that, S4 specifically includes the following: S41. Create M independent robotic arm sorting scenarios in the simulation software Unity. Train the robotic arm using the ML-Agents plugin for reinforcement learning. Add an agent to each scenario, and all agents share the same policy network. and value network ; S42. A variant of the centralized training distributed execution architecture, in which each agent interacts with the environment in parallel in its own randomized environment, outputs actions using a shared policy network, and independently collects experience data. S43. In each round of update iteration, all agents perform sampling for T time steps in parallel, and each agent utilizes a shared policy network. Based on the current state Output Action Rewards for interacting with the environment and the state at the next moment ; S44. Collect trajectory data tuples generated by all agents. The data is aggregated and stored in a global experience buffer to break the temporal correlation of single-agent sampling data; S45. Utilizing shared value networks Evaluate the value of each state and calculate the generalized advantage estimate. The calculation formula is: ； ； in, For timing difference error, As a discount factor, It is the smoothing factor for GAE; S46. Introducing weighting coefficients : ； in, The preset key sample adjustment constant; S47. To balance exploration and utilization, design a dynamic entropy coefficient. : ； in, It represents the percentage of tasks that succeeded in the current training batch; and These are the maximum and minimum values ​​of the entropy coefficient, respectively. It is the decay rate constant; S48. Construct a composite objective function that includes an adaptive entropy regularization term and a key sample weighting term, and update the policy network by maximizing this function: ； in, The standard PPO shearing objective function; For policy entropy; The key sample weight coefficients are the absolute values ​​of the odds values. Positive correlation, used to reinforce learning of high-value experiences; This is the loss term in the value function, used to update the value network parameters; This is the value loss coefficient; The adaptive entropy coefficient is related to the success rate of the current batch of tasks. Negative correlation; when Increase at lower levels To encourage exploration, when Decrease when higher To accelerate convergence.

6. The method for lateral separation control of coal gangue based on MAGS-PPO according to claim 1, characterized in that, S5 specifically includes the following: S51. Design a phased learning strategy. The first phase is: freeze the conveyor belt movement in the simulation environment and set the target gangue to a kinematically fixed state. S52. Set the stage adjustment factor of the potential energy guided reward item as follows: =1, ; When the agent's average contact success rate exceeds a preset threshold over N consecutive rounds... Then, the process moves to the next stage, where the agent is trained to quickly identify and reach the target gangue in a static environment. S53. Second phase of design: Restore the conveyor belt to run at the rated speed, release the kinematic fixation of the gangue, enable the rigid body collision simulation of the physics engine, and randomly generate the gangue positions in the entire workspace. S54. Set the stage adjustment factor as follows: ,and It is dynamically related to the contact state, that is =(1- When the agent's task success rate exceeds a preset threshold in N consecutive rounds... Then, proceed to the next stage; During this phase, the training agent is used to laterally push moving gangue into the recycling area in a dynamic environment. S55. Design Phase 3: While maintaining the dynamic environment of Phase 2, a randomization mechanism for the physical parameter domain is superimposed. At the beginning of each round, the friction coefficient of the conveyor belt, the mass of the gangue, the lighting conditions, and the pose of the gangue are randomly sampled and disturbed. S56. Maintain the same reward weight configuration as in the second phase, and introduce an adaptive entropy regularization mechanism to dynamically reduce exploration noise based on the success rate until the policy network fully converges.

7. The method for lateral separation control of coal gangue based on MAGS-PPO according to claim 1, characterized in that, S6 specifically includes the following: S61. In Unity, set the dynamic friction coefficient of the conveyor belt surface to a random variable to simulate the friction force change under real working conditions. S62. Set the mass of the target gangue as a random variable to simulate the inertia difference of gangue of different sizes and densities; S63. In Unity, set the light intensity and light source angle to random variables; S64. At the beginning of each training round, the initial generation position and rotation attitude of the target gangue on the conveyor belt are randomly sampled. S65. Based on the dynamic parameter space constructed above, perform adversarial training by adjusting hyperparameters and other data until the average success rate of the policy network converges in the full parameter domain.

8. The method for lateral separation control of coal gangue based on MAGS-PPO according to claim 1, characterized in that, S7 specifically includes the following: S71. After the policy network is trained, first deploy the model in the Unity simulation environment to verify the running effect; S72. Export the trained model in ONNX format and deploy it on the ROS platform to drive the robotic arm to perform operations.

Images