Industrial robot control method and system based on deep reinforcement learning

Abstract

The invention discloses an industrial robot control method and system based on deep reinforcement learning. The method comprises the following steps that a virtual simulation environment comprising a mechanical arm, a to-be-detected object, a conveying line and a detection table body is constructed; designing a state space and an action space of deep reinforcement learning; designing a reward function for guiding a deep reinforcement learning training process, wherein the reward function is a weighted combination of a task reward, an efficiency reward, a security reward and a skill reward; performing staged training on the strategy network by adopting a deep reinforcement learning algorithm aiming at maximizing accumulated rewards and strategy entropies based on a reward function; and the trained strategy network is deployed into a control system of the mechanical arm, the control system generates a control action according to the current state obtained in real time, and the mechanical arm is driven to execute the feeding and discharging task of the to-be-detected object. According to the invention, the control precision can be ensured, the system reliability can be improved, and high production flexibility can be maintained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent manufacturing, and particularly relates to an industrial robot control method and system based on deep reinforcement learning. BACKGROUND

[0002] Inspection and detection is a core process in the sorting, testing and recycling process of recycled electric energy meters. This process usually requires a large number of old electric energy meters to be taken out from the material box and accurately placed on a professional test bench for up to dozens of electrical performance and function tests. After the test is completed, it is placed in the recycling or scrap area according to the results. At present, the power-on detection of electric energy meters mostly adopts a mode combining manual operation with professional test benches. In this mode, the feeding and discharging, wiring operation and result determination are all completed by manual operation, and the single detection time is relatively long, usually 15 to 30 minutes. When facing the task of full performance detection of a large number of electric energy meters, relying on manual feeding and discharging will lead to low production efficiency and high labor cost, and the standardization of the test results is easily affected by human factors.

[0003] In order to solve the above problems, the industry has tried to use six-axis robot arms and other automatic equipment to replace manual feeding and discharging operations. However, the existing mechanical arm automatic control technology scheme is not mature, mainly including two kinds: the first is the traditional pre-programmed control method. This method controls the movement of the mechanical arm by pre-setting a fixed trajectory program. The disadvantage is that the scheme is rigid, and it needs to write and debug the movement program for different models and batches of electric energy meters, which cannot adapt to the subtle changes in the production environment, and the flexibility is poor. The second is the teaching programming method based on visual assistance. This method identifies the position of the electric energy meter through the vision system, but still needs manual teaching of key path points in path planning. Although it has made some progress compared to the first scheme, its flexibility and intelligence are still limited, and it cannot realize truly autonomous operation.

[0004] Chinese invention patent application with publication number CN120206521A discloses a composite robot CNC feeding and discharging method and device, the method of which includes: step one, obtaining the best feeding and discharging pose of TCP through teaching; step two, adjusting the end pose of the mechanical arm to the best shooting pose of the 3D camera on the feeding seat, recording the adjustment amount ΔT3D1 for step three, and recording the coordinate ZTCP3D of TCP for use in 3D camera shooting positioning in the deployment process of other CNC machine tools; calculating and recording the best feeding and discharging pose in the 3D camera coordinate system T3D; step three, adjusting the end pose of the mechanical arm again to the best shooting pose of the 3D positioning mark plate by the 3D camera, recording the TCP coordinate ZTCP2D for use in 3D camera shooting positioning in the feeding and discharging stage. Record the first best feeding and discharging pose T2D1 in the 3D camera stereo coordinate system relative to the center of the 3D positioning mark plate on the first CNC.

[0005] After long-term high-speed operation, the rotating mechanism and other components of the mechanical arm will produce cumulative errors, affecting the repeatability accuracy, and further causing the failure of grabbing. At the same time, the inertia vibration caused by high-speed motion will also affect the accurate positioning of the end effector. The existing control scheme lacks the ability of environment perception, autonomous learning and real-time decision-making, and cannot adjust its motion strategy in real time according to the dynamic changes of the working environment (such as changes in illumination, appearance of temporary obstacles). SUMMARY

[0006] The application provides an industrial robot control method and system based on deep reinforcement learning, aiming to solve the problems of lack of environment perception, autonomous learning and real-time decision-making ability, and cumulative error leading to reduced positioning accuracy in the prior art.

[0007] To solve the above technical problems, on the one hand, the application provides an industrial robot control method based on deep reinforcement learning, comprising the following steps: Constructing a virtual simulation environment including a mechanical arm, an object to be detected, a conveying line and a detection table body; Designing a state space and an action space of deep reinforcement learning; Designing a reward function for guiding the training process of deep reinforcement learning, the reward function being a weighted combination of task reward, efficiency reward, safety reward and skill reward; Based on the reward function, a deep reinforcement learning algorithm aiming to maximize the cumulative reward and the policy entropy is used to train the policy network in stages; Deploying the trained policy network to the control system of the real mechanical arm, and the control system generates control actions according to the real-time acquired current state to drive the mechanical arm to perform the feeding and discharging task of the object to be detected.

[0008] Preferably, the construction of the virtual simulation environment includes setting physical engine parameters, mechanical arm dynamics parameters, visual environment parameters and task parameters; The physical engine parameters include gravitational acceleration 9.81 m / s² and friction coefficient 0.3-0.7; The mechanical arm dynamics parameters include joint friction coefficient ±20%, damping coefficient ±15% and load mass ±10%; The visual environment parameters include random changes in illumination intensity 300-1000 lux and random replacement of electric meter textures; The task parameters are the initial position and attitude of the object to be detected in the magazine, which are randomly generated.

[0009] Preferably, the state space includes: mechanical arm body state, end effector state, conveying line state, anti-collision information, task state, clamp opening and closing state, and force sensor reading.

[0010] Preferably, the action space comprises: incremental displacement of the joints, gripper control commands, conveyor motion control, and detection table body start-stop control.

[0011] Preferably, the task reward is configured to provide a positive reward when the robot arm successfully completes a preset sub-task node; the sub-task node at least includes successfully grasping the to-be-detected object, successfully placing the to-be-detected object on the detection table body, or successfully completing a complete feeding and discharging cycle. The efficiency reward comprises: a time penalty applied to each action step, and a distance reward provided based on the reduction in distance between the end effector of the robot arm and the target point. The safety reward comprises: a collision penalty applied when the robot arm collides, and an over-force penalty applied when it is detected that the torque of the joint of the robot arm exceeds a preset safety threshold. The skill reward comprises: a trajectory smoothness reward or penalty set based on the change in the joint angle vector at adjacent time points, and an energy consumption optimization reward set based on a comparison between the actual output torque of the joint and the rated torque.

[0012] Preferably, the deep reinforcement learning training adopts a flexible action-evaluation algorithm.

[0013] Preferably, the staged training at least includes: a first training stage for the robot arm to learn basic motor skills, a second training stage for the robot arm to learn to grasp and place the to-be-detected object, and a third training stage for the robot arm to learn to complete the complete feeding and discharging process in cooperation with peripheral devices.

[0014] Preferably, after deploying the policy network to the control system of the real robot arm, the method further comprises: collecting interaction data generated by the real robot arm during operation, and fine-tuning the policy network using the interaction data; wherein the fine-tuning of the network parameters adopts a hierarchical learning rate strategy, using a smaller learning rate for the bottom layer of the network and a larger learning rate for the upper layer.

[0015] Preferably, the method further comprises real-time safety monitoring and abnormality handling when driving the robot arm to perform tasks, the monitoring and handling at least including one of: workspace monitoring to constrain joint angles, detect self-collision, or predict environmental collision; dynamics monitoring to limit joint torque or end speed; an abnormality recovery strategy to trigger retry, re-planning, or switching to a safety mode when detecting grasping failure, path blockage, or sensor abnormality.

[0016] In another aspect, the present application also provides an industrial robot control system based on deep reinforcement learning, which is used to implement the industrial robot control method according to the first aspect of the present application, comprising: an industrial robot equipped with an end effector for interacting with an article to be detected; at least one sensor configured to collect information of the state of the industrial robot itself and / or the working environment to form a current state; a controller in communication connection with the industrial robot and the sensor, wherein the controller stores a policy network trained according to the above method, and the controller is configured to: receive real-time data from the at least one sensor to build the current state; input the current state into the policy network to autonomously decide and generate a control action; convert the control action into a control instruction and send it to the industrial robot to drive it to perform the feeding and unloading task of the article to be detected.

[0017] Compared with the prior art, the present application has the following technical effects: 1. The control method proposed by the present application introduces deep reinforcement learning, enabling the robot arm to have the ability of autonomous learning and decision-making, and completely getting rid of the rigid constraints of traditional pre-programming or teaching programming. The policy network can dynamically generate the optimal motion strategy according to the real-time environmental state perceived by the vision system and the sensor. Therefore, the present application can easily adapt to different models and sizes of electric energy meters, as well as the random position and attitude of electric energy meters in the magazine, without the need for reprogramming or manual teaching for each new situation, showing high production flexibility.

[0018] 2. The control method proposed by the present application drives the agent to continuously optimize its motion trajectory and operation tempo in millions of virtual training through a multi-objective composite reward function, especially under the guidance of efficiency reward and skill reward. The learned strategy can not only complete the task, but also complete it with near-optimal time, shortest path and smoothest action, thereby greatly shortening the feeding and unloading cycle of a single electric energy meter. The fully automated process replaces inefficient and tiring manual work, ensuring 24-hour uninterrupted efficient production and ensuring high consistency of each operation, significantly improving the standardization level of the inspection and detection process.

[0019] 3. The safety reward mechanism of the control method proposed by the present application integrates safety as a core learning goal into every aspect of the training. By imposing severe penalties for collisions and excessive force, the agent learns essentially how to avoid obstacles and achieve gentle operation. Combined with multi-layer real-time safety monitoring and exception handling mechanisms during deployment, the present application can actively prevent collisions and effectively handle unexpected situations such as failed grasping, greatly reducing the risk of equipment or energy meter damage due to misoperation, and ensuring the stability and reliability of the production process.

[0020] 4. The control method proposed by the present application adopts a simulation-to-reality development paradigm, and most of the algorithm training and debugging work is completed in a low-cost and efficient virtual environment, avoiding a large number of high-risk attempts on expensive physical devices, significantly shortening the development cycle and reducing research and development costs. After being put into operation, the system can bring considerable long-term economic benefits to enterprises by replacing high labor costs, improving production efficiency, and reducing downtime losses caused by equipment damage.

[0021] 5. The control method proposed by the present application introduces a phased progressive training strategy, which can efficiently train a highly intelligent, robust, and safe robotic arm control strategy. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of the industrial robot control method described in the present application. DETAILED DESCRIPTION

[0023] To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

[0024] Embodiment One The present embodiment is an industrial robot control method based on deep reinforcement learning, as shown in Figure 1 including the following steps one to five: Step one, construct a virtual simulation environment containing a robotic arm, an object to be detected, a conveying line and a detection table body.

[0025] The construction of the virtual simulation environment includes setting physical engine parameters, robotic arm dynamics parameters, visual environment parameters and task parameters; The physical engine parameters include gravitational acceleration 9.81 m / s² and friction coefficient 0.3-0.7; The robotic arm dynamics parameters include joint friction coefficient ±20%, damping coefficient ±15%, and load mass ±10%; The visual environment parameters include random variation of light intensity 300-1000 lux and random replacement of energy meter texture; The task parameter is the initial position and attitude of the to-be-detected article in the magazine.

[0026] The embodiment takes the inspection and detection process of disassembling and returning the electric energy meter for example. The old electric energy meter is clamped and placed on a professional inspection table by a six-axis mechanical arm. Automatic inspection is carried out for 39 inspection subjects of 8 categories of electric energy meters. After the inspection is completed, the old electric energy meter is placed in the area to be scrapped or recycled, and the recyclable old electric energy meter is further confirmed. The embodiment aims to realize the automatic feeding and discharging of the electric energy meter inspection and detection process, improve the standardized management of inspection and production efficiency.

[0027] The simulation training environment of the embodiment is built in the Gazebo simulation platform. The platform is a powerful 3D robot simulation environment and is one of the most commonly used simulation tools in ROS (Robot Operating System). It can check whether the size, joint connection and motion range of the robot are reasonable, and can test the real motion performance of the robot under gravity, friction and collision through the built-in physical engine (Open Dynamic Engine, Bullet, etc.). At the same time, it can realize rapid iteration, modify the robot design and test it easily without processing parts, which greatly saves cost and time.

[0028] Gazebo can safely, quickly and repeatedly develop and test robot systems at very low cost and risk, ensuring that their algorithms and designs are reliable and effective before being deployed to the real world.

[0029] Step two, design the state space and action space of deep reinforcement learning. Before applying the deep reinforcement learning algorithm, the state space and action space of the system need to be designed first. The state space defines the set of environmental information that the agent (such as a six-axis mechanical arm) can perceive at a certain time, which is the basis for the agent to make decisions. The action space defines the set of all possible actions that the agent can perform. At each time step, the agent can select certain actions from the action space to execute based on the current state.

[0030] The six-axis mechanical arm, end effector, conveyor line, electric energy meter box, electric energy meter inspection table body and other devices in the automatic feeding and discharging system of the electric energy meter detection process are agents. The state space vector and action space vector are designed as follows.

[0031] The state space includes: mechanical arm body state, end effector state, conveyor line state, anti-collision information, task state, clamp opening and closing state, and force sensor reading.

[0032] Specifically, the robot body state describes the physical configuration and motion tendency of the robot itself. It consists of two parts: 6 joint angles and 6 joint angular velocities . The joint angles precisely represent the angular positions of the six rotational joints from the base to the wrist at a certain moment, usually in radians. These data are directly obtained from the encoders installed on the joint motors and are the most basic information for determining the current pose of the robot. The joint angular velocities represent the instantaneous rotational speeds of the six joints, in radians per second. It can be obtained by time-differentiating the joint angles. The angular velocity information is crucial for the agent to learn smooth and dynamically stable motions, which can help it predict the next state and suppress vibrations.

[0033] The end effector state describes the position and pose of the robot's end (i.e., the position of the installed gripper) in the three-dimensional world coordinate system. It is not directly measured but is calculated in real-time from the 6 joint angles mentioned above through the forward kinematics model of the robot. Specifically, the end effector state includes a three-dimensional position [x, y, z] and a quaternion pose [qw, qx, qy, qz]. The three-dimensional position represents the Cartesian coordinates of the center point of the end effector in the pre-set world coordinate system; the agent uses this information to calculate the distance between the end and the target point (such as the electricity meter or the placement point on the detection table), which is a key basis for calculating the efficiency reward. The quaternion pose is a four-dimensional vector used to describe the rotational pose (i.e., orientation) of the end effector without singular points. Compared to Euler angles, using quaternions can avoid gimbal lock problems and ensure that the agent can learn effectively in any pose. The agent needs to plan precise alignment actions when grabbing and placing the electricity meter based on the difference between this pose and the target pose.

[0034] The conveyor line state is a state signal used to realize the coordinated work of multiple devices. In this embodiment, it can be a Boolean value or a binary signal (e.g., conveyor_ready = 1 or 0). This signal comes from the main control system of the production line. When the value is 1, it indicates that the box containing the electricity meter has arrived at the designated work station through the conveyor line and the conveyor line has stopped, and the robot can start grabbing; when the value is 0, it indicates that the conveyor line is moving or the work station is empty, and the robot needs to wait. This state ensures that the robot's actions are synchronized with the entire production rhythm.

[0035] The collision avoidance information provides the agent with quantitative information about its proximity to obstacles in the working environment. In this embodiment, it is specifically the distance to the nearest obstacle. In a simulated environment, this distance can be directly calculated by the physics engine. In a real physical environment, this information can be calculated from the environment point cloud model constructed by a 3D vision system (such as a depth camera) or a lidar. The value of this distance directly affects the safety reward, and when the value is below a set safety threshold, a large negative reward is triggered, thus teaching the agent to actively avoid collisions.

[0036] The task state specifies the specific goal of the current task. Specifically, the target energy meter pose , which describes the position and attitude of the target energy meter to be grasped or placed in the world coordinate system. This information is usually provided by a vision system deployed above or to the side of the robot arm through image recognition and pose estimation algorithms. It is the ultimate guide for all the agent's actions and the basis for calculating the distance reward in the task reward and efficiency reward.

[0037] The gripper opening and closing state indicates whether the end gripper is currently open, closed, or in some intermediate state. It can usually be represented by a scalar value, for example, in the range [-1, 1], -1 representing complete closure and 1 representing complete opening. This state is fed back by the gripper's own sensors (such as encoders or Hall sensors). The agent needs to confirm whether the grasping or releasing action has been executed to the end according to this state, and decide the next action accordingly.

[0038] The force sensor reading comes from a six-axis force / torque sensor installed at the wrist of the robot arm, providing precise force feedback when the robot arm end interacts with the environment. It is a six-dimensional vector , representing the force received by the end in the X, Y, Z three axes, and the torque around these three axes, respectively. This reading is used for collision detection, over-force detection, contact confirmation, energy optimization, etc. Collision detection: the instantaneous sharp increase in force is a clear signal of a collision, used to trigger the collision penalty in the safety reward; Over-force detection: if the force value exceeds the pre-set safety threshold during grasping or placing, an over-force penalty is triggered, which is used to teach the agent to "handle with care"; Contact confirmation: a small force change can be used as a signal of successful contact with the energy meter or detection platform; Energy optimization: comparison of torque reading and joint rated torque can be used to calculate the energy optimization term in the skill reward.

[0039] The action space of the deep reinforcement learning agent is designed as a 9-dimensional continuous control vector. This vector is the output of the policy network at each decision time step, which defines all possible operations that the agent can perform. Combining these operations enables the robot arm to accomplish complex tasks ranging from fine motor movements to process coordination. The specific composition and explanation of this action space are as follows: incremental displacement of joints, clamp control command, conveyor line motion control, and detection table body start-stop control.

[0040] Incremental displacement of joints is the core part of the action space, which is a 6-dimensional vector that directly controls the movement of the six joints of the six-axis robot arm. Unlike the control method that directly outputs the target joint angle (absolute displacement), this embodiment adopts the output incremental displacement method. The 6 values output by the policy network are normalized to the interval [-1, 1]. In the control system, this set of normalized values is multiplied by a pre-set scaling factor, thereby converting into the actual target angular velocity of each joint or the small angle change amount in the next control period. The incremental control method makes the movement of the robot arm more smooth and continuous, easy for the policy network to learn, and effectively avoids the violent and unstable movement caused by the sudden change of the output target. It is more consistent with the continuous dynamic process of the physical world, which helps to achieve the trajectory smoothness goal in the skill reward.

[0041] Clamp control command is a 1-dimensional scalar value used to control the opening and closing of the clamp (or gripper) installed at the end of the robot arm. The value output by the policy network is also normalized to the interval [-1, 1]. This value is mapped to the specific instructions of the clamp controller. For example, +1 can be interpreted as a fully open command; -1 can be interpreted as a fully closed command with a pre-set torque; 0 can be interpreted as maintaining the current opening degree or stopping movement. Using continuous value control enables the clamp not only to perform simple open / close actions, but also to achieve adaptive grasping of different size electric energy meters, and even in the future can be extended to force-controlled grasping, i.e., adjusting the clamping force according to the feedback of the force sensor.

[0042] The conveyor_move is a 1-dimensional scalar value that controls the power meter box conveyor line working in coordination with the robot arm. This embodies the idea that the agent not only controls itself, but also coordinates with peripheral devices. Although the value is continuous, in this embodiment it is used as a threshold trigger to perform discrete actions. For example, a threshold of 0.5 is set. When the policy network outputs a conveyor_move value greater than 0.5, the control system sends a digital signal to the production line master PLC to start the conveyor line; when the value does not meet the condition, no signal or a stop signal is sent. This design enables the agent to learn the complete workflow, for example: after successfully grabbing all the power meters in a box, the agent autonomously outputs a high conveyor_move value to start the conveyor line to deliver the next box.

[0043] The bench_control is a 1-dimensional scalar value that controls the working state of the power meter detection bench, which is a key link to achieve full-process automation. Similar to the conveyor line control, the value is also used as a threshold trigger. For example, after the agent successfully places a power meter at the precise position of the detection bench, it learns to output a bench_control value. This signal is sent to the control system of the detection bench, triggering it to start the automatic power-on and verification process for the power meter. Similarly, after the detection is completed, the detection bench can feed back a completion signal to the agent through the state space, and the agent makes a decision on whether to take it down. By introducing this action, the agent learns not only the handling task, but also the complete production collaboration task involving handshaking communication with the detection device, greatly improving the intelligence and integration of the system.

[0044] Step three, design a reward function for guiding the deep reinforcement learning training process, which is a weighted combination of task reward, efficiency reward, safety reward and skill reward. The reward function is a very important part of deep reinforcement learning, which is a mathematical function that feeds back a scalar value Q value, i.e. reward, to the agent at each time step t, according to the current state , the action taken by the agent and the new state caused by . This numerical signal can tell the agent whether the action in the state is good or bad, and how good it is. The fundamental purpose of the reward function is to encode our complex goals into a single signal that the agent can understand and optimize.

[0045] The task reward is configured to provide a positive reward when the robotic arm successfully completes a preset sub-task node; the sub-task node includes at least successfully grabbing the item to be tested, successfully placing the item to be tested on the testing platform, or successfully completing a complete loading and unloading cycle; for example, successfully grabbing an electricity meter +50, successfully placing it on the testing platform +100, and successfully completing a complete electricity meter loading and unloading cycle +500.

[0046] The efficiency bonus includes: a time penalty applied to each action step, and a distance bonus based on the reduction in distance between the robotic arm end effector and the target point; for example, a time penalty of -1 for each additional action step (a loading / unloading cycle will be divided into multiple action steps at fixed time intervals); and a distance bonus from the robotic arm end effector to the target point during the loading / unloading process. , Represents the distance at the previous moment. Represents the distance at the current moment.

[0047] The safety rewards include: a collision penalty applied when the robotic arm collides with another object, and an overforce penalty applied when the joint torque of the robotic arm is detected to exceed a preset safety threshold; for example, a collision penalty of -100, using force sensors and proximity sensors to detect whether the robotic arm collides with other objects, and applying a penalty when a collision occurs; and an overforce penalty. When the force of the robotic arm joint is detected to be greater than the set threshold, an overforce penalty is imposed.

[0048] The skill rewards include: trajectory smoothness rewards or penalties based on the changes in joint angle vectors at adjacent time points, and energy consumption optimization rewards based on the comparison between the actual output torque of the joint and the rated torque; for example, trajectory smoothness. The difference between the joint angle vector at the current moment and the joint angle vector at the previous moment is used as the basis for trajectory smoothness; energy consumption optimization. The system detects the joint torque of the robotic arm, sets a torque rating, and adds a corresponding energy consumption optimization bonus when the actual torque of a certain action step is less than the rating.

[0049] Finally, the multi-objective reward function for this step is expressed as follows:

[0050] in, These represent the final reward, task reward, efficiency reward, safety reward, and skill reward, respectively. are weight coefficients of each reward, respectively, for adjusting the emphasis degree of each reward in different stages. The composite reward function can prompt the automatic feeding and discharging system of the electric energy meter detection process to move with low energy consumption and high smoothness, so that the mechanical arm completes the grabbing and placing of the electric energy meter with a shorter motion trajectory in the shortest possible time, and finally realizes the automatic detection process of the electric energy meter, and avoids the phenomenon of collision and violent swinging in the whole process.

[0051] Step four, based on the reward function, a deep reinforcement learning algorithm aiming to maximize the cumulative reward and the policy entropy is used to train the policy network in stages.

[0052] Specifically, the training of the deep reinforcement learning described in the embodiment adopts a Soft Actor-Critic (SAC) algorithm. The reason for choosing the SAC algorithm is its unique optimization goal: it not only aims to maximize the cumulative reward in the future like traditional reinforcement learning algorithms, but also aims to maximize the entropy of the policy itself. In information theory, the entropy of the policy represents the randomness or uncertainty of its action selection. The goal of maximizing the policy entropy will encourage the agent to explore as many different actions and paths as possible under the premise of completing the task, rather than sticking to a single optimal solution. This feature can bring stronger exploration ability because it can avoid the premature convergence to a local optimal solution in the training process, so as to find a better strategy to complete the task. Since the policy learned by the agent itself contains a certain randomness and diversity, it will perform more stably and adaptively when it is deployed to a real physical environment full of uncertainties and disturbances.

[0053] In the embodiment, the SAC algorithm includes two core networks: a policy network (Actor) and a Q-value network (Critic). The specific network structure is set as follows: The Actor network (policy network) is responsible for deciding the action to be executed according to the current state. The input is the state s, and the output is a probability distribution (usually a Gaussian distribution) of an action. The update goal is to generate an action policy that can produce actions with high expected Q-value (high reward) and high entropy. The input layer has 35 nodes (corresponding to the dimension of the state space), passes through 3 hidden layers each containing 512 nodes, and finally the output layer has 9 nodes (corresponding to the dimension of the action space).

[0054] The critic network (Q-value network) is responsible for evaluating the pros and cons (Q-value) of performing a specific action in a specific state. Its input is the state s and the action a, and the output is the estimated soft Q-value of the state-action pair. The update target is to evaluate the pros and cons of the state-action pair, take the minimum value of the two Q-values, reduce the overestimation of the Q-value, and make the Q-value estimation more accurate. The soft update mechanism prevents the target value from changing drastically. Its input layer has 44 nodes (35 dimensions of the state space + 9 dimensions of the action space), and also passes through 3 hidden layers each containing 512 nodes, and finally the output layer has 1 node representing the evaluated Q-value. To improve the accuracy and stability of the evaluation, this embodiment adopts the design of double Q-value network (Double-Q Network).

[0055] The SAC algorithm also includes an adjustable temperature parameter a, which determines the importance of entropy in the optimization target. It can be a fixed value or a learnable parameter. A key innovation of SAC is that it can automatically adjust a: when the entropy of the policy is lower than the target value, a increases to encourage exploration; when the entropy is too high, a decreases to focus more on rewards.

[0056] The phased training at least includes: a first training phase of enabling the mechanical arm to learn basic motor skills, a second training phase of learning to grasp and place the to-be-detected articles, and a third training phase of learning to complete a complete feeding and discharging process in cooperation with peripheral devices.

[0057] First phase: basic motor skill training (about 1 million time steps) Training goal: The core goal of this phase is to enable the mechanical arm to learn the most basic motor control ability, i.e., to smoothly, quickly and accurately track any target point in space. This is the basis for all subsequent complex operations.

[0058] Task setting: In the simulation environment, no electricity meter is placed. The system will randomly generate a series of three-dimensional target coordinate points in the workspace of the mechanical arm, and require the end effector of the mechanical arm to reach these points in sequence.

[0059] Reward function focus: In this phase, the weight coefficients in the reward function will be tilted towards efficiency rewards and skill rewards. Specifically, the agent will receive higher total rewards for reaching target points faster (less time penalty), taking shorter paths (more distance rewards), smoother actions (trajectory smoothness rewards), and smaller joint torques (energy optimization rewards). The weights of task rewards and safety rewards are adjusted lower.

[0060] Second phase: grasping and placing skill training (about 2 million time steps) Training goal: After mastering the basic motor skills, the goal of this stage is to let the robot learn the core operation skills, i.e. how to accurately grasp and place the electricity meter, and begin to cope with the uncertainty of the target pose.

[0061] Task setting: Introduce electricity meters and detection tables into the simulation environment. The initial position and pose of the electricity meter will be randomly generated within a preset range. The robot needs to start from a random initial state, plan a path, approach and successfully grasp the electricity meter, and then transport and place it to the designated area of the detection table.

[0062] Reward function focus: The focus of this stage of training shifts to task rewards and safety rewards. Successfully grasping the electricity meter and successfully placing it in place will give a huge positive reward. Conversely, any form of collision (with the electricity meter, table or environment) will result in severe punishment. At this time, efficiency and skills are still within the scope of consideration, but their importance has given way to the need to safely complete the core task.

[0063] Third stage: Complete loading and unloading process training (about 3 million time steps) Training goal: Learn and optimize the complete production process working with peripheral devices, making the robot an intelligent unit that can integrate into the entire automated production line.

[0064] Task setting: The simulation environment is upgraded to a complete production station, including a bin containing multiple electricity meters, a conveyor line, and a detection table. The task requires the robot not only to complete grasping and placing, but also to control the start and stop of the conveyor line (for example, after emptying a bin, start the conveyor line to bring the next one) and the detection table (for example, after placing it in place, notify the table to start detection). This stage will introduce higher intensity environmental randomization to simulate various changes in real production.

[0065] Reward function focus: In this final stage, all weight coefficients (K1, K2, K3, K4) of the reward function will tend to balance. The goal is to let the agent improve work efficiency, smoothness and energy consumption performance as much as possible under the premise of ensuring task success rate and safety, so as to learn a comprehensive performance optimal control strategy.

[0066] Step five, deploy the trained strategy network to the control system of the real robot, and the control system generates control actions according to the current state obtained in real time to drive the robot to perform the loading and unloading task of the detection object.

[0067] This embodiment will deploy the strategy network trained in the Gazebo simulation environment to the real robot control system, using a gradual migration strategy, including three migration stages.

[0068] The first migration stage runs under safety supervision, aiming to collect real environment interaction data.

[0069] This stage will establish a multi-layer safety protection mechanism to ensure a smooth transition from simulation to real environment: First, hardware layer safety monitoring, set joint angle soft and hard limit, trigger gradual resistance when approaching physical limit to implement real-time torque monitoring, stop motion immediately when detecting abnormal contact force, install work space electronic fence to limit the movement of the robot arm within the safety area, install emergency stop button and physical isolation device to ensure smooth access for manual intervention.

[0070] Second, control layer safety strategy, design action filtering mechanism to smooth the original action output by DRL and limit speed; implement predictive collision detection to predict trajectory safety within 0.5 seconds in the future through kinematics model; deploy redundant sensor verification to compare data consistency of vision system, encoder and force sensor; set safety fallback strategy to automatically switch to predefined conservative control mode when detecting abnormalities.

[0071] Real environment data collection is also needed, execute known tasks in controlled environment to collect state-action-reward sequences; record sensor raw data and timestamps to establish real world dataset; label success / failure cases, especially pay attention to boundary conditions and abnormal scenarios; collect environment disturbance data to simulate uncertainty factors in actual production.

[0072] The second migration stage fine-tunes the policy network based on real data to adapt to the difference in dynamics.

[0073] Data preprocessing and enhancement perform time alignment and sensor calibration on real data to ensure data quality; apply data enhancement techniques to generate more training samples by adding noise to existing trajectories; construct balanced dataset to ensure balanced distribution of positive and negative samples and different working condition data; perform feature normalization to match the scale of real sensor data with the distribution of simulation training.

[0074] Network parameter fine-tuning uses a hierarchical learning rate strategy, using a smaller learning rate for the bottom layer (feature extraction layer) and a larger learning rate for the higher layer (decision layer); implement elastic weight solidification to protect important connection weights learned in simulation from being changed drastically; use regularization constraints to prevent the network from overfitting limited real data and losing generalization ability; design fine-tuning process to start with simple tasks and gradually increase difficulty to stabilize the learning process.

[0075] The fine-tuning objective function optimization adds a real environment adaptation term to the original SAC objective function.

[0076] The third migration stage deploys an online adaptation mechanism to continuously optimize policy performance.

[0077] Performance monitoring and evaluation: Establish a multi-dimensional performance indicator system including task success rate, cycle time, energy efficiency, security indicators; Implement real-time performance tracking to detect strategy performance degradation or environmental changes; Set adaptive thresholds to trigger retraining when key indicators exceed normal ranges; Deploy anomaly detection algorithms to identify new unseen work scenarios.

[0078] Incremental learning mechanism: Design experience replay management strategy to prioritize data with high learning value; Implement model version control to support safe rollback to previous stable strategy version; Build online validation environment for simulation testing before deploying new strategies; Use ensemble learning approach to maintain multiple strategy networks and make decisions through weighted voting.

[0079] Adaptive optimization strategy: Implement environment change detection to automatically adjust control parameters when detecting system dynamics drift; Deploy learning framework to enable agents to learn how to quickly adapt to new meter models or layouts; Establish human-machine collaboration interface to allow operators to provide a small amount of demonstration or feedback to guide learning direction; Design resource-aware training scheduling to update models during production gaps or low-load periods.

[0080] Long-term maintenance and optimization: Establish long-term performance monitoring logs to analyze performance trends; Regularly conduct benchmark tests to compare performance differences between new strategies and historical versions; Implement A / B testing framework to verify improvement effects under controlled conditions; Build knowledge base system to accumulate successful experiences in solving various abnormal situations.

[0081] In another embodiment of the present application, the method further comprises real-time safety monitoring and abnormal handling when driving the mechanical arm to perform tasks, which at least includes one of the following: Workspace monitoring to constrain joint angles, detect self-collision or predict environmental collision; Dynamics monitoring to limit joint torque or end speed; When detecting failed grasping, path blockage or sensor abnormalities, trigger abnormal recovery strategies such as retry, re-planning or switching to safe mode.

[0082] Specifically, the present embodiment designs a multi-layer safety monitoring mechanism including workspace constraints, dynamics constraints and abnormal recovery strategies.

[0083] Workspace constraints include joint angle limit protection, self-collision detection system and environmental collision prediction.

[0084] Joint angle limit protection, adopts multi-level limit protection system, including soft limit early warning: set 5° safety buffer area before physical hard limit, trigger sound and light alarm and automatically slow down; adaptive limit adjustment: dynamically adjust joint motion range according to current task, avoid unnecessary restrictions; limit learning mechanism: record commonly used joint range in actual work, optimize limit parameter setting; emergency braking strategy: hierarchical braking mechanism, from flexible deceleration to emergency stop, reduce mechanical impact.

[0085] Self-collision detection system, real-time collision prevention, including geometric model detection: establish accurate collision geometry based on robot arm CAD model; trajectory safety analysis: prospective collision detection on action sequence output by DRL; safe distance keeping: set minimum safe distance between different components (static 5cm, dynamic 10cm); protection level grading: distinguish between minor contact risk and serious collision risk, adopt different coping strategies Environmental collision prediction, dynamic environment modeling, including real-time environment perception: fuse multi-sensor data to build dynamic workspace map; safety corridor planning: define priority path and safety area for robot motion; interactive object monitoring: monitor relative position of interactive objects such as energy meter, meter box, and detection table body.

[0086] Dynamics constraints include joint torque limit, end speed limit, and environment perception speed regulation.

[0087] Joint torque limit, including real-time torque monitoring: 1000Hz high-frequency sampling of actual output torque of each joint; dynamic torque threshold: dynamically adjust the upper limit of safe torque according to joint position and speed; overheat protection: monitor motor temperature, establish temperature-torque derating curve; impact force suppression: detect and suppress sudden impact torque to protect transmission system.

[0088] End speed limit, including task-related speed limit: set different maximum speeds for different task stages, approach stage: 0.1m / s, grasp stage: 0.05m / s, transfer stage: 0.3m / s, placement stage: 0.05m / s; Environment perception speed regulation: automatically reduce running speed in crowded areas; smooth acceleration planning: limit jerk to ensure smooth motion; emergency deceleration curve: predefine safe deceleration trajectory to avoid sudden stop impact.

[0089] Abnormal recovery strategy includes retry on failed grasping, path blockage re-planning, sensor anomaly switching to safe mode.

[0090] A retry mechanism is set for the failed grasping, first, the failure cause is diagnosed: the failure cause is analyzed through force sensor and visual feedback, such as position deviation, incorrect posture, object sliding, and abnormal clamp; adaptive retry adjustment: the retry parameters are adjusted according to the failure cause, such as position compensation, posture correction, clamping force adjustment, and contact strategy optimization; the number of retries needs to be managed: the maximum number of retries (usually 3 times) is set to avoid infinite loop; a progressive strategy upgrade is adopted: after multiple failures, the grasping strategy is automatically switched to a more conservative one.

[0091] Path blocking re-planning, including real-time obstacle detection: continuously monitoring temporary obstacles in the workspace; multi-scheme path library: pre-computing multiple alternative paths for quick switching; online re-planning capability: generating a new path in real time based on the current environmental state; safety verification mechanism: collision detection and feasibility verification of the newly planned path.

[0092] Increase the sensor abnormality safety mode, including sensor health monitoring: real-time monitoring of the working state and data quality of each sensor; data consistency check: cross-verification of measurement results of different sensors; fault detection and isolation: quickly identifying faulty sensors and excluding them from the decision chain; degraded operation mode: switching to a safety control mode based on the remaining sensors when a sensor fails.

[0093] Embodiment Two The embodiment is an industrial robot control system based on deep reinforcement learning, which is used to implement the industrial robot control method as described in Embodiment One, comprising: An industrial robot equipped with an end effector for interacting with the to-be-detected article; At least one sensor configured to collect information about the state of the industrial robot itself and / or the working environment to form a current state; A controller in communication with the industrial robot and the sensor, wherein the controller stores a policy network trained according to the method as described in Embodiment One, and the controller is configured to: Receive real-time data from the at least one sensor to build the current state; Input the current state into the policy network to autonomously decide and generate control actions; Convert the control actions into control instructions and send them to the industrial robot to drive it to perform the loading and unloading task of the to-be-detected article.

[0094] It should be emphasized that all the above embodiments, although they are all detailed in the specific application scenario of using a six-axis robot arm to automatically load and unload an electricity meter, are only preferred examples for clearly illustrating the technical solutions of the present application, and do not constitute any form of limitation on the scope of protection of the present application.

[0095] The core idea of the present application is to provide a general intelligent control framework based on deep reinforcement learning. Through the training paradigm from simulation to reality, the original multi-objective composite reward function and the phased progressive learning strategy, the framework can enable the robot arm to autonomously learn the "perception-decision-execution" closed-loop control strategy, thereby mastering the complex skill of grasping target objects from random or semi-sequential stacks and accurately placing them in designated workstations.

[0096] The essence of this methodology is Object-Agnostic. For other different shapes, sizes and weights of the to-be-detected objects or to-be-processed workpieces, such as circuit boards (PCB) and mobile phone middle frames in the electronic manufacturing field, electronic modules on the automobile parts assembly line, precision components in the medical device production, etc., the person skilled in the art only needs to replace the corresponding 3D model in the virtual environment, adjust the identification target of the visual identification algorithm and adapt the end effector (gripper) as needed, so that the core training and control process of the present application can be completely reused without fundamentally changing the underlying algorithm framework.

[0097] Therefore, the technical solution of the present application has strong universality and scalability, and can be widely applied in many intelligent manufacturing fields such as 3C product manufacturing, automotive electronics, medical devices, logistics sorting, etc. Any automation scenario involving grasping and accurately positioning objects from disordered or semi-sequential state can reflect the technical value and beneficial effects of the present application.

[0098] The above is only the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the inventive concept, several modifications and improvements can be made, which are all within the scope of protection of the present application.

Claims

1. A method for controlling industrial robots based on deep reinforcement learning, characterized in that, Includes the following steps: Construct a virtual simulation environment that includes a robotic arm, items to be inspected, a conveyor line, and an inspection platform; Design the state space and action space for deep reinforcement learning; Design a reward function to guide the deep reinforcement learning training process, wherein the reward function is a weighted combination of task reward, efficiency reward, security reward and skill reward; Based on the reward function, a deep reinforcement learning algorithm aimed at maximizing the cumulative reward and policy entropy is used to train the policy network in stages. The trained policy network is deployed to the control system of a real robotic arm. The control system generates control actions based on the real-time acquired current state, driving the robotic arm to perform the loading and unloading tasks of the items to be inspected.

2. The method according to claim 1, characterized in that, The construction of the virtual simulation environment includes setting physics engine parameters, robotic arm dynamics parameters, visual environment parameters, and task parameters; The physical engine parameters include a gravitational acceleration of 9.81 m / s² and a friction coefficient of 0.3-0.

7. The dynamic parameters of the robotic arm include joint friction coefficient ±20%, damping coefficient ±15%, and load mass ±10%. The visual environment parameters include random variations in light intensity from 300 to 1000 lux and random replacement of the energy meter texture. The task parameters are randomly generated based on the initial position and orientation of the item to be detected in the bin.

3. The method according to claim 1, characterized in that, The state space includes: robotic arm body state, end effector state, conveyor line state, collision avoidance information, task state, gripper opening and closing state, and force sensor readings.

4. The method according to claim 1, characterized in that, The motion space includes: incremental joint displacement, fixture control commands, conveyor line motion control, and detection table start / stop control.

5. The method according to claim 1, characterized in that, The task reward is configured to provide a positive reward when the robotic arm successfully completes a preset sub-task node; the sub-task node includes at least successfully grasping the item to be inspected, successfully placing the item to be inspected on the inspection table, or successfully completing a complete loading and unloading cycle. The efficiency reward includes: a time penalty applied to each action step, and a distance reward provided based on the reduction in distance between the end effector of the robotic arm and the target point; The safety rewards include: a collision penalty applied when the robotic arm collides, and an overforce penalty applied when the joint torque of the robotic arm is detected to exceed a preset safety threshold; The skill rewards include: trajectory smoothness rewards or penalties set based on the change in joint angle vectors at adjacent time points, and energy consumption optimization rewards set based on the comparison between the actual output torque of the joint and the rated torque.

6. The method according to claim 1, characterized in that, The deep reinforcement learning training employs a flexible action-evaluation algorithm.

7. The method according to claim 1, characterized in that, The phased training includes at least the following: a first training phase to enable the robotic arm to learn basic motion skills, a second training phase to learn how to grasp and place the items to be tested, and a third training phase to learn how to cooperate with peripheral equipment to complete the entire loading and unloading process.

8. The method according to claim 1, characterized in that, After deploying the policy network to the control system of the real robotic arm, the system further includes: Collect the interaction data generated by the real robotic arm during operation, and use the interaction data to fine-tune the policy network; Among them, the fine-tuning of network parameters adopts a hierarchical learning rate strategy, using a smaller learning rate for the lower layers of the network and a larger learning rate for the higher layers.

9. The method according to claim 1, characterized in that, The method further includes real-time safety monitoring and anomaly handling while driving the robotic arm to perform a task, wherein the monitoring and handling includes at least one of the following: Workspace monitoring for constraining joint angles, detecting self-collisions, or predicting environmental collisions; Dynamic monitoring that limits joint torque or end-effector velocity; When a capture failure, path blockage, or sensor malfunction is detected, an exception recovery strategy is triggered to retry, replan, or switch to a safe mode.

10. An industrial robot control system based on deep reinforcement learning, characterized in that, The system is used to implement the industrial robot control method as described in any one of claims 1-9, including: Industrial robots equipped with end effectors for interacting with items to be inspected; At least one sensor is configured to collect information about the industrial robot’s own state and / or working environment to constitute the current state; A controller, communicatively connected to the industrial robot and sensors, stores a policy network trained according to the method of any one of claims 1-9, and the controller is configured to: Receive real-time data from the at least one sensor to construct the current state; The current state is input into the policy network to make autonomous decisions and generate control actions; The control actions are converted into control commands and sent to the industrial robot to drive it to perform the loading and unloading tasks of the items to be inspected.

Images