Intelligent agent decision-making system based on reinforcement learning

Abstract

The invention provides an intelligent agent decision-making system based on reinforcement learning, which comprises a three-dimensional visual sensor, a mechanical arm controller, a central processing unit, a conveyor belt simulation platform and an industrial Ethernet switch which are connected through the switch. Pose data of objects on the conveying belt are collected in real time through the three-dimensional vision sensor, and the mechanical arm controller executes track grabbing action and feeds back a result. The central processing unit is provided with a plurality of object pose points and grabbing track points, controls the conveying belt simulation platform to generate the set pose points, coordinates the robot, the sensor and the controller to complete pose evaluation, track decision and grabbing reward calculation under each pose point, and obtains a pose state value, a track action value and a grabbing reward value. The robot is connected with the sensor and the controller through a data interface and installed on the conveyor belt simulation platform, and sorting decision optimization in a dynamic environment is achieved. According to the method, the grabbing precision, the trajectory planning efficiency and the anti-offset capability of the industrial robot can be improved, and the labor cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology for intelligent agents, and more specifically, to an intelligent agent decision-making system based on reinforcement learning. Background Art

[0002] In the field of industrial production, with the continuous expansion of production scale and the increasing requirements for production efficiency and product quality, agent-based intelligent control has been widely used. In the field of intelligent sorting, traditional industrial robots mainly rely on preset programs and fixed position information to perform sorting tasks. Their working principle is to obtain the position information of objects through sensors, and then grasp and sort according to a preset trajectory. Although this sorting method has improved production efficiency to a certain extent, it has obvious limitations. For example, when the position, shape, or posture of an object changes, the robot often cannot accurately identify and grasp it, resulting in a high sorting failure rate. In addition, when faced with complex and changing industrial environments, traditional sorting robots lack adaptive capabilities and are unable to adjust their grasping strategies according to real-time environmental changes, which limits their application in a wider range of scenarios.

[0003] In the process of implementing the embodiments of the present invention, at least the following problems or defects exist in the existing technology: traditional industrial sorting robots have poor adaptability to the object's posture and are unable to accurately assess the actual posture of the object and effectively match it with the preset posture, resulting in low grasping accuracy; in terms of trajectory decision-making, there is a lack of effective optimization mechanism, and the grasping trajectory cannot be dynamically adjusted according to the actual grasping success rate and trajectory execution time, making it difficult to further improve grasping efficiency; in addition, the calculation of grasping rewards in the existing technology is not accurate enough, and it cannot accurately reflect the grasping effect, which in turn affects the robot's learning and optimization process. These problems restrict the performance of industrial sorting robots in complex dynamic environments and cannot meet the needs of modern industrial production for high-efficiency and high-precision sorting. Summary of the Invention

[0004] The present invention provides an intelligent agent decision-making system and method based on reinforcement learning.

[0005] In a first aspect of the present invention, there is provided an intelligent agent decision-making system based on reinforcement learning, comprising:

[0006] 3D vision sensor, robotic arm controller, central processing unit, conveyor belt simulation platform, industrial Ethernet switch;

[0007] The three-dimensional vision sensor, the robotic arm controller, the conveyor belt simulation platform, and the central processing unit are connected via an industrial Ethernet switch; the industrial sorting robot is installed on the conveyor belt simulation platform; the industrial sorting robot is connected to the three-dimensional vision sensor and the robotic arm controller via a data interface;

[0008] The three-dimensional vision sensor is used to collect the position data of the conveyor belt objects in real time under the control of the central processing unit; the robotic arm controller is used to execute the grasping trajectory action of the industrial sorting robot and feed back the execution results to the central processing unit;

[0009] The central processor sets multiple object pose points and multiple grasping trajectory points according to the dynamic environment characteristics of the industrial sorting robot, controls the conveyor belt simulation platform to generate the set object pose points in sequence, and at each object pose point, completes pose evaluation, trajectory decision-making, grasping reward calculation by controlling the industrial sorting robot, three-dimensional vision sensor and robotic arm controller, and obtains pose state value, trajectory action value and grasping reward value.

[0010] Furthermore, in the posture evaluation function, the central processing unit sends instructions to the industrial sorting robot at each set object posture point, so that the industrial sorting robot obtains the preset object coordinate features;

[0011] The 3D vision sensor captures the actual position of objects on the conveyor belt in real time, and analyzes the center coordinates of the objects as measured position features and uploads them to the central processor;

[0012] The central processing unit calculates the posture positioning deviation based on the preset object coordinate features of each object posture point and the measured object center coordinates, and writes the posture positioning deviation into the storage unit of the industrial sorting robot.

[0013] Furthermore, in the trajectory decision function, the central processor sends instructions to the industrial sorting robot at each set object pose point, causing the industrial sorting robot to perform multiple sampling grabbing trajectory points, and the sampling grabbing trajectory points all have preset end gripper speeds;

[0014] The robot controller executes each sampled grasping trajectory point action, records the actual grasping success rate and trajectory execution time and uploads it to the central processor;

[0015] Based on the difference between the actual grasping success rate and the preset grasping success rate threshold, the central processing unit calculates the trajectory optimization weight of each grasping trajectory point under each object posture point in combination with the trajectory execution time, and writes the trajectory optimization weight into the strategy database of the industrial sorting robot.

[0016] Furthermore, in the grasping reward calculation function, the central processor sends instructions to the three-dimensional vision sensor and the industrial sorting robot at each set object pose point, so that the three-dimensional vision sensor monitors the object offset of multiple sampling pose points;

[0017] The industrial sorting robot executes a preset reward calculation strategy based on the object offset, generates the actual grasping reward value, calculates its deviation from the preset reward benchmark, and uploads it to the central processor;

[0018] Based on the deviation and the object posture characteristics, the central processing unit calculates the reward correction coefficient of each grasping trajectory point under each object posture point, and writes the reward correction coefficient into the reward model library of the industrial sorting robot.

[0019] Furthermore, the industrial sorting robot includes an RGB-D vision module, a trajectory planning module, and a six-axis robotic arm execution module; the trajectory planning module includes a cascaded posture feature extractor, a Q-value decider, and a joint angle generator.

[0020] Furthermore, the central processing unit uses a deep double-Q network algorithm to calculate the policy gradient update amount of the trajectory optimization weight and the reward value update amount of the reward correction coefficient.

[0021] Furthermore, the policy gradient update amount includes a policy learning rate α and a policy decay factor β;

[0022] At each object pose point, it is calculated using the following formula:

[0023] The calculation formula of the strategy attenuation factor β is:

[0024]

[0025] The calculation formula of the policy learning rate α is:

[0026]

[0027] Among them, n is the number of sampled grasping trajectory points under a single object pose point, S state is the coordinate of the i-th object pose point collected by the 3D vision sensor, Q actual is the actual grasping success rate deviation value of the i-th trajectory action fed back by the robot controller.

[0028] Furthermore, the reward value update includes a reward scaling factor γ and a reward offset δ; the reward offset δ is calculated using the following formula at each object pose point:

[0029]

[0030] The reward scaling factor γ is calculated as:

[0031]

[0032] Among them, n is the number of sampled pose points under a single object pose point, O offsetis the object center coordinate offset of the i-th sampling pose point, R real The deviation of the grasping reward value for the i-th pose point reported by the industrial sorting robot.

[0033] Furthermore, the system also includes a real-time database; the real-time database stores historical training data of posture state values, trajectory action values ​​and grasping reward values ​​generated by the central processing unit.

[0034] In a second aspect of the present invention, a method for intelligent agent decision-making based on reinforcement learning is provided, comprising:

[0035] Obtain the current object position points on the conveyor belt through the three-dimensional vision sensor;

[0036] The industrial sorting robot reads the corresponding trajectory optimization weights based on the current object pose points and target grasping trajectory points, and generates the joint space optimized trajectory;

[0037] According to the current object pose point and reward calculation point, read the reward scaling factor γ and reward offset δ to calculate the grasping success rate reward value;

[0038] According to the current object pose point and pose evaluation point, the strategy learning rate α and strategy attenuation factor β are read to generate anti-offset pose compensation instructions.

[0039] The above embodiments of the present invention have at least the following beneficial effects:

[0040] 1. Through the coordinated control of the 3D vision sensor and the central processing unit, high-precision real-time acquisition and deviation calculation of the position of objects on the conveyor belt are achieved, solving the problem of grasping failure caused by object position offset in traditional sorting systems and improving the positioning accuracy and stability of sorting operations.

[0041] 2. The deep dual-Q network algorithm is used to dynamically optimize the grasping trajectory weight and reward correction coefficient, enabling the robotic arm to adaptively adjust its action strategy based on the actual grasping success rate and execution time, overcoming the problem of poor adaptability of fixed trajectory planning in complex environments and improving sorting efficiency and success rate.

[0042] 3. Through a closed-loop feedback mechanism of posture evaluation, trajectory decision-making, and grasping reward calculation, combined with dynamic adjustment of the strategy learning rate and attenuation factor, the system's ability to resist interference from changes in object posture is enhanced, solving the problem of insufficient grasping reliability of traditional sorting robots caused by environmental fluctuations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0044] Figure 1 A schematic diagram of the structure of an intelligent agent decision-making system based on reinforcement learning provided by one embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a process flow of an intelligent agent decision-making method based on reinforcement learning provided by one embodiment of the present invention;

[0046] Figure 3 The figure schematically shows the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0048] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0049] It should be noted that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0050] Reference below Figure 1 , Figure 1 This is a schematic diagram of the structure of an intelligent agent decision system based on reinforcement learning provided by an embodiment of the present invention. Figure 1 As shown in FIG, an intelligent agent decision-making system based on reinforcement learning includes:

[0051] 3D vision sensor 101, robotic arm controller 102, central processing unit 103, conveyor belt simulation platform 104, industrial Ethernet switch 105;

[0052] The three-dimensional vision sensor, the robotic arm controller, the conveyor belt simulation platform, and the central processing unit are connected via an industrial Ethernet switch; the industrial sorting robot is installed on the conveyor belt simulation platform; the industrial sorting robot is connected to the three-dimensional vision sensor and the robotic arm controller via a data interface;

[0053] The three-dimensional vision sensor is used to collect the position data of the conveyor belt objects in real time under the control of the central processing unit; the robotic arm controller is used to execute the grasping trajectory action of the industrial sorting robot and feed back the execution results to the central processing unit;

[0054] The central processor sets multiple object pose points and multiple grasping trajectory points according to the dynamic environment characteristics of the industrial sorting robot, controls the conveyor belt simulation platform to generate the set object pose points in sequence, and at each object pose point, completes pose evaluation, trajectory decision-making, grasping reward calculation by controlling the industrial sorting robot, three-dimensional vision sensor and robotic arm controller, and obtains pose state value, trajectory action value and grasping reward value.

[0055] It should be noted that the present invention relates to an intelligent agent decision-making system based on reinforcement learning, which realizes efficient and intelligent sorting operations by integrating multiple key components. Among them, the three-dimensional vision sensor is used to collect the posture data of objects on the conveyor belt in real time. The posture data here refers to the position and posture information of the object, including the center coordinates, angles and other parameters of the object. The robotic arm controller is responsible for executing the grasping trajectory action of the industrial sorting robot and feeding back the execution results to the central processing unit. The central processing unit is the control core of the entire system and is responsible for processing various data and instructions. The conveyor belt simulation platform provides the robot with a platform that simulates the actual working environment for testing and optimizing sorting strategies. The industrial Ethernet switch ensures high-speed data transmission and communication between the various components. The industrial sorting robot is connected to the three-dimensional vision sensor and the robotic arm controller through a data interface, thereby realizing accurate identification and grasping operations of objects.

[0056] Specifically, a 3D vision sensor is a device that can capture three-dimensional information about an object in real time. It acquires depth and position information by emitting and receiving light signals of specific wavelengths. The robotic arm controller is a key component controlling the motion of the robot's manipulator arm. It precisely executes grasping actions according to instructions from the central processing unit (CPU) and provides feedback to the CPU on various grasping process parameters, such as grasp success rate and trajectory execution time. The CPU is a high-performance computing unit that processes and analyzes collected data based on preset algorithms and models to perform functions such as pose estimation, trajectory decision-making, and grasp reward calculation. The conveyor belt simulation platform is a programmable device that can simulate conveyor belt operation under different speeds and load conditions, providing diverse training scenarios for the robot. An industrial Ethernet switch is a high-speed network device that supports real-time data transmission, ensuring latency-free communication between various system components. The industrial sorting robot is an automated device that integrates vision, control, and execution functions. It uses an RGB-D vision module to acquire color and depth information about objects, a trajectory planning module to generate the optimal grasping trajectory, and a six-axis robotic arm execution module to perform the actual grasping operation.

[0057] Preferably, when performing pose assessment, the central processing unit calculates pose positioning deviations based on preset object coordinate features and the measured object center coordinates. The preset object coordinate features here refer to theoretical coordinate values ​​set based on a standard model of the object during system initialization, while the measured object center coordinates refer to the actual position of the object captured in real time by the 3D vision sensor. The central processing unit determines the pose deviation by comparing the difference between the two and stores it in the robot's memory unit for subsequent trajectory planning and grasping strategy adjustment. During the trajectory decision process, the central processing unit calculates the optimization weight of each grasping trajectory point based on the difference between the actual grasping success rate and the preset grasping success rate threshold, combined with the trajectory execution time. The actual grasping success rate is fed back by the robot arm controller after executing a grasping action, while the preset grasping success rate threshold is an ideal value set based on historical data and experience, used to evaluate the effectiveness of the current grasping strategy. Trajectory execution time refers to the time it takes for the robot arm to start the movement and complete the grasp. The central processing unit uses these parameters to comprehensively evaluate the quality of each grasping trajectory and updates the trajectory optimization weight in the strategy database accordingly.

[0058] In some embodiments, in the posture evaluation function, the central processor sends instructions to the industrial sorting robot at each set object posture point, so that the industrial sorting robot obtains the preset object coordinate features;

[0059] The 3D vision sensor captures the actual position of objects on the conveyor belt in real time, and analyzes the center coordinates of the objects as measured position features and uploads them to the central processor;

[0060] The central processing unit calculates the posture positioning deviation based on the preset object coordinate features of each object posture point and the measured object center coordinates, and writes the posture positioning deviation into the storage unit of the industrial sorting robot.

[0061] It should be noted that the posture evaluation function is a key link in the intelligent agent decision-making system. Its purpose is to optimize the robot's grasping action through accurate posture deviation calculation. The central processing unit sends instructions to the industrial sorting robot at each set object posture point, so that the robot can obtain the preset object coordinate features. The preset object coordinate features here refer to the theoretical coordinate values ​​set according to the standard model of the object when the system is initialized, including parameters such as the center position and posture of the object. The three-dimensional vision sensor captures the actual posture of the object on the conveyor belt in real time, and parses the object center coordinates as the measured posture features and uploads them to the central processing unit. Based on the difference between the preset object coordinate features and the measured object center coordinates, the central processing unit calculates the posture positioning deviation and stores the deviation in the robot's storage unit for subsequent grasping strategy adjustments.

[0062] Specifically, during the pose assessment process, the central processing unit first sends instructions to the industrial sorting robot based on the preset object pose points. These pose points are pre-set based on the possible positions and postures of objects in actual application scenarios. The preset object coordinate features refer to the ideal position and posture parameters of the object at these pose points, such as the object's center coordinates and rotation angle. The 3D vision sensor uses its RGB-D vision module to capture the actual pose of objects on the conveyor belt in real time, parses the object's actual center coordinates, and uploads these as measured pose features to the central processing unit. After receiving this data, the central processing unit calculates the difference between the preset object coordinate features and the measured object center coordinates, namely the pose positioning deviation. This deviation value reflects the degree of difference between the actual pose of the object and the preset pose, which is important for subsequent grasping trajectory planning and motion adjustment.

[0063] Preferably, in the specific implementation process of posture evaluation, the central processing unit will calculate the posture positioning deviation based on the difference between the preset object coordinate features and the measured object center coordinates. This calculation process can be achieved through simple geometric operations. For example, the distance between two coordinate points is calculated to obtain the position deviation, and the angle difference is calculated to obtain the posture deviation. The central processing unit stores these deviation values ​​in the robot's storage unit so that the subsequent trajectory planning module can adjust the grasping trajectory according to these deviation values, thereby improving the accuracy and success rate of grasping. Specifically, the central processing unit will compare the preset object center coordinates and the measured object center coordinates to calculate the position deviation. At the same time, it will also compare the preset angle and the measured angle to calculate the posture deviation. These deviation values ​​will be used to adjust the robot's grasping action so that it can grasp objects more accurately.

[0064] In some embodiments, in the trajectory decision function, the central processor sends instructions to the industrial sorting robot at each set object pose point, causing the industrial sorting robot to perform multiple sampling grasping trajectory points, and the sampling grasping trajectory points all have preset end gripper speeds;

[0065] The robot controller executes each sampled grasping trajectory point action, records the actual grasping success rate and trajectory execution time and uploads it to the central processor;

[0066] Based on the difference between the actual grasping success rate and the preset grasping success rate threshold, the central processing unit calculates the trajectory optimization weight of each grasping trajectory point under each object posture point in combination with the trajectory execution time, and writes the trajectory optimization weight into the strategy database of the industrial sorting robot.

[0067] It should be noted that the trajectory decision function is a core component of the intelligent agent decision-making system. Its purpose is to improve the robot's grasping success rate and efficiency by optimizing the grasping trajectory. At each set object pose, the central processing unit (CPU) sends instructions to the industrial sorting robot, instructing it to execute actions at multiple sample grasping trajectory points. These sample grasping trajectory points are pre-set and used to evaluate grasping performance under different trajectories. The robot arm controller executes these actions, records the actual grasping success rate and trajectory execution time, and then uploads this data to the CPU. Based on the difference between the actual grasping success rate and the preset grasping success rate threshold, combined with the trajectory execution time, the CPU calculates the trajectory optimization weights for each grasping trajectory point at each object pose and stores these weights in the industrial sorting robot's policy database. This process ensures that the robot can select the optimal grasping trajectory for different object poses.

[0068] Specifically, during the trajectory decision process, the central processing unit (CPU) first sends instructions to the industrial sorting robot based on the set object pose, instructing the robot to execute multiple sampled grasping trajectory points. These sampled grasping trajectory points are pre-set based on possible grasping paths in actual application scenarios and are used to evaluate grasping performance under different trajectories. As the robot arm controller executes these actions, it records the actual grasping success rate and trajectory execution time. The actual grasping success rate is the ratio of the number of successful grasps to the total number of attempts after executing a grasping action; the trajectory execution time is the time it takes the robot arm to complete the grasp from the start of the action. After receiving this data, the CPU calculates the trajectory optimization weight for each grasping trajectory point at each object pose based on the difference between the actual grasping success rate and the preset grasping success rate threshold, combined with the trajectory execution time. The preset grasping success rate threshold is an ideal value set based on historical data and experience, and is used to evaluate the effectiveness of the current grasping strategy. The trajectory optimization weight reflects the quality of each grasping trajectory, and the CPU stores these weights in the robot's strategy database for subsequent grasping trajectory selection.

[0069] Preferably, during the specific implementation of trajectory decision-making, the central processing unit will calculate the trajectory optimization weight of each grasping trajectory point under each object pose point based on the difference between the actual grasping success rate and the preset grasping success rate threshold, combined with the trajectory execution time. Specifically, the central processing unit will first compare the actual grasping success rate with the preset grasping success rate threshold and calculate the deviation of the success rate. Then, combined with the trajectory execution time, the performance of each grasping trajectory will be comprehensively evaluated. For example, if the actual grasping success rate of a grasping trajectory is higher than the preset threshold and the trajectory execution time is short, then the optimization weight of this trajectory will be higher. The central processing unit will update the policy database based on these weights, so that the robot can select the optimal grasping trajectory in subsequent grasping operations. This process can be implemented through a series of algorithms. For example, a deep double Q network algorithm can be used to calculate the policy gradient update amount of the trajectory optimization weight, thereby realizing dynamic optimization of the grasping trajectory.

[0070] In some embodiments, in the grasping reward calculation function, the central processor sends instructions to the three-dimensional vision sensor and the industrial sorting robot at each set object pose point, so that the three-dimensional vision sensor monitors the object offset of multiple sampling pose points;

[0071] The industrial sorting robot executes a preset reward calculation strategy based on the object offset, generates the actual grasping reward value, calculates its deviation from the preset reward benchmark, and uploads it to the central processor;

[0072] Based on the deviation and the object posture characteristics, the central processing unit calculates the reward correction coefficient of each grasping trajectory point under each object posture point, and writes the reward correction coefficient into the reward model library of the industrial sorting robot.

[0073] It should be noted that the grasping reward calculation function is a key component of the intelligent agent's decision-making system. Its purpose is to optimize the robot's grasping strategy by evaluating grasping performance. At each set object pose, the central processing unit (CPU) sends instructions to the 3D vision sensor and the industrial sorting robot, instructing the 3D vision sensor to monitor the object offset at multiple sampled poses. Object offset refers to the distance the object moves relative to its preset position during the grasping process. The industrial sorting robot then executes the preset reward calculation strategy based on the object offset, generates the actual grasping reward value, and calculates its deviation from the preset reward benchmark. This data is then uploaded to the CPU. Based on this deviation and the object pose characteristics, the CPU calculates the reward correction coefficients for each grasping trajectory point at each object pose and stores these coefficients in the industrial sorting robot's reward model library. This process ensures that the robot can adjust its grasping strategy based on the feedback from the grasping performance, thereby improving the success rate and efficiency of grasping.

[0074] Specifically, the key components and parameters involved in the grasping reward calculation function include a 3D vision sensor, an industrial sorting robot, object offset, a preset reward calculation strategy, the actual grasping reward value, a preset reward benchmark, and a reward correction factor. The 3D vision sensor monitors the object offset in real time during the grasping process. This offset reflects the accuracy and stability of the grasping action. Based on the monitored object offset, the industrial sorting robot executes a preset reward calculation strategy, an algorithm that evaluates grasping effectiveness based on object offset. The actual grasping reward value is calculated based on this strategy and reflects the effectiveness of the current grasping action. The preset reward benchmark is an ideal value used to assess whether the actual grasping reward value meets expectations. The reward correction factor is calculated based on the deviation between the actual grasping reward value and the preset reward benchmark and is used to adjust the parameters of the subsequent grasping strategy. These parameters and components work together to enable the robot to self-optimize based on feedback on grasping effectiveness.

[0075] Preferably, during the implementation of the grasping reward calculation process, the central processing unit (CPU) executes a preset reward calculation strategy based on the object's offset. The specific implementation steps of this strategy include: First, the 3D vision sensor monitors the object's offset during the grasping process and uploads this data to the CPU. The CPU then calculates the actual grasping reward value based on the object's offset according to the preset reward calculation strategy. The CPU then compares the actual grasping reward value with a preset reward benchmark and calculates the deviation. Based on this deviation and the object's pose characteristics, the CPU calculates a reward correction coefficient. These coefficients reflect the difference between the grasping performance and the ideal state and are used to adjust the parameters of subsequent grasping strategies. For example, if the actual grasping reward value is lower than the preset reward benchmark, indicating an unsatisfactory grasping performance, the CPU calculates a larger reward correction coefficient to adjust the subsequent grasping strategy and improve the grasping success rate. This process can be implemented using a deep double-Q network algorithm, which dynamically adjusts the reward value based on the actual grasping performance, thereby optimizing the robot's grasping strategy.

[0076] In some embodiments, the industrial sorting robot includes an RGB-D vision module, a trajectory planning module, and a six-axis robotic arm execution module; the trajectory planning module includes a cascaded posture feature extractor, a Q-value decider, and a joint angle generator.

[0077] It should be noted that the industrial sorting robot in the present invention includes an RGB-D vision module, a trajectory planning module, and a six-axis robotic arm execution module, which work together to achieve efficient sorting operations. The RGB-D vision module is used to obtain the color and depth information of the object, the trajectory planning module is responsible for generating the optimal grasping trajectory based on this information, and the six-axis robotic arm execution module is responsible for performing the actual grasping action. The trajectory planning module contains a cascaded posture feature extractor, a Q-value decider, and a joint angle generator. These components together ensure that the robot can dynamically adjust the grasping strategy according to the posture characteristics of the object, thereby improving the accuracy and efficiency of grasping.

[0078] Specifically, the RGB-D vision module is an advanced visual sensor that can simultaneously acquire color images and depth information of an object. Color images are used to identify the color and shape of an object, while depth information is used to determine the object's spatial position. The trajectory planning module is the robot's brain, performing complex calculations and making decisions based on the information provided by the RGB-D vision module. The pose feature extractor is responsible for extracting the object's position and posture features from the visual data. These features include parameters such as the object's center coordinates and rotation angle. The Q-value decider evaluates the pros and cons of different grasping trajectories based on these features and a preset reward model, and selects the optimal trajectory. The joint angle generator calculates the motion angles of each joint of the robotic arm based on the selected trajectory, thereby generating specific grasping action instructions. The six-axis robotic arm execution module is the execution part of the robot, which accurately performs grasping actions according to the instructions generated by the trajectory planning module.

[0079] Preferably, in the specific implementation of the trajectory planning module, the pose feature extractor extracts the pose features of the object from the image data acquired by the RGB-D vision module. For example, it can use image processing algorithms to identify the object's outline and calculate the object's center coordinates and rotation angle. The Q-value decider evaluates the advantages and disadvantages of different grasping trajectories based on these pose features and a preset reward model. Specifically, it considers factors such as the grasping trajectory's success rate, execution time, and object offset to calculate the Q-value (expected reward) for each trajectory. The joint angle generator calculates the motion angle of each joint based on the selected trajectory and the robot's dynamic model. These angle values ​​are sent to the six-axis robot execution module to achieve precise grasping. This entire process can be implemented using deep learning algorithms, such as a deep double-Q network algorithm, to optimize the decision-making process of the Q-value decider, thereby improving the robot's grasping performance.

[0080] In some embodiments, the central processing unit uses a deep double-Q network algorithm to calculate the policy gradient update amount of the trajectory optimization weight and the reward value update amount of the reward correction coefficient.

[0081] It should be noted that the central processing unit uses a deep dual-Q network algorithm to calculate policy gradient updates for trajectory optimization weights and reward value updates for the reward correction coefficient, which is a key technical means for achieving intelligent decision-making in this invention. The deep dual-Q network algorithm is a reinforcement learning algorithm used to optimize the robot's grasping trajectory and reward calculation. Policy gradient updates are used to adjust the grasping strategy to make it more adaptable to dynamic environments, while reward value updates are used to optimize the reward model to better reflect grasping results. This algorithm enables the robot to dynamically adjust its grasping strategy in complex industrial environments, improving the success rate and efficiency of grasping.

[0082] Specifically, the Deep Dual Q Network algorithm is a reinforcement learning algorithm based on deep learning. It uses two Q networks to evaluate and optimize grasping trajectories. Policy gradient updates update the grasping policy parameters by calculating the gradient of the policy function. These parameters include the policy learning rate and the policy decay factor. The policy learning rate determines the speed of policy updates, while the policy decay factor adjusts the magnitude of policy updates. Reward value updates update the reward model parameters by calculating the gradient of the reward function. These parameters include the reward scaling factor and the reward offset. The reward scaling factor adjusts the size of the reward value, while the reward offset adjusts the reward baseline. The setting and updating of these parameters directly impact the robot's grasping decisions and reward calculations.

[0083] Preferably, when implementing the deep dual-Q network algorithm, the central processing unit (CPU) first initializes two Q networks: one for evaluating the value of the current policy and the other for selecting the optimal grasping trajectory. At each object pose, the CPU calculates a policy gradient update based on the object pose data collected by the 3D vision sensor and the grasp success rate deviation reported by the robotic arm controller. Specifically, the CPU calculates the policy gradient update based on a preset policy learning rate and policy decay factor, combined with the difference between the actual grasp success rate and a preset grasp success rate threshold. Similarly, the CPU calculates the reward value update based on a preset reward scaling factor and reward offset, combined with the object offset and the actual grasp reward value deviation. These updates are used to adjust parameters in the policy database and reward model library, thereby optimizing the robot's grasping strategy and reward calculation. This process can be achieved through a series of training steps, for example, by simulating different object poses and grasping scenarios, and having the robot perform numerous grasping attempts in a virtual environment, thereby continuously optimizing its grasping strategy and reward model.

[0084] In some embodiments, the policy gradient update amount includes a policy learning rate α and a policy decay factor β;

[0085] At each object pose point, it is calculated using the following formula:

[0086] The calculation formula of the strategy attenuation factor β is:

[0087]

[0088] The calculation formula of the policy learning rate α is:

[0089]

[0090] Among them, n is the number of sampled grasping trajectory points under a single object pose point, S state is the coordinate of the i-th object pose point collected by the 3D vision sensor, Q actualis the actual grasping success rate deviation value of the i-th trajectory action fed back by the robot controller.

[0091] It's important to note that calculating policy gradient updates is a crucial step in the deep double-Q network algorithm, which is used to optimize the grasping strategy of an industrial sorting robot. Policy gradient updates consist of a policy learning rate α and a policy decay factor β, two parameters that determine the speed and magnitude of policy updates. The policy learning rate α determines the step size of the policy adjustment during each update, while the policy decay factor β adjusts the degree of attenuation of the policy update to prevent excessive oscillation during the update process. By properly setting these two parameters, the robot can quickly adapt and optimize its grasping strategy in dynamic environments.

[0092] Specifically, the calculation of the policy gradient update involves several key parameters: the policy learning rate α, the policy decay factor β, the number of sampled grasp trajectory points n at the object pose point, the object pose point coordinates collected by the 3D vision sensor, and the actual grasp success rate deviation of the trajectory action reported by the robot controller. The policy learning rate α is a positive number that controls the step size of the policy update. A larger α value results in faster policy updates but may cause instability, while a smaller α value results in slower policy updates but greater stability. The policy decay factor β, a number between 0 and 1, controls the degree of policy update decay to prevent excessive oscillation during the policy update process. The number of sampled grasp trajectory points n at the object pose point represents the number of grasp attempts made at each object pose point. A larger number provides a more robust basis for policy updates. The object pose point coordinates collected by the 3D vision sensor provide precise information about the object's position and posture on the conveyor belt, while the actual grasp success rate deviation of the trajectory action reported by the robot controller reflects the effectiveness of the current grasping strategy.

[0093] Preferably, when calculating the policy gradient update, the initial values ​​of the policy learning rate α and the policy decay factor β must first be determined. These values ​​can be preset based on historical data and experience, or adjusted through experimentation. For example, the policy learning rate α can be set to 0.01, and the policy decay factor β can be set to 0.99. At each object pose point, the 3D vision sensor collects the object's pose data, and the robotic arm controller performs a grasping action, recording the actual grasping success rate deviation. The policy decay factor β and the policy learning rate α are then calculated based on this data. The policy decay factor β can be calculated by comparing the actual grasping success rate deviation with a preset threshold. A larger difference indicates a smaller β value, indicating a need for a more significant policy adjustment. The policy learning rate α can be calculated by analyzing the correlation between the grasping success rate deviation and the pose data. A higher correlation indicates a larger α value, indicating a need for a faster policy update. In this way, the central processing unit can dynamically adjust the policy gradient update, thereby optimizing the robot's grasping strategy and improving its adaptability and grasping success rate in complex environments.

[0094] In some embodiments, the reward value update includes a reward scaling factor γ and a reward offset δ; the reward offset δ is calculated using the following formula at each object pose point:

[0095]

[0096] The reward scaling factor γ is calculated as:

[0097]

[0098] Among them, n is the number of sampled pose points under a single object pose point, O offset is the object center coordinate offset of the i-th sampling pose point, R real The deviation of the grasping reward value for the i-th pose point reported by the industrial sorting robot.

[0099] It's important to note that the calculation of reward updates is a crucial step in the Deep Double Q Network algorithm, which optimizes the reward model for industrial sorting robots. The reward update consists of a reward scaling factor γ and a reward offset δ, which determine the magnitude and baseline of reward adjustments. The reward scaling factor γ adjusts the reward value so that it better reflects the quality of the grasping performance; the reward offset δ adjusts the reward baseline to ensure comparability across different grasping scenarios. By properly setting these two parameters, the robot can dynamically adjust its reward model based on grasping performance, thereby optimizing its grasping strategy.

[0100] Specifically, the calculation of reward updates involves several key parameters: the reward scaling factor γ, the reward offset δ, the number of sampled pose points n for each object pose, the offset of the object center coordinates of the sampled pose points, and the deviation of the grasp reward reported by the industrial sorting robot. The reward scaling factor γ is a positive number that controls the magnitude of the reward adjustment. A larger γ value results in larger reward changes, which has a greater impact on policy updates; a smaller γ value results in smaller reward changes, leading to smoother policy updates. The reward offset δ is a constant that adjusts the benchmark for reward values ​​to ensure comparability across different grasping scenarios. The number of sampled pose points n for each object pose point represents the number of grasp attempts performed at each object pose point. A larger number provides a more comprehensive basis for updating the reward model. The offset of the object center coordinates of the sampled pose points reflects the actual position deviation of the object during the grasp, while the deviation of the grasp reward value reflects the difference between the current grasping result and the preset target.

[0101] Preferably, when calculating the reward update, the initial values ​​of the reward scaling factor γ and the reward offset δ must first be determined. These values ​​can be preset based on historical data and experience, or adjusted through experimentation. For example, the reward scaling factor γ can be set to 0.1, and the reward offset δ can be set to 0.05. At each object pose point, a 3D vision sensor collects object pose data, and an industrial sorting robot performs a grasping action, recording the grasping reward deviation. The reward offset δ and the reward scaling factor γ are then calculated based on this data. The reward offset δ can be calculated by analyzing the average grasping reward deviation. A lower average value indicates a smaller δ value, indicating a baseline for increasing the reward. The reward scaling factor γ can be calculated by analyzing the correlation between the grasping reward deviation and the object's center coordinate offset. A higher correlation indicates a larger γ value, indicating a larger reward adjustment is needed. In this way, the central processing unit can dynamically adjust the reward update, thereby optimizing the robot's reward model and improving its adaptability and grasping success rate in complex environments.

[0102] In some embodiments, the system further includes a real-time database; the real-time database stores historical training data of posture state values, trajectory action values, and grasping reward values ​​generated by the central processor.

[0103] It's important to note that the system includes a real-time database that stores historical training data generated by the central processor, including pose states, trajectory action values, and grasping rewards. This data is crucial for the robot's learning and optimization, as it provides essential information for the deep double-Q network algorithm, enabling the robot to dynamically adjust its grasping strategy based on historical experience. The real-time database ensures timely and available data, enabling the robot to quickly adapt to environmental changes during actual operation, improving sorting efficiency and accuracy.

[0104] Specifically, a real-time database is a high-performance data storage system that can receive and store various data generated by the central processing unit in real time. The pose state value refers to the position and posture information of the object on the conveyor belt. This information is collected by the three-dimensional vision sensor and processed by the central processing unit. The trajectory action value refers to the specific parameters of the robot arm when performing a grasping action, including the grasping trajectory, joint angles, etc. The grasping reward value is a reward value calculated based on the grasping effect, reflecting the success rate and efficiency of the grasping action. These data are stored in the real-time database for subsequent analysis and learning. The design of the real-time database needs to take into account the fast reading and writing speed of the data, large-capacity storage, and the security and reliability of the data. For example, a distributed database architecture can be adopted to improve the efficiency of data processing and the scalability of the system.

[0105] Preferably, during the specific implementation of a real-time database, the following steps can be used to build and manage the database. First, define the database structure, including the data table structure for posture state values, trajectory action values, and grasping reward values. Each data table should contain necessary fields such as timestamp, object ID, posture coordinates, grasping trajectory parameters, and reward values. Second, establish a data collection and storage process to ensure that data generated by the central processing unit can be written to the database in real time. Message queue technology can be used to buffer data and prevent data loss. Then, implement data query and analysis capabilities so that the robot algorithm can learn and optimize based on historical data. For example, the SQL query language can be used to retrieve data within a specific time period, or data analysis tools can be used to discover patterns and trends in the data. Finally, regularly maintain and optimize the database, including data backup, index optimization, and performance monitoring, to ensure efficient operation. Through these steps, the real-time database can effectively support the learning and optimization process of industrial sorting robots, improving their adaptability and performance in complex environments.

[0106] The above embodiments of the present invention have the following beneficial effects:

[0107] 1. Through the coordinated control of the 3D vision sensor and the central processing unit, high-precision real-time acquisition and deviation calculation of the position of objects on the conveyor belt are achieved, solving the problem of grasping failure caused by object position offset in traditional sorting systems and improving the positioning accuracy and stability of sorting operations.

[0108] 2. The deep dual-Q network algorithm is used to dynamically optimize the grasping trajectory weight and reward correction coefficient, enabling the robotic arm to adaptively adjust its action strategy based on the actual grasping success rate and execution time, overcoming the problem of poor adaptability of fixed trajectory planning in complex environments and improving sorting efficiency and success rate.

[0109] 3. Through a closed-loop feedback mechanism of posture evaluation, trajectory decision-making, and grasping reward calculation, combined with dynamic adjustment of the strategy learning rate and attenuation factor, the system's ability to resist interference from changes in object posture is enhanced, solving the problem of insufficient grasping reliability of traditional sorting robots caused by environmental fluctuations.

[0110] like Figure 2 As shown, some embodiments provide an intelligent agent decision-making method based on reinforcement learning, the method comprising:

[0111] S1, obtain the current object position point of the conveyor belt through the three-dimensional vision sensor;

[0112] S2, the industrial sorting robot reads the corresponding trajectory optimization weights based on the current object pose points and the target grasping trajectory points, and generates the joint space optimized trajectory;

[0113] S3. Based on the current object pose and reward calculation point, read the reward scaling factor γ and the reward offset δ to calculate the grasping success rate reward value;

[0114] S4. According to the current object pose point and pose evaluation point, read the strategy learning rate α and strategy attenuation factor β to generate anti-offset pose compensation instructions.

[0115] It is understandable that the steps recorded in the intelligent agent decision-making method based on reinforcement learning are similar to those in the reference Figure 1 Therefore, the modules, features, and beneficial effects described above for the intelligent agent decision-making system based on reinforcement learning are also applicable to the intelligent agent decision-making method based on reinforcement learning and the operations contained therein, and will not be repeated here.

[0116] Reference below Figure 3, which shows a schematic structural diagram of an electronic device 300 suitable for implementing some embodiments of the present invention. The electronic devices in some embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0118] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0119] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, or a tablet.

[0120] The above descriptions are merely some preferred embodiments of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features having similar functions disclosed in the embodiments of the present invention.

Claims

1. An intelligent agent decision-making system based on reinforcement learning, characterized in that: The system includes a three-dimensional vision sensor, a robotic arm controller, a central processing unit, a conveyor belt simulation platform, and an industrial Ethernet switch; The three-dimensional vision sensor, the robotic arm controller, the conveyor belt simulation platform, and the central processing unit are connected via an industrial Ethernet switch; the industrial sorting robot is installed on the conveyor belt simulation platform; the industrial sorting robot is connected to the three-dimensional vision sensor and the robotic arm controller via a data interface; The three-dimensional vision sensor is used to collect the position data of the conveyor belt objects in real time under the control of the central processing unit; the robotic arm controller is used to execute the grasping trajectory action of the industrial sorting robot and feed back the execution results to the central processing unit; The central processor sets multiple object pose points and multiple grasping trajectory points according to the dynamic environment characteristics of the industrial sorting robot, controls the conveyor belt simulation platform to generate the set object pose points in sequence, and at each object pose point, completes pose evaluation, trajectory decision-making, grasping reward calculation by controlling the industrial sorting robot, three-dimensional vision sensor and robotic arm controller, and obtains pose state value, trajectory action value and grasping reward value.

2. The agent decision system according to claim 1, characterized in that: In the posture evaluation function, the central processor sends instructions to the industrial sorting robot at each set object posture point, so that the industrial sorting robot obtains the preset object coordinate features; The 3D vision sensor captures the actual position of objects on the conveyor belt in real time, and analyzes the center coordinates of the objects as measured position features and uploads them to the central processor; The central processing unit calculates the posture positioning deviation based on the preset object coordinate features of each object posture point and the measured object center coordinates, and writes the posture positioning deviation into the storage unit of the industrial sorting robot.

3. The intelligent agent decision system according to claim 1, characterized in that: In the trajectory decision function, the central processor sends instructions to the industrial sorting robot at each set object pose point, causing the industrial sorting robot to perform multiple sampling grabbing trajectory points, each of which has a preset end gripper speed; The robot controller executes each sampled grasping trajectory point action, records the actual grasping success rate and trajectory execution time and uploads it to the central processor; Based on the difference between the actual grasping success rate and the preset grasping success rate threshold, the central processing unit calculates the trajectory optimization weight of each grasping trajectory point under each object posture point in combination with the trajectory execution time, and writes the trajectory optimization weight into the strategy database of the industrial sorting robot.

4. The agent decision system according to claim 1, characterized in that: In the grasping reward calculation function, the central processor sends instructions to the three-dimensional vision sensor and the industrial sorting robot at each set object pose point, so that the three-dimensional vision sensor monitors the object offset of multiple sampling pose points; The industrial sorting robot executes a preset reward calculation strategy based on the object offset, generates the actual grasping reward value, calculates its deviation from the preset reward benchmark, and uploads it to the central processor; Based on the deviation and the object posture characteristics, the central processing unit calculates the reward correction coefficient of each grasping trajectory point under each object posture point, and writes the reward correction coefficient into the reward model library of the industrial sorting robot.

5. The intelligent agent decision system according to claim 1, characterized in that: The industrial sorting robot includes an RGB-D vision module, a trajectory planning module, and a six-axis robotic arm execution module; the trajectory planning module includes a cascaded posture feature extractor, a Q-value decider, and a joint angle generator.

6. The intelligent agent decision system according to claim 3 or 4, characterized in that: The central processing unit uses a deep double Q network algorithm to calculate the policy gradient update amount of the trajectory optimization weight and the reward value update amount of the reward correction coefficient.

7. The intelligent agent decision system according to claim 6, characterized in that: The policy gradient update amount includes the policy learning rate α and the policy attenuation factor β; At each object pose point, it is calculated using the following formula: The calculation formula of the strategy attenuation factor β is: The calculation formula of the policy learning rate α is: Among them, n is the number of sampled grasping trajectory points under a single object pose point, S state is the coordinate of the i-th object pose point collected by the 3D vision sensor, Q actual is the actual grasping success rate deviation value of the i-th trajectory action fed back by the robot controller.

8. The intelligent agent decision system according to claim 6, characterized in that: The reward value update includes the reward scaling factor γ and the reward offset δ; it is calculated by the following formula at each object pose point: The calculation formula of the reward offset δ is: The reward scaling factor γ is calculated as: Among them, n is the number of sampled pose points under a single object pose point, O offset is the object center coordinate offset of the i-th sampling pose point, R real The deviation of the grasping reward value for the i-th pose point reported by the industrial sorting robot.

9. The intelligent agent decision-making system according to claim 1, characterized in that: The system further includes a real-time database; the real-time database stores historical training data of posture state values, trajectory action values, and grasping reward values ​​generated by the central processing unit.

10. Dynamic decision-making method for industrial sorting robots, characterized in that: The industrial sorting robot is an industrial sorting robot in the system according to any one of claims 1 to 9; the method comprises: Obtain the current object position points on the conveyor belt through the three-dimensional vision sensor; The industrial sorting robot reads the corresponding trajectory optimization weights based on the current object pose points and target grasping trajectory points, and generates the joint space optimized trajectory; According to the current object pose point and reward calculation point, read the reward scaling factor γ and reward offset δ to calculate the grasping success rate reward value; According to the current object pose point and pose evaluation point, the strategy learning rate α and strategy attenuation factor β are read to generate anti-offset pose compensation instructions.

Images