End-to-end autonomous motion decision-making method based on deep reinforcement learning

Abstract

The invention discloses an end-to-end autonomous motion decision method based on deep reinforcement learning. The method relates to the fields of multi-robot motion, machine learning, integrated decision and the like. The method comprises the following steps: firstly, realizing accurate identification of a glass product by adopting improved Mamba-YOLOv10; and then, aiming at the data dependence problem of end-to-end autonomous motion, proposing that an empirical playback mechanism HER and a designed reward function are applied to a deep Q network to realize an HER-DQN integrated decision framework. Compared with other methods, the end-to-end autonomous motion decision-making method not only can accurately identify glass products under complex illumination conditions and environmental interference, but also can improve the stability and robustness of the biochemical laboratory transfer robot during autonomous motion decision-making, and the designed end-to-end autonomous motion decision-making method has real-time performance and high efficiency, and is suitable for popularization and application. The transfer robot in the biochemical laboratory can safely and autonomously complete execution tasks, and the transfer robot can be applied to the fields of agriculture, industry, service industry and the like and has beneficial effects.

Description

Technical Field

[0001] The present invention relates to the fields of multi-robot motion, machine learning, integrated decision-making, and specifically to an end-to-end autonomous motion decision-making method based on deep reinforcement learning. Background Art

[0002] With the rapid development of biochemical research and the widespread application of high-throughput analytical technology, the cross-regional transportation of laboratory vessels has become a key link in the modern scientific research process. In order to improve the efficiency of transportation, biochemical laboratory transportation robots have also developed rapidly. In the sample transportation of biochemical laboratories, the accurate identification of glassware is the core link to ensure safe transportation. However, the current biochemical laboratories generally adopt a transparent layout and use transparent acrylic laboratory tables. Although the space utilization rate is maximized and the operation is convenient, the refractive properties of acrylic materials and the light transmittance and high reflectivity of glassware themselves will cause mirror reflection on the surface of the laboratory vessel. At the same time, the visual interference caused by the transmitted light will significantly affect the accurate identification of the characteristic points of the vessel by the transportation robot; secondly, because glassware is easily broken due to collision, vibration or improper operation during transportation, it will cause liquid samples to leak and pollute the environment and threaten the health of the experimenters. At the same time, the speed of the transportation robot during transportation, the sudden change, emergency obstacle avoidance, vibration and other non-smooth movement methods may also increase the risk of glassware breakage and leakage, so there are extremely high requirements for the safety and stability of the transportation process.

[0003] Existing autonomous motion systems based on robotic arms or mobile platforms generally adopt a multi-module architecture of perception-planning-decision-execution. Although it can realize autonomous and safe navigation of robots, the heterogeneous communication between modules causes time delays that affect the real-time control of robots. The accumulation of non-deterministic delays between subsystems will cause motion trajectory deviations, resulting in a decrease in the positioning accuracy of the robot's end effector. In contrast, the end-to-end integrated decision-making control method has a closed-loop characteristic in which sensor data directly drives the output of motion commands. The transfer robot can effectively improve its work efficiency when performing tasks and can perform real-time feedback processing on complex changing environments, becoming a hot technology for mobile robots to complete autonomous motion.

[0004] In view of the above problems and practical needs, the present invention designs an end-to-end autonomous motion decision-making method based on deep reinforcement learning. First, in view of the unique environmental characteristics of the biochemical laboratory mentioned above, the Mamba model is combined with the YOLOv10 algorithm. Because of its dynamic sparse convolutional neural network structure, it can suppress mirror reflection artifacts through an adaptive weight distribution mechanism in strong reflection interference scenes, and at the same time, it uses multi-scale residual connections to enhance the edge sharpness perception of glass contours under weak light conditions. It can effectively identify laboratory glass doors, glassware, etc., and prevent the collision of transparent obstacles during robot transportation; secondly, in view of the problem of end-to-end autonomous motion dependence on data, experience replay (Hindsight Experience Replay, HER) is introduced to improve the stability and sample utilization rate during training. Finally, in view of the problems of poor real-time performance, low efficiency, and uneven motion caused by the traditional modularization of autonomous motion, a reward function is designed that is safety-oriented and ensures the smoothness of motion and the efficiency of completing tasks. The experience replay mechanism and the designed reward function were applied to the deep Q network, and the design of an integrated decision-making framework based on the Hindsight Experience Replay-Deep Q-Network (HER-DQN) was realized. This allows the transfer robot to use the HER-DQN model to achieve safe and efficient autonomous navigation in the biochemical laboratory, avoid the breakage of fragile samples and leakage of liquid samples, and realize the safe transportation of samples. Summary of the invention

[0005] The present invention designs an end-to-end autonomous motion decision-making method based on deep reinforcement learning.

[0006] The specific implementation steps are as follows:

[0007] Step 1: Construct a virtual environment and system architecture. In this method, the proposed algorithm needs to be repeatedly trained and tested in a dynamically changing environment. First, an initial environment is set, and the mobile robot system, pedestrians, other glass experimental tools and other elements are added through software tools to build and improve the virtual environment. The mobile robot system consists of a robot body, a sensor module, a processing module and an execution module. The sensor module is responsible for using the target detection results of Mamba-YOLOv10 to identify glass products and surrounding complex environment information; the processing module is used to process the information collected by the sensor to perform tasks such as positioning, identification and navigation; the execution module controls the robot movement according to the calculation results of the processing module.

[0008] Step 2: Dynamic recognition of glass products. The Mamba model uses a dynamic sparse convolutional neural network structure, which, combined with the YOLOv10 algorithm, can perform feature analysis on glass products under complex lighting conditions and environmental interference, providing sample information for the decision-making process. Training the Mamba-YOLOv10 model enables the robot to identify and locate glass products in a long-distance environment, and also extract information such as the location, category, and confidence of the glass products, providing data support for subsequent decision-making processes and obstacle avoidance.

[0009] The degree of match between the predicted box and the true box is called the consistent match metric, and the formula is as follows:

[0010] ,

[0011] in, represents the spatial prior, is the classification confidence, Represent the predicted box and the true box respectively, It is the intersection of the predicted box and the true box;

[0012] Target score The calculation formula is:

[0013] ,

[0014] is the target score of the nth positive sample, Represents the maximum value of IoU among all positive samples, Represents the positive sample used to measure the degree of alignment between the predicted box and the true box. Similarly for the maximum value;

[0015] Classification loss calculation formula:

[0016] l n =− oh n [ t n ⋅ log s ( x n ) + ( 1 − t n ) ⋅ log ( 1 − s ( x n ) ) ] ,

[0017] are the coefficients of the loss function, represents the predicted value of the nth sample, represents the activation function;

[0018] Coordinate loss calculation formula:

[0019] ,

[0020] is the intersection-over-union ratio of the nth positive sample, the target score is the weight coefficient, is the time when the nth sample bounding box is formed;

[0021] The total loss is the weighted sum of classification loss, coordinate loss, and confidence loss, and the formula is as follows:

[0022] ,

[0023] in, is the weighting coefficient, is the confidence loss.

[0024] Step 3: Design state space and action space. Use the Experience Replay-Deep Q Network (HER-DQN) algorithm to achieve navigation, set the state in the virtual environment built in step 1, set the pedestrian state, and set the robot state. The robot's state information is a key component of operation. This information includes the robot's precise position, the speed and direction of movement, and other data that may affect the robot's behavior. Similarly, setting the specific position of pedestrians, the speed of movement, and the direction of travel are important for understanding and predicting the behavior of pedestrians and robots.

[0025] Assume the position of the robot in the environment is , the location of the target point is , the position of the obstacle closest to the robot is ,but is the distance between the robot and the target location, is the distance between the robot and the obstacle. At the same time, the angle between the robot position and the target location is calculated as , the angle between the robot and the obstacle is .choose As the state space of the robot in the environment, it can express the state of the robot at a certain moment in the environment.

[0026] The action space is used to describe the robot's actions such as moving forward, backward, and changing direction in the current environment. After setting the robot's starting point and target point, the robot moves as a particle in the environment. In the actual movement process, the robot's movement process is a continuous state, so the robot's action space is a discrete action. The specific actions include moving forward, backward, turning left, turning right, turning left 45°, and turning right 45°. The purpose of adding corner actions is to improve the robot's ability to explore corner situations in the environment and avoid increasing the number of training times due to too little action space.

[0027] Step 4: Design a reward function. Designing a reward function mainly includes two aspects: reaching the target and avoiding obstacles. Designing a suitable reward function requires considering multiple factors such as path length, obstacle avoidance effect, and speed to the target point. In this method, a design scheme of three comprehensive reward functions is proposed, which considers target point reward, obstacle avoidance reward, and speed reward.

[0028] Step 4.1: Set the reward for reaching the target location. When the robot successfully reaches the target location, a large positive reward should be given to encourage the robot to complete the decision-making task quickly and efficiently. Therefore, when the robot is near the target location, the reward function should output a large positive value to encourage the robot to find the target point, and when a negative value is obtained, it is a penalty for the mobile robot;

[0029] R t a r g e t = { 1 , d m [ − 0 . 5 , 0 . 5 ] 20 ⋅ d m − 1 , − 1 < d m <− 0 . 5 , 0 . 5 < d m < 1 0 , others ,

[0030] in It is the reward for reaching the destination. is the maximum distance between the robot and the target location.

[0031] Step 4.2: Set the obstacle avoidance reward function. In order to ensure that the robot can effectively avoid obstacles in a complex and changing environment and reduce the risk of contact with surrounding objects and people, an obstacle avoidance reward function is designed. This function can not only help the robot accurately locate and avoid various obstacles, but also enhance the robot's autonomous learning ability. When the robot identifies an obstacle and takes measures to maintain a safe distance, the system will automatically give the robot a certain positive reward. When the robot fails to detect the obstacle in time or approaches or collides with the obstacle during the task, it will give a negative reward. Positive rewards are incentives for good performance of the robot, while negative rewards are penalties for improper behavior. Through this combination of reward and punishment strategy, the robot can continuously improve its obstacle avoidance ability and improve its adaptability and reliability in different environments.

[0032] ,

[0033] in is the obstacle avoidance reward function, is the distance from the robot to the obstacle.

[0034] Step 4.3: Set speed rewards. The robot should not move too fast in a complex road environment to avoid behaviors such as moving too fast, colliding, and slipping. At the same time, the robot should be encouraged to reach the destination more efficiently, and a speed reward mechanism should be added to the decision-making process. When the robot is able to move faster than the predetermined speed, the system should give positive feedback and rewards in a timely manner. This method can ensure that the robot can complete the task in the shortest time and the best path according to the set goals.

[0035] Set the speed reward function to:

[0036] ,

[0037] Among all the reward functions designed above, the safety of the robot when transferring products is the most important, followed by the requirement to reach the transfer location. When the above three reward functions conflict, first ensure safety and avoid collision, then control the speed reward to avoid collision, and finally reach the target point as required. The final overall system reward function is :

[0038] ,

[0039] in, is the absolute distance from the robot to the target location, It's time. , , They are reaching the target point, obstacle avoidance and speed reward functions, , , is the coefficient of each reward function, satisfying .

[0040] Step 5: Calculate the temporal difference target (TD-Target). In HER-DQN, TD-Target is used to calculate the loss function. The network parameters of the Q value are updated by minimizing the mean square error between the predicted Q value and TD-Target to ensure that the optimal strategy can be effectively learned.

[0041] TD-Target is calculated by the following formula:

[0042] ,

[0043] Set the loss function of the HER-DQN algorithm:

[0044] L ( i i ) = F [ ( R + l max A ' Q ( S ' , A ' , i ¯ i ) − Q ( S , A , i i ) ) 2 ] ,

[0045] Substitute into TD-Target and simplify:

[0046] L ( i i ) = F [ ( y i − Q ( S , A , i i ) ) 2 ] ,

[0047] in is the discount factor, It's the state. It's a reward. It's behavior. are the coefficients of the function, It is the state of the next moment. It is the behavior of the next moment. are the parameters of the Q network.

[0048] Step 6: Training experience replay system. The robot continues to interact with the environment, storing experience to update the neural network. The robot can also continuously interact and learn with the environment during the training of the HER-DQN algorithm, including the target location information detected by Mamba-YOLOv10, and can adjust the best path according to the complexity of the ground. Finally, it learns to choose the best action in a given state and form a complete environmental state.

[0049] Step 7: Model evaluation test, control execution output, through simulation environment testing and experiments, comprehensively evaluate the performance of the HER-DQN model in the robot's end-to-end autonomous motion decision-making. The reward function plays a key role in the testing and control execution stages. By reasonably designing the reward function and updating the Q value, the mobile robot can be guided to complete the task efficiently. The specific steps are:

[0050] Step 1: Test in a simulation environment to observe whether the algorithm model can help the robot to accurately identify glass products, make decisions during transportation, avoid obstacles, and other tasks. Record the reward value for each test, calculate the average reward to evaluate the performance, and evaluate the stability and consistency of the model through multiple tests.

[0051] Step 2: Visual test, use visualization tools (such as MATLAB) to draw reward curves and observe the learning process and convergence of the model;

[0052] Step 3: Control execution output, the control execution converts the output of the HER-DQN model into actual robot control instructions, and trains the HER-DQN model to select actions by designing a reward function. According to the current state input model, select the action with the highest Q value;

[0053] Step 4: Convert actions into control instructions. Convert the actions output by the HER-DQN model into control instructions that can be executed by the robot.

[0054] The present invention designs a robot autonomous motion decision-making method to solve the problem of safe and efficient autonomous navigation of a biochemical laboratory transport robot. Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0055] 1. The improved Mamba-YOLOv10 target detection model of the present invention can accurately identify the characteristics of glass products in biochemical laboratories under complex lighting and environmental interference through algorithm optimization;

[0056] 2. The improved HER-DQN algorithm of the present invention improves the stability and robustness of the biochemical laboratory transport robot in autonomous motion decision-making, and also improves the smoothness of the transport robot during operation;

[0057] 3. The end-to-end autonomous motion decision-making method designed by the present invention is real-time and efficient, and can enable the biochemical laboratory transfer robot to complete its tasks safely and autonomously. This method has beneficial effects in agriculture, industry, service industry and other fields.

[0058] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is an overall flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Attached Figure 1 It is an overall flow chart for the implementation of the present invention. This embodiment provides an end-to-end autonomous motion decision-making method based on deep reinforcement learning. Taking the Biochemical Laboratory Sample Transfer Robot (BLSTR) as an example, it specifically includes the following processes: building a virtual environment and system architecture, dynamic identification and precise positioning of glass products, designing state space and action space, designing reward function, calculating loss function according to time difference target, training experience playback system, model evaluation test, and controlling execution output.

[0062] The specific implementation instructions are as follows:

[0063] Implementation step 1: The environment and the agent are the main bodies of the deep reinforcement learning system. In this embodiment, the environment is set as a biochemical laboratory, and the agent is a biochemical laboratory sample transfer robot (BLSTR). The environment determination and system architecture are defined as follows.

[0064] Step 1.1: Build a closed biochemical laboratory environment. According to the actual environment of the biochemical laboratory, use the Gazebo simulator in the ROS system to build the real environment of the biochemical laboratory. It can simulate the robot's behavior of identifying glassware and transporting samples in the biochemical laboratory.

[0065] Step 1.2: Add the biochemistry laboratory transfer robot system, which consists of the robot body, sensor module, processing module and execution module. The sensor module is responsible for identifying glass products and surrounding interference environment information using the target detection results of Mamba-YOLOv10; the processing module is used to process the information collected by the sensor to perform tasks such as positioning, identification and navigation; the execution module controls the robot movement according to the calculation results of the processing module.

[0066] Implementation step 2: Dynamic identification of glass products. The Mamba-YOLOv10 algorithm is used to analyze the characteristics of glass products under different light conditions on the collected data to provide sample information for the decision-making process. The Mamba-YOLOv10 model is trained to enable BLSTR to identify and locate glass products in a long-distance environment. It can also extract information such as the location, category, and confidence of glass products, providing data support for subsequent decision-making processes and obstacle avoidance.

[0067] The degree of match between the predicted box and the true box is called the consistent match metric, and the formula is as follows:

[0068] ,

[0069] in, represents the spatial prior, is the classification confidence, Represent the predicted box and the true box respectively, It is the intersection of the predicted box and the true box;

[0070] Target score The calculation formula is:

[0071] ,

[0072] is the target score of the nth positive sample, Represents the maximum value of IoU among all positive samples, Represents the positive sample used to measure the degree of alignment between the predicted box and the true box. Similarly for the maximum value;

[0073] Classification loss calculation formula:

[0074] l n =− oh n [ t n ⋅ log s ( x n ) + ( 1 − t n ) ⋅ log ( 1 − s ( x n ) ) ] ,

[0075] are the coefficients of the loss function, represents the predicted value of the nth sample, represents the activation function;

[0076] Coordinate loss calculation formula:

[0077] ,

[0078] is the intersection-over-union ratio of the nth positive sample, the target score is the weight coefficient, is the time when the nth sample bounding box is formed;

[0079] The total loss is the weighted sum of classification loss, coordinate loss, and confidence loss, and the formula is as follows:

[0080] ,

[0081] in, is the weighting coefficient, is the confidence loss.

[0082] Implementation step 3: Design state space and action space. Use the Experience Replay-Deep Q Network (HER-DQN) algorithm to implement navigation, and set the researcher state and robot (BLSTR) state in the virtual environment built in step 1. Setting the state information of BLSTR is a key component of operation. This information includes the precise location of BLSTR, the speed and direction of movement, and data that may affect BLSTR's behavior. Similarly, setting the specific location of the researcher, the speed of movement, and the direction of travel are important for understanding and predicting the behavior of the researcher and the robot.

[0083] Assume that the location of BLSTR in the laboratory is , the location of the target point is , the position of the obstacle closest to BLSTR is ,but is the distance between BLSTR and the target location, is the distance between BLSTR and the obstacle. At the same time, the angle between the BLSTR position and the target location is calculated as , the angle between BLSTR and the obstacle is .choose As the state space of BLSTR in the laboratory, it can express the state of BLSTR in the laboratory at a certain moment.

[0084] The action space is used to describe the actions that BLSTR takes in the current laboratory environment, such as moving forward, moving backward, changing direction, etc. After setting the starting point and target point of BLSTR, BLSTR moves as a particle in the environment. In the actual movement process, the movement process of BLSTR is a continuous state, so the action space of BLSTR is a discrete action. The specific actions include moving forward, moving backward, turning left, turning right, turning left 45°, and turning right 45°. The purpose of adding corner actions is to improve BLSTR's ability to explore corner situations in the environment and avoid increasing the number of training times due to too little action space.

[0085] Implementation step 4: Design reward function. Designing reward function mainly includes two aspects: reaching the target and avoiding obstacles. Designing a suitable reward function needs to consider multiple factors such as path length, obstacle avoidance effect, speed to reach the target point, etc. In this method, a design scheme of three comprehensive reward functions is proposed, which considers target point reward, obstacle avoidance reward, and speed reward.

[0086] Step 4.1: Set the reward for reaching the target location. When BLSTR successfully reaches the target location, a large positive reward should be given to encourage BLSTR to complete the decision-making task quickly and efficiently. Therefore, when the robot is near the target location, the reward function should output a large positive value to encourage BLSTR to find the target point, and when a negative value is obtained, it is a penalty for the mobile robot;

[0087] R t a r g e t = { 1 , d m [ − 0 . 5 , 0 . 5 ] 20 ⋅ d m − 1 , − 1 < d m <− 0 . 5 , 0 . 5 < d m < 1 0 , others ,

[0088] in It is the reward for reaching the destination. is the maximum distance between the robot and the target location.

[0089] Step 4.2: Set the obstacle avoidance reward function. In order to ensure that the robot can effectively avoid obstacles in a complex and changing environment and reduce the risk of contact with surrounding objects and people, an obstacle avoidance reward function is designed. This function can not only help the robot accurately locate and avoid various obstacles, but also enhance the autonomous learning ability of BLSTR. When BLSTR identifies an obstacle and takes measures to maintain a safe distance, the system will automatically give BLSTR a certain positive reward. When BLSTR fails to find the obstacle in time or approaches or collides with the obstacle during the task, it will give a negative reward. Positive rewards are incentives for BLSTR to perform well, while negative rewards are penalties for inappropriate behavior. Through this combination of reward and punishment strategy, the robot can continuously improve its obstacle avoidance ability and improve its adaptability and reliability in different environments.

[0090] ,

[0091] in is the obstacle avoidance reward function, is the distance from the robot to the obstacle.

[0092] Step 4.3: Set speed reward. BLSTR should not move too fast in complex road environments to avoid behaviors such as moving too fast, collisions, and slipping. At the same time, the robot should be encouraged to reach the destination more efficiently, and a speed reward mechanism should be added to the decision-making process. When BLSTR is able to move faster than the predetermined speed, the system should give positive feedback and rewards in a timely manner. This method can ensure that the robot can complete the task in the shortest time and the best path according to the set goals.

[0093] Set the speed reward function to:

[0094] ,

[0095] Among all the reward functions designed above, the safety of the robot when transferring products is the most important, followed by the requirement to reach the transfer location. When the above three reward functions conflict, first ensure safety and avoid collision, then control the speed reward to avoid collision, and finally reach the target point as required. The final overall system reward function is :

[0096] ,

[0097] in, is the absolute distance from the robot to the target location, It's time. , , They are reaching the target point, obstacle avoidance and speed reward functions, , , is the coefficient of each reward function, satisfying .

[0098] Step 5: Calculate the temporal difference target (TD-Target). In HER-DQN, TD-Target is used to calculate the loss function. The network parameters of the Q value are updated by minimizing the mean square error between the predicted Q value and TD-Target to ensure that the optimal strategy can be effectively learned.

[0099] TD-Target is calculated by the following formula:

[0100] ,

[0101] Set the loss function of the HER-DQN algorithm:

[0102] L ( i i ) = F [ ( R + l max A ' Q ( S ' , A ' , i ¯ i ) − Q ( S , A , i i ) ) 2 ] ,

[0103] Substitute into TD-Target and simplify:

[0104] L ( i i ) = F [ ( y i − Q ( S , A , i i ) ) 2 ] ,

[0105] in is the discount factor, It's the state. It's a reward. It's behavior. are the coefficients of the function, It is the state of the next moment. It is the behavior of the next moment. are the parameters of the Q network.

[0106] Implementation step 6: Training experience replay system. BLSTR continues to interact with the environment, storing experience to update the neural network. The robot can also continuously interact and learn with the environment during the training of the HER-DQN algorithm, including the target location information detected by Mamba-YOLOv10, and can adjust the optimal path according to the complexity of the ground. Finally, it learns to choose the best action in a given state and forms a complete environmental state.

[0107] Implementation step 7: Model evaluation test, control execution output, through simulation environment testing and experiments, comprehensively evaluate the performance of the HER-DQN model in BLSTR end-to-end autonomous motion decision-making. The reward function plays a key role in the testing and control execution stages. By reasonably designing the reward function and updating the Q value, the mobile robot can be guided to complete the task efficiently. The specific steps are:

[0108] Step 1: Test in a simulation environment to observe whether the algorithm model can help the robot to accurately identify glass products, make decisions during transportation, avoid obstacles, and other tasks. Record the reward value for each test, calculate the average reward to evaluate the performance, and evaluate the stability and consistency of the model through multiple tests.

[0109] Step 2: Visual test, use visualization tools (such as MATLAB) to draw reward curves and observe the learning process and convergence of the model;

[0110] Step 3: Control execution output, the control execution converts the output of the HER-DQN model into actual robot control instructions, and trains the HER-DQN model to select actions by designing a reward function. According to the current state input model, select the action with the highest Q value;

[0111] Step 4: Convert actions into control instructions. Convert the actions output by the HER-DQN model into control instructions that can be executed by the robot.

Claims

1. An end-to-end autonomous motion decision-making method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Build a virtual environment and system architecture; Step 2: Dynamic identification of glass products; Step 3: Design state space and action space; Step 4: Design three reward functions: reward for reaching the target point, reward for avoiding obstacles, and reward for speed; Step 5: Calculate the temporal difference target TD-Target and loss function; Step 6: Training experience replay mechanism; Step 7: Model evaluation test, control execution output; Step 1: Build environment test; Step 2: Visualize and draw the reward function curve; Step 3: Control the execution output and design the reward function to train the HER-DQN model to select actions; Step 4: The actions output by the HER-DQN model are converted into instructions to control the robot’s execution.

2. The end-to-end autonomous motion decision-making method based on deep reinforcement learning according to claim 1, characterized in that: The dynamic identification of glass products described in step 2 is specifically implemented according to the following steps: Measuring the degree of match between the predicted box and the true box is called consistent matching metric, and the formula is as follows: ， in, represents the spatial prior, is the classification confidence, Represent the predicted box and the true box respectively, It is the intersection of the predicted box and the true box; Target score The calculation formula is: ， is the target score of the nth positive sample, Represents the maximum value of IoU among all positive samples, Represents the positive sample used to measure the degree of alignment between the predicted box and the true box. Similarly for the maximum value; Classification loss calculation formula: ， are the coefficients of the loss function, represents the predicted value of the nth sample, represents the activation function; Coordinate loss calculation formula: ， is the intersection-over-union ratio of the nth positive sample, the target score is the weight coefficient, is the time when the bounding box of the nth sample is formed; The total loss is the weighted sum of classification loss, coordinate loss, and confidence loss, and the formula is as follows: ， in, is the weighting coefficient, is the confidence loss.

3. The end-to-end autonomous motion decision-making method based on deep reinforcement learning according to claim 1, characterized in that: Step 4 describes the design of three reward functions: the target point reward, the obstacle avoidance reward, and the speed reward. The specific implementation steps are as follows: Set the target point reward function to: ， in It is the reward for reaching the destination. is the maximum distance between the robot and the target location; Set the obstacle avoidance reward function to: ， in is the obstacle avoidance reward function, is the distance from the robot to the obstacle; Set the speed reward function to: ， Among all the reward functions designed above, when the three reward functions conflict, different reward coefficients are designed to determine the priority of the reward. The final overall system reward function is : ， in, is the absolute distance from the robot to the target location, It's time. , , They are reaching the target point, obstacle avoidance and speed reward functions, , , is the coefficient of each reward function, satisfying .

4. The end-to-end autonomous motion decision-making method based on deep reinforcement learning according to claim 1, characterized in that: The calculation of the time difference target TD-Target and the loss function described in step 5 is implemented in the following steps: TD-Target is calculated by the following formula: ， Set the loss function of the HER-DQN algorithm: ， Substitute into TD-Target and simplify: ， in is the discount factor, It's the state. It's a reward. It's behavior. are the coefficients of the function, It is the state of the next moment. It is the behavior of the next moment. are the parameters of the Q network.

Images