Mobile robot dynamic obstacle avoidance control method and system based on reinforcement learning

Abstract

The invention discloses a mobile robot dynamic obstacle avoidance control method and system based on reinforcement learning, and relates to the technical field of robot control. The problems that a traditional mobile robot obstacle avoidance method lags in decision and is poor in adaptability in a dynamic complex environment, and an existing intelligent obstacle avoidance scheme is difficult to deal with environment sudden changes are solved. According to the method, efficient obstacle avoidance of the mobile robot in a dynamic environment is realized through reinforcement learning, a precise state space is constructed through multi-sensor fusion and feature extraction, and a reinforcement learning network can autonomously learn and optimize a strategy to adapt to complex dynamic changes; the reward function design gives consideration to obstacle avoidance, target guidance and action rationality, the robustness and efficiency of a strategy are improved, the obstacle avoidance capability and task execution efficiency of the mobile robot in a complex scene are effectively improved, meanwhile, through an obstacle avoidance quality analysis and correction mechanism, real-time monitoring and dynamic adjustment of the obstacle avoidance process are achieved, and the method has the advantages of being high in practicability and easy to popularize. And the reliability and the safety of obstacle avoidance are further improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method and system for dynamic obstacle avoidance control of mobile robots based on reinforcement learning. Background Technology

[0002] Traditional obstacle avoidance methods for mobile robots, such as rule-based methods or traditional model-based methods, often exhibit limitations when dealing with dynamic, complex, and uncertain environments. For example, Chinese patent CN115129070B discloses an intelligent obstacle avoidance system and method for warehouse robots under the Internet of Things (IoT), including a data acquisition and control module, a planar model building module, and an obstacle avoidance analysis module. The data acquisition and control module is used for data acquisition and system control; the planar model building module is used to build planar models of the warehouse robot and goods; and the obstacle avoidance analysis module is used to perform planar analysis of the goods and obtain obstacle avoidance paths. This invention uses a distance measurement unit to detect the distance to objects around the warehouse robot and an image acquisition unit to acquire image data of the goods. The image data is then analyzed, taking into full account the shape of the goods and their placement on the warehouse robot, thus preventing accidents caused by collisions between goods during obstacle avoidance and making the obstacle avoidance of the warehouse robot more intelligent.

[0003] While the aforementioned patents can take into account the shape and arrangement of goods to optimize obstacle avoidance strategies to some extent, they still belong to the traditional mode of pre-set processes. When faced with dynamic scenarios such as personnel movement and temporary equipment movement in the warehouse environment, their obstacle avoidance decisions lack real-time performance and flexibility, making it difficult to meet the needs of efficient and safe operation of mobile robots in modern complex environments. This can easily lead to collisions or mission failures, limiting the application scope and efficiency of robots in complex environments. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic obstacle avoidance control method and system for mobile robots based on reinforcement learning. By using multi-source sensor fusion perception, deep reinforcement learning network optimization decision-making, dynamic reward feedback mechanism, and intelligent obstacle avoidance quality analysis and correction strategy, it can achieve accurate perception, efficient decision-making, and adaptive adjustment of the dynamic environment, effectively improving the obstacle avoidance ability and task execution efficiency of mobile robots in complex scenarios, thereby solving the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: Reinforcement learning-based dynamic obstacle avoidance control methods for mobile robots include: Environmental perception and state construction: Based on the sensor group carried by the mobile robot, environmental information is collected in real time and preprocessed to extract environmental information features. Combined with the mobile robot's own parameters, the current state space of the mobile robot is obtained. Action decision and strategy generation: Construct a reinforcement learning network model, input the current state space of the mobile robot into the trained reinforcement learning network model, generate a mapping strategy from the state space to the action space, and output the optimal action in the current state; Action execution and environment interaction: The optimal action in the current state is matched with the instruction database to obtain the corresponding action instruction, and then sent to the drive control system of the mobile robot. After the mobile robot executes the action, it again uses the sensor group to perceive changes in environmental information in real time and obtain a new state space.

[0006] Furthermore, environmental perception and state construction specifically include: Environmental information acquisition: The surrounding environment is scanned using LiDAR to acquire 3D point cloud data and determine the distance, orientation, and approximate shape of obstacles; images of the surrounding environment are captured using a visual sensor to determine obstacle characteristics; and nearby obstacles are detected in real time using an ultrasonic sensor. Data preprocessing: Denoising the point cloud data acquired by the LiDAR; Image processing of the surrounding environment images acquired by the visual sensor; Smoothing the ultrasonic sensor data. Environmental information feature extraction: The preprocessed LiDAR point cloud data is divided into different obstacle clusters, and the geometric features of each obstacle cluster are extracted; the semantic features of obstacles in the surrounding environment image are extracted; and the relative speed of the obstacles is calculated by combining the distance data of the ultrasonic sensor. State space construction: The extracted environmental information features are fused with the mobile robot's own parameters to construct a multi-dimensional state vector, which serves as the current state space of the mobile robot.

[0007] Furthermore, in the state space, the mobile robot's own parameters include position, velocity, acceleration, and orientation angle, and the environmental information includes the position, velocity, and shape of obstacles and the position of the target point. The lidar point cloud data is converted into polar coordinates with the mobile robot as the origin and integrated into a multi-dimensional state vector.

[0008] Furthermore, the reinforcement learning network model includes an input layer, a hidden layer, and an output layer. The input layer receives state space information, and the number of neurons in the input layer is determined according to the dimension of the state space. The hidden layer extracts and abstracts features from the state space information of the input layer through a multi-layer neural network. The output layer outputs the motion space of the mobile robot, which includes different combinations of speed and steering commands.

[0009] Furthermore, action decision-making and strategy generation also include: Model training optimization: An experience pool is established using an experience replay mechanism to store experience samples generated by the interaction between the mobile robot and the environment. This determines the actions, rewards, and next state information of the mobile robot in different states. The stored experience samples are randomly selected from the experience pool for training. At the same time, the training of the reinforcement learning network model is optimized based on the distance gradient and angle gradient between the mobile robot and the target point. Reward Feedback: Determine the reward function based on the running results of the mobile robot after performing actions, and calculate the reward value of the mobile robot's action execution result based on the reward function; Policy optimization: Feedback the reward value into the reinforcement learning network model, update the parameters of the reinforcement learning network model, optimize the mapping policy from the state space to the action space, and dynamically adjust the decision logic.

[0010] Furthermore, the reward function specifically includes: Obstacle avoidance reward: The reward value is determined based on the distance relationship between the mobile robot and the obstacle. If the mobile robot successfully avoids the obstacle and the distance between the mobile robot and the obstacle is greater than the safety threshold, a positive reward is given; if the mobile robot collides with the obstacle or the distance is too small and less than the safety threshold, a negative reward is given; if the mobile robot does not detect the obstacle or maintains a safe distance from the obstacle and does not encounter any special circumstances during the movement, a zero reward is given. Goal-oriented reward: The reward value is determined based on the change in distance between the mobile robot and the target point. When the mobile robot moves towards the target point and the distance between the current position and the target point decreases compared to the previous moment, a positive reward is given. The greater the decrease in distance, the higher the reward value. When the mobile robot successfully reaches the target point, a preset additional positive reward is given. Action Reasonableness Reward: Whether to give a reward value is determined based on the range of change of the mobile robot's action. If the range of change of the parameters of the mobile robot's action is within the preset reasonable range, a positive reward is output; if the range of change of the action parameters exceeds the preset reasonable range, a negative reward is output. Furthermore, action execution and interaction with the environment also include: Based on the current operating status of the mobile robot after it performs an action, the task analysis is performed to determine the current obstacle avoidance task execution information of the mobile robot. Based on the current obstacle avoidance task execution information of the mobile robot, an obstacle avoidance quality analysis is performed to obtain obstacle avoidance quality evaluation factors. Then, obstacle avoidance quality assessment data is calculated based on the obstacle avoidance quality evaluation factors to obtain obstacle avoidance quality analysis data. Based on the preset obstacle avoidance standards, the obstacle avoidance quality analysis data is standardized and analyzed to determine whether the obstacle avoidance quality analysis data meets the obstacle avoidance standards, and the obstacle avoidance quality analysis results are obtained. When the obstacle avoidance quality analysis result is that the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard, the cause analysis is carried out based on the obstacle avoidance quality analysis data to determine the reasons why the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard and to obtain the reasons for the non-standard obstacle avoidance. Based on the reasons for non-standard obstacle avoidance and the results of the mobile robot's task analysis, it is determined whether the reasons for non-standard obstacle avoidance can be corrected during the operation of the mobile robot, and the results of the cause analysis are obtained.

[0011] Furthermore, when the cause analysis results indicate that the non-standard obstacle avoidance can be corrected during the operation of the mobile robot, obstacle avoidance correction information is generated based on the cause of the non-standard obstacle avoidance. At the same time, corresponding operation instructions are generated based on the current task progress deviation of the mobile robot and the real-time monitored environmental information. The corresponding operation instructions are then corrected and adjusted using the correction information of the cause of the non-standard obstacle avoidance. The mobile robot is guided to adjust its obstacle avoidance strategy according to the corrected and adjusted operation instructions.

[0012] Furthermore, when the cause analysis results indicate that the obstacle avoidance is not standardized and cannot be corrected during the operation of the mobile robot, corresponding operation instructions are generated based on the current task progress deviation of the mobile robot and the real-time monitored environmental information. At the same time, the difference between the obstacle avoidance quality analysis data and the obstacle avoidance standard is determined, and a hazard warning message is generated based on the difference data. While driving the mobile robot to adjust the obstacle avoidance strategy according to the operation instructions, a warning prompt is issued based on the hazard warning message. This invention provides another technical solution: a mobile robot dynamic obstacle avoidance control system based on reinforcement learning, comprising: The environmental perception module is connected to the sensor group carried by the mobile robot. It is used to collect environmental information in real time, preprocess the collected environmental information and extract features to construct the current state space of the mobile robot. The action decision module is used to receive state space information from the deep reinforcement learning network and calculate and generate action commands for the mobile robot through the trained reinforcement learning network model. The motion execution module is connected to the drive control system of the mobile robot. It is used to receive motion commands and drive the mobile robot to perform corresponding speed and turning actions. The reward feedback optimization module is used to calculate the reward value based on the running results of the mobile robot after performing actions, and feed the reward value back to the reinforcement learning network model. At the same time, it updates the network parameters and optimizes the obstacle avoidance strategy of the mobile robot.

[0013] Compared with the prior art, the beneficial effects of the present invention are: The present invention proposes a dynamic obstacle avoidance control method for mobile robots based on reinforcement learning. This method achieves efficient obstacle avoidance for mobile robots in dynamic environments through reinforcement learning. Multi-sensor fusion and feature extraction construct an accurate state space, and the reinforcement learning network can autonomously learn and optimize strategies to adapt to complex dynamic changes. The reward function design takes into account obstacle avoidance, goal orientation, and action rationality, improving the robustness and efficiency of the strategy. At the same time, through obstacle avoidance quality analysis and correction mechanisms, real-time monitoring and dynamic adjustment of the obstacle avoidance process are realized, further improving the reliability and safety of obstacle avoidance. Attached Figure Description

[0014] Figure 1 This is a flowchart of the reinforcement learning-based dynamic obstacle avoidance control method for mobile robots according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] To address the technical challenges of traditional mobile robot obstacle avoidance methods, such as decision-making lag and poor adaptability in dynamic and complex environments, and the inability of existing intelligent obstacle avoidance solutions to cope with sudden environmental changes, please refer to [link to relevant documentation]. Figure 1 This embodiment provides the following technical solution: Reinforcement learning-based dynamic obstacle avoidance control methods for mobile robots include: Environmental perception and state construction: Based on the sensor group carried by the mobile robot, including lidar, vision sensor and ultrasonic sensor, environmental information is collected in real time and preprocessed to extract environmental information features. Combined with the mobile robot's own parameters, the current state space of the mobile robot is obtained. The state space is used to characterize the mobile robot's operating status in the environment. Action decision and strategy generation: Construct a reinforcement learning network model, input the current state space of the mobile robot into the trained reinforcement learning network model, generate a mapping strategy from the state space to the action space, and output the optimal action in the current state; In this embodiment, the reinforcement learning network model includes an input layer, a hidden layer, and an output layer. The input layer receives state space information, and the number of neurons in the input layer is determined according to the dimension of the state space. For example, if the state vector contains 50 features, the input layer has 50 neurons to receive the preprocessed and fused multidimensional state vector. The hidden layer extracts and abstracts features from the state space information of the input layer through a multi-layer neural network, using a multi-layer fully connected layer or convolutional layer structure, such as setting 3 hidden layers with 128, 64, and 32 neurons in each layer, respectively. The input information is nonlinearly transformed by an activation function (such as the ReLU function). The output layer outputs the motion space of the mobile robot, which includes different combinations of speed and turning commands. The number of neurons in the output layer corresponds to the dimension of the motion space. If the motion space contains 10 different combinations of speed and turning commands, the output layer has 10 neurons to output the value or probability corresponding to each action. Action execution and environment interaction: The optimal action in the current state is matched with the instruction database to obtain the corresponding action instruction, and then sent to the drive control system of the mobile robot to drive the mobile robot to perform the corresponding speed and turning actions. After the mobile robot performs the action, it once again perceives changes in environmental information in real time based on the sensor group to obtain a new state space.

[0017] In this embodiment, a high-dimensional state space is constructed through multi-sensor fusion and feature extraction, enabling the robot to perceive complex environments in real time. Deep features are extracted based on a reinforcement learning network model to generate optimal action strategies, improving the efficiency and stability of obstacle avoidance strategy generation in complex dynamic environments. The action execution scheme achieves rapid matching and execution of strategies to instructions. The introduction of a quality analysis and correction mechanism enables real-time evaluation of obstacle avoidance effects and dynamic adjustment of strategies, avoiding collisions while optimizing paths. This allows the robot to respond quickly to environmental changes and achieves efficient obstacle avoidance in complex dynamic environments.

[0018] In this embodiment, environmental perception and state construction specifically include: Environmental information acquisition: The system uses LiDAR to scan the surrounding environment and acquire 3D point cloud data. The point cloud data can accurately determine the distance, orientation, and approximate shape of obstacles. It also uses visual sensors to capture images of the surrounding environment and determine the type, color, texture, and other features of obstacles. Ultrasonic sensors are used to detect nearby obstacles in real time and provide high-precision distance measurement data. Data preprocessing: The point cloud data acquired by the LiDAR is denoised using algorithms such as statistical filtering and voxel filtering to remove outliers caused by environmental interference or equipment errors; the surrounding environment images acquired by the vision sensor are processed by grayscale conversion, noise reduction, and contrast enhancement to improve image quality and facilitate subsequent feature extraction; the ultrasonic sensor data is smoothed using methods such as moving average filtering to eliminate data jitter and ensure the accuracy of distance measurement. Environmental information feature extraction: The preprocessed LiDAR point cloud data is divided into different obstacle clusters, and the geometric features such as centroid coordinates, size, and bounding box orientation of each obstacle cluster are extracted; the semantic features of obstacles in the surrounding environment image are extracted, such as determining whether the obstacle is a pedestrian, vehicle, or fixed object; the relative velocity of the obstacle is calculated by combining the distance data from the ultrasonic sensor, which is determined by the ratio of the distance change between adjacent time moments to the time interval. State Space Construction: The extracted environmental information features are fused with the mobile robot's own parameters to construct a multi-dimensional state vector, which serves as the current state space of the mobile robot. For example, the state vector can be represented as [distance to obstacle 1, orientation of obstacle 1, velocity of obstacle 1, distance to obstacle 2, ..., mobile robot position X, mobile robot position Y, mobile robot velocity, mobile robot heading angle]. This state vector comprehensively represents the mobile robot's operating status in the environment. In the state space, the mobile robot's own parameters include position, velocity, acceleration, and heading angle, while the environmental information includes the position, velocity, and shape of obstacles, as well as the position of the target point. The LiDAR point cloud data is converted into polar coordinates with the mobile robot as the origin and integrated into the multi-dimensional state vector.

[0019] In this embodiment, three sensors work together to achieve comprehensive and multi-dimensional information collection of the surrounding environment. Adaptive preprocessing and feature extraction methods are adopted for the data characteristics of different sensors to ensure data quality and feature validity. The lidar point cloud data is converted into polar coordinate representation and integrated into the state vector, which fits the robot's local perception characteristics. This enables the constructed multi-dimensional state vector to accurately and comprehensively represent the robot's operating status, providing a solid foundation for subsequent obstacle avoidance decisions. This significantly improves the robot's perception and adaptability in dynamic and complex environments, thereby enhancing the accuracy and efficiency of environmental information processing.

[0020] In this embodiment, action decision-making and strategy generation further include: Model training optimization: An experience pool is established using an experience replay mechanism to store experience samples generated by the interaction between the mobile robot and the environment. Each experience sample consists of a quadruple of "state, action, reward, and next state" to determine the mobile robot's actions, rewards, and next state information in different states. Experience samples stored in the experience pool are randomly selected for training to break the correlation between data and improve the training stability and convergence speed of the deep reinforcement learning network. Simultaneously, to accelerate algorithm convergence, a guidance mechanism based on distance and angle gradients is introduced. The training of the reinforcement learning network model is optimized based on the distance and angle gradients between the mobile robot and the target point, enabling the mobile robot to move towards the target point faster and avoid getting trapped in local optima. Reward Feedback: A reward function is determined based on the results of the mobile robot's actions. The reward value for each action is calculated using this function, specifically including: Obstacle avoidance reward: The reward value is determined based on the distance relationship between the mobile robot and the obstacle. If the mobile robot successfully avoids the obstacle and the distance to the obstacle is greater than the safety threshold, a positive reward is given to encourage the mobile robot to continue to adopt effective obstacle avoidance actions. If the mobile robot collides with the obstacle or the distance is too small to the safety threshold, a negative reward is given to prompt the mobile robot to avoid such dangerous behaviors. If the mobile robot does not detect the obstacle or maintains a safe distance from the obstacle and does not encounter any special circumstances during the operation, a zero reward is given to maintain its stable operation in a non-threatening environment. Goal-oriented reward: The reward value is determined based on the change in distance between the mobile robot and the target point. When the mobile robot moves towards the target point and the distance between the current position and the target point decreases compared to the previous moment, a positive reward is given. The greater the decrease in distance, the higher the reward value. When the mobile robot successfully reaches the target point, a preset additional positive reward is given to guide the mobile robot to complete the task as soon as possible. Action Reasonableness Reward: Whether to give a reward value is determined based on the range of change of the mobile robot's action. If the range of change of parameters such as the speed and turning angle of the mobile robot's action is within the preset reasonable range, a positive reward is output; if the range of change of action parameters exceeds the preset reasonable range, a negative reward is output. In this embodiment, reasonable and smooth actions performed by the mobile robot are rewarded to prevent the mobile robot from making frequent sudden turns or stops, thereby ensuring the stability and efficiency of the mobile robot's movement. Strategy optimization: The reward value is fed back into the reinforcement learning network model, and the parameters of the reinforcement learning network model are updated through the backpropagation algorithm. The mapping strategy from the state space to the action space is optimized, so that the mobile robot can continuously improve its obstacle avoidance ability and decision accuracy in the subsequent obstacle avoidance process. The decision logic is dynamically adjusted to continuously improve obstacle avoidance ability and decision accuracy, and gradually learn the optimal obstacle avoidance strategy in complex dynamic environments.

[0021] In this embodiment, a guidance mechanism based on distance and angle gradients is used to specifically optimize the training direction, enabling the robot to move quickly toward the target point and effectively avoid local optima. The reward function is finely designed from three dimensions: obstacle avoidance, target orientation, and action rationality. Multi-target collaborative guidance of robot behavior can more comprehensively shape the robot's decision-making logic. By feeding back the reward value to update the network parameters, dynamic optimization of the strategy is achieved, enabling the robot to continuously evolve its obstacle avoidance strategy in complex dynamic environments.

[0022] In this embodiment, the action execution and interaction with the environment also include: Based on the current operating status of the mobile robot after it performs an action, the task analysis is performed to determine the current obstacle avoidance task execution information of the mobile robot, including whether it has deviated from the preset obstacle avoidance path, the actual distance change from the obstacle, and whether it is moving towards the target point. Based on the current obstacle avoidance task execution information of the mobile robot, an obstacle avoidance quality analysis is performed to obtain obstacle avoidance quality evaluation factors, such as obstacle avoidance path length, speed stability during obstacle avoidance, and minimum safe distance from obstacles. Obstacle avoidance quality assessment data is then calculated based on the obstacle avoidance quality evaluation factors to obtain obstacle avoidance quality analysis data. Based on the preset obstacle avoidance standards, the obstacle avoidance quality analysis data is standardized and analyzed to determine whether the obstacle avoidance quality analysis data meets the obstacle avoidance standards, and the obstacle avoidance quality analysis results are obtained. When the obstacle avoidance quality analysis result is that the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard, the cause analysis is carried out based on the obstacle avoidance quality analysis data to determine the reasons why the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard, and the reasons for the non-standard obstacle avoidance are obtained, such as sensor data error, reinforcement learning model strategy deviation, and actuator response delay. Based on the reasons for non-standard obstacle avoidance and the results of the mobile robot's task analysis, it is determined whether the reasons for non-standard obstacle avoidance can be corrected during the operation of the mobile robot, and the results of the cause analysis are obtained. When the cause analysis results indicate that the non-standard obstacle avoidance can be corrected during the operation of the mobile robot, obstacle avoidance correction information is generated based on the cause of the non-standard obstacle avoidance. At the same time, corresponding operation instructions are generated based on the current task progress deviation of the mobile robot (such as the difference between the remaining time to the target point and the estimated time) and the real-time monitored environmental information. The corresponding operation instructions are corrected and adjusted using the correction information of the cause of the non-standard obstacle avoidance. The mobile robot is guided to adjust its obstacle avoidance strategy according to the corrected and adjusted operation instructions. When the cause analysis results indicate that the obstacle avoidance is not standardized and cannot be corrected during the operation of the mobile robot, corresponding operation instructions are generated based on the current task progress deviation of the mobile robot and the real-time monitored environmental information. At the same time, the difference between the obstacle avoidance quality analysis data and the obstacle avoidance standard is determined, and a hazard warning message is generated based on the difference data. While driving the mobile robot to adjust the obstacle avoidance strategy according to the operation instructions, a warning prompt is issued based on the hazard warning message.

[0023] In this embodiment, a multi-dimensional quality evaluation system, including path length and speed stability, is established through real-time task analysis and obstacle avoidance quality quantification assessment. Compared with the traditional method that relies solely on collision detection, this system enables refined analysis of obstacle avoidance performance. The introduction of a standardized analysis and cause localization mechanism accurately identifies non-standard causes such as sensor errors and strategy deviations, overcoming the limitations of ambiguous fault localization in existing technologies. Dynamic correction and early warning strategies are implemented for different scenarios, adjusting operational instructions in real time for correctable problems and generating hazard warnings for uncorrectable problems. This enables the robot to have self-optimizing capabilities in complex environments, significantly enhancing its adaptability and reliability in dynamic scenarios.

[0024] To better implement a reinforcement learning-based dynamic obstacle avoidance control method for mobile robots, this invention provides a reinforcement learning-based dynamic obstacle avoidance control system for mobile robots, comprising: The environmental perception module is connected to the sensor group carried by the mobile robot. It is used to collect environmental information in real time, preprocess the collected environmental information and extract features to construct the current state space of the mobile robot. The action decision module is used to receive state space information from the deep reinforcement learning network and calculate and generate action commands for the mobile robot through the trained reinforcement learning network model. The motion execution module is connected to the drive control system of the mobile robot. It is used to receive motion commands and drive the mobile robot to perform corresponding speed and turning actions. The reward feedback optimization module is used to calculate the reward value based on the running results of the mobile robot after performing actions, and feed the reward value back to the reinforcement learning network model. At the same time, it updates the network parameters and optimizes the obstacle avoidance strategy of the mobile robot.

[0025] In this embodiment, the environmental perception module collects and preprocesses environmental information in real time to construct a state space, the action decision module generates action instructions with the help of a deep reinforcement learning network, the action execution module drives the robot to complete speed and turning actions, and the reward feedback optimization module calculates the reward value and updates the network parameters based on the running results. It has the advantages of strong environmental adaptability, high dynamic obstacle avoidance efficiency, outstanding decision autonomy, wide application scenarios, and can significantly reduce the cost of manual programming. It can effectively improve the obstacle avoidance reliability and operation intelligence level of mobile robots in complex dynamic environments.

[0026] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A dynamic obstacle avoidance control method for mobile robots based on reinforcement learning, characterized in that, include: Environmental perception and state construction: Based on the sensor group carried by the mobile robot, environmental information is collected in real time and preprocessed to extract environmental information features. Combined with the mobile robot's own parameters, the current state space of the mobile robot is obtained. Action decision and strategy generation: Construct a reinforcement learning network model, input the current state space of the mobile robot into the trained reinforcement learning network model, generate a mapping strategy from the state space to the action space, and output the optimal action in the current state; Action execution and environment interaction: The optimal action in the current state is matched with the instruction database to obtain the corresponding action instruction, and then sent to the drive control system of the mobile robot. After the mobile robot executes the action, it again uses the sensor group to perceive changes in environmental information in real time and obtain a new state space.

2. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 1, characterized in that, Environmental perception and state construction specifically include: Environmental information acquisition: The surrounding environment is scanned using LiDAR to acquire 3D point cloud data and determine the distance, orientation, and approximate shape of obstacles; images of the surrounding environment are captured using a visual sensor to determine obstacle characteristics; and nearby obstacles are detected in real time using an ultrasonic sensor. Data preprocessing: Denoising the point cloud data acquired by the LiDAR; Image processing of the surrounding environment images acquired by the visual sensor; Smoothing the ultrasonic sensor data. Environmental information feature extraction: The preprocessed LiDAR point cloud data is divided into different obstacle clusters, and the geometric features of each obstacle cluster are extracted; the semantic features of obstacles in the surrounding environment image are extracted; and the relative speed of the obstacles is calculated by combining the distance data of the ultrasonic sensor. State space construction: The extracted environmental information features are fused with the mobile robot's own parameters to construct a multi-dimensional state vector, which serves as the current state space of the mobile robot.

3. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 2, characterized in that, In the state space, the mobile robot's own parameters include position, velocity, acceleration, and orientation angle. Environmental information includes the position, velocity, and shape of obstacles, as well as the position of the target point. The LiDAR point cloud data is converted into polar coordinates with the mobile robot as the origin and integrated into a multi-dimensional state vector.

4. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 3, characterized in that, The reinforcement learning network model includes an input layer, a hidden layer, and an output layer. The input layer receives state space information, and the number of neurons in the input layer is determined according to the dimension of the state space. The hidden layer extracts and abstracts features from the state space information of the input layer through a multi-layer neural network. The output layer outputs the motion space of the mobile robot, which includes different combinations of speed and steering commands.

5. The reinforcement learning-based dynamic obstacle avoidance control method for mobile robots as described in claim 4, characterized in that, Action decision-making and strategy generation also include: Model training optimization: An experience pool is established using an experience replay mechanism to store experience samples generated by the interaction between the mobile robot and the environment. This determines the actions, rewards, and next state information of the mobile robot in different states. The stored experience samples are randomly selected from the experience pool for training. At the same time, the training of the reinforcement learning network model is optimized based on the distance gradient and angle gradient between the mobile robot and the target point. Reward Feedback: Determine the reward function based on the running results of the mobile robot after performing actions, and calculate the reward value of the mobile robot's action execution result based on the reward function; Policy optimization: Feedback the reward value into the reinforcement learning network model, update the parameters of the reinforcement learning network model, optimize the mapping policy from the state space to the action space, and dynamically adjust the decision logic.

6. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 5, characterized in that, The reward function specifically includes: Obstacle avoidance reward: The reward value is determined based on the distance relationship between the mobile robot and the obstacle. If the mobile robot successfully avoids the obstacle and the distance between the mobile robot and the obstacle is greater than the safety threshold, a positive reward is given; if the mobile robot collides with the obstacle or the distance is too small and less than the safety threshold, a negative reward is given; if the mobile robot does not detect the obstacle or maintains a safe distance from the obstacle and does not encounter any special circumstances during the movement, a zero reward is given. Goal-oriented reward: The reward value is determined based on the change in distance between the mobile robot and the target point. When the mobile robot moves towards the target point and the distance between the current position and the target point decreases compared to the previous moment, a positive reward is given. The greater the decrease in distance, the higher the reward value. When the mobile robot successfully reaches the target point, a preset additional positive reward is given. Action Reasonableness Reward: Whether to give a reward value is determined based on the range of change of the mobile robot's action. If the range of change of the parameters of the mobile robot's action is within the preset reasonable range, a positive reward is output; if the range of change of the action parameters exceeds the preset reasonable range, a negative reward is output.

7. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 1, characterized in that, Action execution and interaction with the environment also include: Based on the current operating status of the mobile robot after it performs an action, the task analysis is performed to determine the current obstacle avoidance task execution information of the mobile robot. Based on the current obstacle avoidance task execution information of the mobile robot, an obstacle avoidance quality analysis is performed to obtain obstacle avoidance quality evaluation factors. Then, obstacle avoidance quality assessment data is calculated based on the obstacle avoidance quality evaluation factors to obtain obstacle avoidance quality analysis data. Based on the preset obstacle avoidance standards, the obstacle avoidance quality analysis data is standardized and analyzed to determine whether the obstacle avoidance quality analysis data meets the obstacle avoidance standards, and the obstacle avoidance quality analysis results are obtained. When the obstacle avoidance quality analysis result is that the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard, the cause analysis is carried out based on the obstacle avoidance quality analysis data to determine the reasons why the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard and to obtain the reasons for the non-standard obstacle avoidance. Based on the reasons for non-standard obstacle avoidance and the results of the mobile robot's task analysis, it is determined whether the reasons for non-standard obstacle avoidance can be corrected during the operation of the mobile robot, and the results of the cause analysis are obtained.

8. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 7, characterized in that, When the cause analysis results indicate that the non-standard obstacle avoidance can be corrected during the operation of the mobile robot, obstacle avoidance correction information is generated based on the cause of the non-standard obstacle avoidance. At the same time, corresponding operation instructions are generated based on the current task progress deviation of the mobile robot and the real-time monitored environmental information. The corresponding operation instructions are then corrected and adjusted using the correction information of the non-standard obstacle avoidance, and the mobile robot is guided to adjust its obstacle avoidance strategy according to the corrected and adjusted operation instructions.

9. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 7, characterized in that, When the cause analysis results indicate that the obstacle avoidance is not standardized and cannot be corrected during the operation of the mobile robot, corresponding operation instructions are generated based on the current task progress deviation of the mobile robot and the real-time monitored environmental information. At the same time, the difference between the obstacle avoidance quality analysis data and the obstacle avoidance standard is determined, and a hazard warning message is generated based on the difference data. While driving the mobile robot to adjust the obstacle avoidance strategy according to the operation instructions, a warning prompt is issued based on the hazard warning message.

10. A mobile robot dynamic obstacle avoidance control system based on reinforcement learning, employing the mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 1, characterized in that... include: The environmental perception module is connected to the sensor group carried by the mobile robot. It is used to collect environmental information in real time, preprocess the collected environmental information and extract features to construct the current state space of the mobile robot. The action decision module is used to receive state space information from the deep reinforcement learning network and calculate and generate action commands for the mobile robot through the trained reinforcement learning network model. The motion execution module is connected to the drive control system of the mobile robot. It is used to receive motion commands and drive the mobile robot to perform corresponding speed and turning actions. The reward feedback optimization module is used to calculate the reward value based on the running results of the mobile robot after performing actions, and feed the reward value back to the reinforcement learning network model. At the same time, it updates the network parameters and optimizes the obstacle avoidance strategy of the mobile robot.

Images