Reinforcement learning-based mobile robot dynamic obstacle avoidance control method and system

Abstract

The application discloses a mobile robot dynamic obstacle avoidance control method and system based on reinforcement learning, and relates to the technical field of robot control. In order to solve the problems that the traditional mobile robot obstacle avoidance method has decision lag and poor adaptability in a dynamic complex environment, and the existing intelligent obstacle avoidance scheme is difficult to cope with environmental mutations, the application realizes efficient obstacle avoidance of the mobile robot in the dynamic environment through reinforcement learning, a precise state space is constructed through multi-sensor fusion and feature extraction, the reinforcement learning network can autonomously learn and optimize the strategy, and adapt to complex dynamic changes; the reward function design takes into account obstacle avoidance, target orientation and action rationality, improves the robustness and efficiency of the strategy, effectively improves the obstacle avoidance ability and task execution efficiency of the mobile robot in a complex scene, meanwhile, through obstacle avoidance quality analysis and correction mechanism, real-time monitoring and dynamic adjustment of the obstacle avoidance process are realized, and the reliability and safety of the obstacle avoidance are further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a mobile robot dynamic obstacle avoidance control method and system based on reinforcement learning. BACKGROUND

[0002] Traditional mobile robot obstacle avoidance methods, such as rule-based methods or traditional model-based methods, often show limitations when dealing with dynamic, complex and uncertain environments. For example, a Chinese patent with publication number CN115129070B discloses a warehouse robot intelligent obstacle avoidance system and method under the Internet of Things, which includes a collection control module, a plane model establishment module and an obstacle avoidance analysis module. The collection control module is used for data collection and system control. The plane model establishment module is used to establish a plane model of the warehouse robot and the goods. The obstacle avoidance analysis module is used for plane analysis of the goods to obtain an obstacle avoidance path. The present application detects the distance of the objects around the warehouse robot through the distance measurement unit, and obtains the image data of the goods through the image acquisition unit, and analyzes the image data, fully considers the shape of the goods and the placement form of the goods on the warehouse robot, so that when the warehouse robot avoids obstacles, accidental collisions between goods are avoided, and the obstacle avoidance of the warehouse robot is more intelligent.

[0003] Although the above-mentioned patent can consider the shape and placement form of the goods to some extent to optimize the obstacle avoidance strategy, it still belongs to the traditional mode of pre-set process. When facing dynamic scenes such as personnel shuttling and temporary movement of equipment in the warehouse environment, the real-time and flexibility of its obstacle avoidance decision are insufficient, which is difficult to meet the demand of efficient and safe operation of mobile robots in modern complex environment, and is easy to cause collision or task failure, which limits the application range and efficiency of robots in complex environment. SUMMARY

[0004] The purpose of the present application is to provide a mobile robot dynamic obstacle avoidance control method and system based on reinforcement learning, which realizes accurate perception, efficient decision and adaptive adjustment of dynamic environment through multi-source sensor fusion perception, deep reinforcement learning network optimization decision, dynamic reward feedback mechanism and intelligent obstacle avoidance quality analysis and correction strategy, effectively improves the obstacle avoidance ability and task execution efficiency of mobile robots in complex scenes, to solve the problems raised in the above background technology.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] The mobile robot dynamic obstacle avoidance control method based on reinforcement learning comprises:

[0007] Environment perception and state construction: Real-time collection of environmental information based on the sensor group carried by the mobile robot and preprocessing, extraction of environmental information features, combination with the parameters of the mobile robot, and acquisition of the current state space of the mobile robot;

[0008] Action decision and strategy generation: Construct a reinforcement learning network model, input the current state space of the mobile robot into the trained reinforcement learning network model, generate a mapping strategy from the state space to the action space, and output the optimal action under the current state;

[0009] Action execution and environment interaction: Match the optimal action under the current state with the instruction database to obtain the corresponding action instruction, and send it to the drive control system of the mobile robot. After the mobile robot performs the action, real-time perception of environmental information changes based on the sensor group again to obtain a new state space.

[0010] Further, the environment perception and state construction specifically include:

[0011] Environmental information collection: Use laser radar to scan the surrounding environment and obtain three-dimensional point cloud data to determine the distance, orientation and approximate shape of the obstacle. Based on the vision sensor, take pictures of the surrounding environment to determine the characteristics of the obstacle. Real-time detection of close-range obstacles is performed through ultrasonic sensors;

[0012] Data preprocessing: Denoising of point cloud data obtained by laser radar; image processing of surrounding environment images collected by vision sensors; smoothing of ultrasonic sensor data;

[0013] Environment information feature extraction: Divide the preprocessed laser radar point cloud data into different obstacle clusters and extract the geometric features of each obstacle cluster; extract the semantic features of the obstacles in the surrounding environment images; calculate the relative speed of the obstacles in combination with the distance data of the ultrasonic sensor;

[0014] State space construction: Fuse the extracted environmental information features with the parameters of the mobile robot to construct a multi-dimensional state vector as the current state space of the mobile robot.

[0015] Further, in the state space, the parameters of the mobile robot include position, velocity, acceleration, orientation angle, environmental information includes position, velocity, shape of the obstacle and position of the target point, and the laser radar point cloud data is converted into polar coordinates with the mobile robot as the origin and integrated into the multi-dimensional state vector.

[0016] Further, the reinforcement learning network model comprises an input layer, a hidden layer and an output layer, the input layer receives state space information, wherein the number of input layer neurons is determined according to the state space dimension; the hidden layer extracts and abstracts the state space information of the input layer through a multi-layer neural network; and the output layer outputs the action space of the mobile robot, the action space comprising different speed and steering instruction combinations.

[0017] Further, the action decision and policy generation further comprises:

[0018] Model training optimization: an experience pool is established by using an experience replay mechanism, for storing experience samples generated by the interaction between the mobile robot and the environment, determining the action, reward and next state information of the mobile robot in different states, and randomly extracting the stored experience samples from the experience pool for training; at the same time, the training of the reinforcement learning network model is optimized according to the distance gradient and angle gradient between the mobile robot and the target point;

[0019] Reward feedback: determining a reward function according to the running result of the action executed by the mobile robot, and calculating the result reward value of the action execution of the mobile robot based on the reward function;

[0020] Policy optimization: feeding back the reward value to the reinforcement learning network model, updating the parameters of the reinforcement learning network model, optimizing the mapping strategy from the state space to the action space, and dynamically adjusting the decision logic.

[0021] Further, the reward function specifically comprises:

[0022] Obstacle avoidance reward: determining the reward value based on the distance relationship between the mobile robot and the obstacle, giving a positive reward if the mobile robot successfully avoids the obstacle and the distance between the mobile robot and the obstacle is greater than a safety threshold; giving a negative reward if the mobile robot collides with the obstacle or the distance is less than the safety threshold; and giving a zero reward if the mobile robot does not detect the obstacle or keeps a safe distance from the obstacle during driving and does not encounter special circumstances;

[0023] Target-oriented reward: determining the reward value according to the distance change between the mobile robot and the target point, giving a positive reward when the mobile robot moves towards the target point and the distance between the current position and the target point decreases compared with the previous time, and the larger the distance reduction, the higher the reward value; and giving a preset additional positive reward when the mobile robot successfully reaches the target point;

[0024] Action rationality reward: determining whether to give a reward value according to the action change amplitude of the mobile robot, outputting a positive reward if the parameter change amplitude of the action executed by the mobile robot is within a preset reasonable range; and outputting a negative reward if the action parameter change amplitude exceeds the preset reasonable range;

[0025] Further, the action execution interacts with the environment, and further includes:

[0026] According to the current running state of the mobile robot after executing the action, task analysis is performed to determine the current obstacle avoidance task execution information of the mobile robot;

[0027] According to the current obstacle avoidance task execution information of the mobile robot, obstacle avoidance quality analysis is performed to obtain obstacle avoidance quality evaluation factors, and obstacle avoidance quality evaluation data is calculated according to the obstacle avoidance quality evaluation factors, and obstacle avoidance quality analysis data is obtained;

[0028] The obstacle avoidance quality analysis data is combined with the preset obstacle avoidance standard to perform standardized analysis, and it is determined whether the obstacle avoidance quality analysis data meets the obstacle avoidance standard, and an obstacle avoidance quality analysis result is obtained;

[0029] When the obstacle avoidance quality analysis result is that the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard, the reason analysis is performed according to the obstacle avoidance quality analysis data, and the reason causing the obstacle avoidance quality analysis data not to meet the obstacle avoidance standard is determined, and an obstacle avoidance non-standard reason is obtained;

[0030] Based on the obstacle avoidance non-standard reason and the task analysis result of the mobile robot, it is determined whether the obstacle avoidance non-standard reason can be corrected in the running process of the mobile robot, and a reason analysis result is obtained.

[0031] Further, when the reason analysis result is that the obstacle avoidance non-standard reason can be corrected in the running process of the mobile robot, the obstacle avoidance correction information is generated based on the obstacle avoidance non-standard reason, and the corresponding operation instruction is generated according to the current task progress deviation of the mobile robot and the real-time monitored environment information, and the obstacle avoidance non-standard reason is corrected and adjusted by using the correction information, and the mobile robot is guided to adjust the obstacle avoidance strategy according to the corrected and adjusted operation instruction.

[0032] Further, when the reason analysis result is that the obstacle avoidance non-standard reason cannot be corrected in the running process of the mobile robot, the corresponding operation instruction is generated according to the current task progress deviation of the mobile robot and the real-time monitored environment information, and the difference data between the obstacle avoidance quality analysis data and the obstacle avoidance standard is determined, the danger warning information is generated according to the difference data, and the mobile robot is driven to adjust the obstacle avoidance strategy according to the operation instruction, and the danger warning information is used for warning prompt.

[0033] The present application provides another technical scheme, a mobile robot dynamic obstacle avoidance control system based on reinforcement learning, comprising:

[0034] The environment perception module is connected with the sensor group carried by the mobile robot, and is used for real-time collection of environment information, pre-processing and feature extraction of the collected environment information, and construction of the current state space of the mobile robot;

[0035] An action decision module is configured to receive state space information according to the deep reinforcement learning network, and generate an action instruction of the mobile robot through calculation of the trained reinforcement learning network model.

[0036] An action execution module is connected with a drive control system of the mobile robot, configured to receive the action instruction and drive the mobile robot to perform corresponding speed and steering actions.

[0037] A reward feedback optimization module is configured to calculate a reward value according to a running result after the mobile robot performs the action, and feed back the reward value to the reinforcement learning network model, while updating network parameters to optimize the obstacle avoidance strategy of the mobile robot.

[0038] Compared with the prior art, the method has the following beneficial effects:

[0039] The method for dynamic obstacle avoidance control of the mobile robot based on reinforcement learning realizes efficient obstacle avoidance of the mobile robot in a dynamic environment through reinforcement learning, constructs an accurate state space through multi-sensor fusion and feature extraction, and enables the reinforcement learning network to autonomously learn and optimize the strategy to adapt to complex dynamic changes. The reward function design takes into account obstacle avoidance, target orientation and action rationality, improves the robustness and efficiency of the strategy, and simultaneously realizes real-time monitoring and dynamic adjustment of the obstacle avoidance process through an obstacle avoidance quality analysis and correction mechanism, further improving the reliability and safety of obstacle avoidance. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The method for dynamic obstacle avoidance control of the mobile robot based on reinforcement learning. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0042] To solve the technical problems of the traditional mobile robot obstacle avoidance method in a dynamic and complex environment, such as decision lag, poor adaptability, and difficulty of existing intelligent obstacle avoidance schemes to cope with environmental mutations, please refer to Figure 1 The technical solutions of the present embodiment are as follows:

[0043] The method for dynamic obstacle avoidance control of the mobile robot based on reinforcement learning comprises:

[0044] Environment perception and state construction: based on the sensor group carried by the mobile robot, including laser radar, visual sensor, ultrasonic sensor, real-time collection of environmental information and preprocessing, extraction of environmental information features, combination of mobile robot parameters, acquisition of the current state space of the mobile robot, and the state space is used to represent the running situation of the mobile robot in the environment;

[0045] Action decision and strategy generation: a reinforcement learning network model is constructed, the current state space of the mobile robot is input into the trained reinforcement learning network model, a mapping strategy from the state space to the action space is generated, and the optimal action under the current state is output;

[0046] In this embodiment, the reinforcement learning network model includes an input layer, a hidden layer and an output layer, the input layer receives state space information, wherein the number of input layer neurons is determined according to the state space dimension, for example, if the state vector contains 50 features, 50 neurons are set in the input layer to receive the multi-dimensional state vector after preprocessing and fusion; the hidden layer extracts and abstracts the state space information of the input layer through a multi-layer neural network, adopts a multi-layer fully connected layer or a convolutional layer structure, such as setting 3 layers of hidden layer, the number of neurons in each layer is 128, 64 and 32 respectively, and the input information is nonlinearly transformed through an activation function (such as ReLU function); the output layer outputs the action space of the mobile robot, the action space includes different combinations of speed and steering instructions, the number of output layer neurons corresponds to the dimension of the action space, if the action space contains 10 different combinations of speed and steering instructions, 10 neurons are set in the output layer to output the value or probability corresponding to each action;

[0047] Action execution and environment interaction: the optimal action under the current state is matched with the instruction database to obtain the corresponding action instruction, which is sent to the drive control system of the mobile robot to drive the mobile robot to perform the corresponding speed and steering action, and the mobile robot performs the action again, and then the sensor group is used to real-time perceive the change of the environmental information and obtain a new state space.

[0048] In this embodiment, a high-dimensional state space is constructed through multi-sensor fusion and feature extraction, so that the robot can real-time perceive the complex environment, deep features are extracted based on the reinforcement learning network model to generate an optimal action strategy, and the efficiency and stability of the obstacle avoidance strategy generation in a complex dynamic environment are improved, the action execution scheme realizes the rapid matching and execution of the strategy to the instruction, the quality analysis and correction mechanism is introduced, the obstacle avoidance effect can be real-time evaluated and the strategy can be dynamically adjusted, collision can be avoided and the path can be optimized, so that the robot can quickly respond to the environmental changes, and efficient obstacle avoidance in a complex dynamic environment is realized.

[0049] In this embodiment, the environment perception and state construction specifically includes:

[0050] Environment information collection: laser radar is used to scan the surrounding environment and obtain three-dimensional point cloud data, which can accurately determine the distance, direction and approximate shape of obstacles; visual sensors are used to capture images of the surrounding environment to determine the type, color and texture of obstacles; ultrasonic sensors are used to detect obstacles at close range in real time and provide high-precision distance measurement data.

[0051] Data preprocessing: the point cloud data obtained by the laser radar is denoised, statistical filtering, voxel filtering and other algorithms are used to remove abnormal points caused by environmental interference or equipment errors; the surrounding environment images collected by the visual sensor are grayed, denoised, and contrast-enhanced for image processing to improve image quality and facilitate subsequent feature extraction; the ultrasonic sensor data is smoothed by sliding average filtering and other methods to eliminate data jitter and ensure the accuracy of distance measurement;

[0052] Environment information feature extraction: the preprocessed laser radar point cloud data is divided into different obstacle clusters, and the geometric features of each obstacle cluster such as centroid coordinates, size, bounding box direction are extracted; the semantic features of obstacles in the surrounding environment images are extracted, such as determining whether the obstacle is a pedestrian, vehicle or fixed object; combined with the distance data of the ultrasonic sensor, the relative speed of the obstacle is calculated by the ratio of the distance change and the time interval between adjacent time points;

[0053] State space construction: the extracted environmental information features and mobile robot parameters are fused to construct a multi-dimensional state vector as the current state space of the mobile robot; for example, the state vector can be represented as [obstacle 1 distance, obstacle 1 direction, obstacle 1 speed, obstacle 2 distance, …, mobile robot position X, mobile robot position Y, mobile robot speed, mobile robot heading angle], which fully represents the running situation of the mobile robot in the environment, the state space includes the position, speed, acceleration, orientation angle of the mobile robot itself, the position, speed, shape of the obstacle and the position of the target point, and the laser radar point cloud data is converted into polar coordinates with the mobile robot as the origin and integrated into the multi-dimensional state vector.

[0054] In this embodiment, through the cooperative work of the three sensors, all-around and multi-dimensional information collection of the surrounding environment is realized, the adaptive preprocessing and feature extraction method is adopted according to the data characteristics of different sensors to ensure the data quality and feature effectiveness, the laser radar point cloud data is converted into polar coordinate representation and integrated into the state vector, which is consistent with the local perception characteristics of the robot, so that the multi-dimensional state vector constructed can accurately and comprehensively represent the robot running situation, providing a solid foundation for subsequent obstacle avoidance decision-making, significantly improving the perception ability and adaptability of the robot in dynamic and complex environment, and improving the accuracy and efficiency of environmental information processing.

[0055] In this embodiment, the action decision and strategy generation further include:

[0056] Model training optimization: an experience pool is established by using an experience replay mechanism, which is used to store experience samples generated by the mobile robot interacting with the environment, each experience sample is composed of a four-tuple of "state, action, reward, next state", the action, reward and next state information of the mobile robot in different states are determined, and the stored experience samples are randomly extracted from the experience pool for training, so as to break the correlation between data, improve the training stability and convergence speed of the deep reinforcement learning network; at the same time, in order to accelerate the convergence of the algorithm, a guidance mechanism based on distance gradient and angle gradient is introduced, the distance gradient and angle gradient between the mobile robot and the target point are used to optimize the training of the reinforcement learning network model, so that the mobile robot can move faster towards the target point, and the local optimal solution is avoided;

[0057] Reward feedback: the reward function is determined according to the running result of the mobile robot after executing the action, and the result reward value of the action execution of the mobile robot is calculated based on the reward function, which specifically includes:

[0058] Obstacle avoidance reward: the reward value is determined based on the distance relationship between the mobile robot and the obstacle, if the mobile robot successfully avoids the obstacle and the distance between the mobile robot and the obstacle is greater than a safety threshold, a positive reward is given to encourage the mobile robot to continue to adopt effective obstacle avoidance actions; if the mobile robot collides with the obstacle or the distance is less than the safety threshold, a negative reward is given to prompt the mobile robot to avoid such dangerous behavior; if the mobile robot does not detect the obstacle or keeps a safe distance from the obstacle during driving and does not encounter special circumstances, a zero reward is given to maintain its stable running state in a non-threatening environment;

[0059] Target-oriented reward: the reward value is determined according to the distance change between the mobile robot and the target point, when the mobile robot moves towards the target point and the distance between the current position and the target point is smaller than that at the previous moment, a positive reward is given, the larger the distance reduction, the higher the reward value; when the mobile robot successfully reaches the target point, a preset additional positive reward is given to guide the mobile robot to complete the task as soon as possible;

[0060] Action rationality reward: judging whether to give a reward value according to the change range of the action of the mobile robot, if the change range of the speed, turning angle and other parameters of the action of the mobile robot is within the preset reasonable range, a positive reward is output; if the change range of the action parameters exceeds the preset reasonable range, a negative reward is output;

[0061] In the embodiment, a certain reward is given to the reasonable and smooth action performed by the mobile robot, so as to avoid unreasonable actions such as frequent sharp turning and sudden stopping of the mobile robot, so as to ensure the stability and efficiency of the movement of the mobile robot;

[0062] Strategy optimization: the reward value is fed back to the reinforcement learning network model, the reinforcement learning network model parameters are updated through the back propagation algorithm, the mapping strategy from the state space to the action space is optimized, the obstacle avoidance ability and decision accuracy of the mobile robot are continuously improved in the subsequent obstacle avoidance process, the decision logic is dynamically adjusted, the obstacle avoidance ability and decision accuracy are continuously improved, and the optimal obstacle avoidance strategy in the complex dynamic environment is gradually learned.

[0063] In the embodiment, the guidance mechanism based on distance and angle gradient is used to optimize the training direction, so that the robot can quickly move towards the target point, effectively avoid local optimal dilemma, the reward function is designed in three dimensions of obstacle avoidance, target orientation and action rationality, and the robot behavior is guided by multiple targets, so that the robot decision logic can be more comprehensively shaped. The reward value is fed back to update the network parameters, the strategy is dynamically optimized, and the robot continuously evolves the obstacle avoidance strategy in the complex dynamic environment.

[0064] In the embodiment, the action execution and the environment interaction also include:

[0065] According to the current running state of the mobile robot after performing the action, task analysis is performed to determine the current obstacle avoidance task execution information of the mobile robot, including whether to deviate from the preset obstacle avoidance path, the actual distance change from the obstacle, whether to move towards the target point, etc.

[0066] According to the current obstacle avoidance task execution information of the mobile robot, obstacle avoidance quality analysis is performed to obtain obstacle avoidance quality evaluation factors such as obstacle avoidance path length, speed stability in the obstacle avoidance process, and minimum safety distance from the obstacle, and to calculate obstacle avoidance quality evaluation data according to the obstacle avoidance quality evaluation factors to obtain obstacle avoidance quality analysis data;

[0067] The obstacle avoidance quality analysis data is combined with the preset obstacle avoidance standard for standardized analysis to determine whether the obstacle avoidance quality analysis data meets the obstacle avoidance standard, and the obstacle avoidance quality analysis result is obtained;

[0068] When the obstacle avoidance quality analysis result is that the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard, the reasons for the obstacle avoidance quality analysis data not meeting the obstacle avoidance standard are determined according to the obstacle avoidance quality analysis data, and the obstacle avoidance non-standard reasons such as sensor data error, reinforcement learning model strategy deviation, and actuator response delay are obtained;

[0069] Based on the obstacle avoidance non-standard reasons and the task analysis result of the mobile robot, it is determined whether the obstacle avoidance non-standard reasons can be corrected in the running process of the mobile robot, and a reason analysis judgment result is obtained.

[0070] When the reason analysis judgment result is that the obstacle avoidance non-standard reasons can be corrected in the running process of the mobile robot, obstacle avoidance correction information is generated based on the obstacle avoidance non-standard reasons, and corresponding operation instructions are generated according to the current task progress deviation (such as the difference between the remaining time and the expected time to the target point) of the mobile robot and the real-time monitored environment information. The correction information of the obstacle avoidance non-standard reasons is used to correct and adjust the corresponding operation instructions, and the mobile robot is guided to adjust the obstacle avoidance strategy according to the corrected and adjusted operation instructions.

[0071] When the reason analysis judgment result is that the obstacle avoidance non-standard reasons cannot be corrected in the running process of the mobile robot, corresponding operation instructions are generated according to the current task progress deviation of the mobile robot and the real-time monitored environment information, and the difference data between the obstacle avoidance quality analysis data and the obstacle avoidance standard is determined. The difference data is used to generate a danger warning information, and the mobile robot is driven to adjust the obstacle avoidance strategy according to the operation instructions, and a warning prompt is performed according to the danger warning information.

[0072] In this embodiment, through real-time task analysis and obstacle avoidance quality quantitative evaluation, a multi-dimensional quality evaluation system including path length, speed stability and the like is established, compared with the traditional method relying only on collision detection, the fine analysis of the obstacle avoidance effect is realized, the standardized analysis and reason positioning mechanism are introduced, the non-standard reasons such as sensor error and strategy deviation can be accurately identified, and the limitation of fuzzy fault positioning in the prior art is broken through. The dynamic correction and warning strategy is implemented in different scenes, the operation instructions are adjusted in real time for correctable problems, and danger warning is generated for uncorrectable problems, so that the robot has the strategy self-optimization ability in complex environment, and the adaptability and reliability of the robot in dynamic scene are significantly enhanced.

[0073] In order to better realize the dynamic obstacle avoidance control method of the mobile robot based on reinforcement learning, the present application provides a dynamic obstacle avoidance control system of a mobile robot based on reinforcement learning, comprising:

[0074] An environment perception module is connected with a sensor group carried by the mobile robot, used for collecting environment information in real time, pre-processing and feature extraction of the collected environment information, and constructing a current state space of the mobile robot.

[0075] An action decision module is configured to receive state space information according to the deep reinforcement learning network, and to generate action instructions of the mobile robot by calculation of the trained reinforcement learning network model;

[0076] An action execution module is connected with the drive control system of the mobile robot, and is configured to receive the action instructions and drive the mobile robot to perform corresponding speed and steering actions;

[0077] A reward feedback optimization module is configured to calculate a reward value according to a running result after the mobile robot performs the actions, and to feed back the reward value to the reinforcement learning network model, while updating the network parameters to optimize the obstacle avoidance strategy of the mobile robot.

[0078] In the embodiment, the environment perception module is used to collect and pre-process the environmental information in real time to construct the state space, the action decision module is used to generate the action instructions by means of the deep reinforcement learning network, the action execution module is used to drive the robot to complete the speed and steering actions, and the reward feedback optimization module is used to calculate the reward value according to the running result and update the network parameters, so that the mobile robot has the advantages of strong environmental adaptability, high dynamic obstacle avoidance efficiency, outstanding decision autonomy, wide application scenarios, and significant reduction of artificial programming cost, and can effectively improve the obstacle avoidance reliability and the running intelligent level of the mobile robot in the complex dynamic environment.

[0079] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions and the inventive concept of the present application within the disclosed technical scope, which should be covered within the protection scope of the present application.

Claims

1. A dynamic obstacle avoidance control method for mobile robots based on reinforcement learning, characterized in that, include: Environmental perception and state construction: Based on the sensor group carried by the mobile robot, environmental information is collected in real time and preprocessed to extract environmental information features. Combined with the mobile robot's own parameters, the current state space of the mobile robot is obtained. Action decision and strategy generation: Construct a reinforcement learning network model, input the current state space of the mobile robot into the trained reinforcement learning network model, generate a mapping strategy from the state space to the action space, and output the optimal action in the current state; Action execution and environment interaction: The optimal action in the current state is matched with the instruction database to obtain the corresponding action instruction, and then sent to the drive control system of the mobile robot. After the mobile robot executes the action, it again uses the sensor group to perceive changes in environmental information in real time and obtain a new state space. Action execution and interaction with the environment also include: Based on the current operating status of the mobile robot after it performs an action, the task analysis is performed to determine the current obstacle avoidance task execution information of the mobile robot. Based on the current obstacle avoidance task execution information of the mobile robot, an obstacle avoidance quality analysis is performed to obtain obstacle avoidance quality evaluation factors. Then, obstacle avoidance quality assessment data is calculated based on the obstacle avoidance quality evaluation factors to obtain obstacle avoidance quality analysis data. Based on the preset obstacle avoidance standards, the obstacle avoidance quality analysis data is standardized and analyzed to determine whether the obstacle avoidance quality analysis data meets the obstacle avoidance standards, and the obstacle avoidance quality analysis results are obtained. When the obstacle avoidance quality analysis result is that the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard, the cause analysis is carried out based on the obstacle avoidance quality analysis data to determine the reasons why the obstacle avoidance quality analysis data does not meet the obstacle avoidance standard and to obtain the reasons for the non-standard obstacle avoidance. Based on the reasons for non-standard obstacle avoidance and the results of the mobile robot's task analysis, it is determined whether the reasons for non-standard obstacle avoidance can be corrected during the operation of the mobile robot, and the results of the cause analysis are obtained.

2. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 1, characterized in that, Environmental perception and state construction specifically include: Environmental information acquisition: LiDAR is used to scan the surrounding environment, acquire 3D point cloud data, and determine the distance, orientation, and approximate shape of obstacles; images of the surrounding environment are captured based on visual sensors to determine obstacle characteristics; and ultrasonic sensors are used to detect nearby obstacles in real time. Data preprocessing: Denoising the point cloud data acquired by the LiDAR; Image processing of the surrounding environment images acquired by the visual sensor; Smoothing the ultrasonic sensor data. Environmental information feature extraction: The preprocessed LiDAR point cloud data is divided into different obstacle clusters, and the geometric features of each obstacle cluster are extracted; the semantic features of obstacles in the surrounding environment image are extracted; and the relative speed of the obstacles is calculated by combining the distance data of the ultrasonic sensor. State space construction: The extracted environmental information features are fused with the mobile robot's own parameters to construct a multi-dimensional state vector, which serves as the current state space of the mobile robot.

3. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 2, characterized in that, In the state space, the mobile robot's own parameters include position, velocity, acceleration, and orientation angle. Environmental information includes the position, velocity, and shape of obstacles, as well as the position of the target point. The LiDAR point cloud data is converted into polar coordinates with the mobile robot as the origin and integrated into a multi-dimensional state vector.

4. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 3, characterized in that, The reinforcement learning network model includes an input layer, a hidden layer, and an output layer. The input layer receives state space information, and the number of neurons in the input layer is determined according to the dimension of the state space. The hidden layer extracts and abstracts features from the state space information of the input layer through a multi-layer neural network. The output layer outputs the motion space of the mobile robot, which includes different combinations of speed and steering commands.

5. The reinforcement learning-based dynamic obstacle avoidance control method for mobile robots as described in claim 4, characterized in that, Action decision-making and strategy generation also include: Model training optimization: An experience pool is established using an experience replay mechanism to store experience samples generated by the interaction between the mobile robot and the environment. This determines the actions, rewards, and next state information of the mobile robot in different states. The stored experience samples are randomly selected from the experience pool for training. At the same time, the training of the reinforcement learning network model is optimized based on the distance gradient and angle gradient between the mobile robot and the target point. Reward Feedback: Determine the reward function based on the running results of the mobile robot after performing actions, and calculate the reward value of the mobile robot's action execution result based on the reward function; Policy optimization: Feedback the reward value into the reinforcement learning network model, update the parameters of the reinforcement learning network model, optimize the mapping policy from the state space to the action space, and dynamically adjust the decision logic.

6. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 5, characterized in that, The reward function specifically includes: Obstacle avoidance reward: The reward value is determined based on the distance relationship between the mobile robot and the obstacle. If the mobile robot successfully avoids the obstacle and the distance between the mobile robot and the obstacle is greater than the safety threshold, a positive reward is given; if the mobile robot collides with the obstacle or the distance is too small and less than the safety threshold, a negative reward is given; if the mobile robot does not detect the obstacle or maintains a safe distance from the obstacle and does not encounter any special circumstances during the movement, a zero reward is given. Goal-oriented reward: The reward value is determined based on the change in distance between the mobile robot and the target point. When the mobile robot moves towards the target point and the distance between the current position and the target point decreases compared to the previous moment, a positive reward is given. The greater the decrease in distance, the higher the reward value. When the mobile robot successfully reaches the target point, a preset additional positive reward is given. Action Reasonableness Reward: Whether to give a reward value is determined based on the range of change of the mobile robot's action. If the range of change of the parameters of the mobile robot's action is within the preset reasonable range, a positive reward is output; if the range of change of the action parameters exceeds the preset reasonable range, a negative reward is output.

7. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 1, characterized in that, When the cause analysis results indicate that the non-standard obstacle avoidance can be corrected during the operation of the mobile robot, obstacle avoidance correction information is generated based on the cause of the non-standard obstacle avoidance. At the same time, corresponding operation instructions are generated based on the current task progress deviation of the mobile robot and the real-time monitored environmental information. The corresponding operation instructions are then corrected and adjusted using the correction information of the non-standard obstacle avoidance, and the mobile robot is guided to adjust its obstacle avoidance strategy according to the corrected and adjusted operation instructions.

8. The mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 1, characterized in that, When the cause analysis results indicate that the obstacle avoidance is not standardized and cannot be corrected during the operation of the mobile robot, corresponding operation instructions are generated based on the current task progress deviation of the mobile robot and the real-time monitored environmental information. At the same time, the difference between the obstacle avoidance quality analysis data and the obstacle avoidance standard is determined, and a hazard warning message is generated based on the difference data. While driving the mobile robot to adjust the obstacle avoidance strategy according to the operation instructions, a warning prompt is issued based on the hazard warning message.

9. A mobile robot dynamic obstacle avoidance control system based on reinforcement learning, employing the mobile robot dynamic obstacle avoidance control method based on reinforcement learning as described in claim 1, characterized in that... include: The environmental perception module is connected to the sensor group carried by the mobile robot. It is used to collect environmental information in real time, preprocess the collected environmental information and extract features to construct the current state space of the mobile robot. The action decision module is used to receive state space information from the deep reinforcement learning network and calculate and generate action commands for the mobile robot through the trained reinforcement learning network model. The motion execution module is connected to the drive control system of the mobile robot. It is used to receive motion commands and drive the mobile robot to perform corresponding speed and turning actions. The reward feedback optimization module is used to calculate the reward value based on the running results of the mobile robot after performing actions, and feed the reward value back to the reinforcement learning network model. At the same time, it updates the network parameters and optimizes the obstacle avoidance strategy of the mobile robot.

Images