Reinforcement learning based adaptive planner parameter tuning method and apparatus

Abstract

The application discloses a kind of based on reinforcement learning's adaptive planner parameter tuning method and device, method includes: at low frequency control frequency, in parameter adjustment network, according to the input environmental data, using reinforcement learning algorithm to adaptively adjust the parameters of local planner;At medium frequency control frequency, in local planner, according to the adjusted parameter, the planning trajectory of robot travel is generated and the feedforward speed is generated according to the planning trajectory;At high frequency control frequency, in controller, according to the feedforward speed and the feedback speed calculated by error compensator, generate final control instruction;Parameter adjustment network and controller are alternately trained and tested iteratively, and according to the test result, the parameters of local planner and the final control instruction generated by controller are continuously optimized.The application can improve system robustness and navigation efficiency, effectively reduce tracking error and enhance obstacle avoidance ability, can complete more accurate, efficient and reliable robot navigation task in complex environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot control, and particularly relates to a self-adaptive planner parameter tuning method and device based on reinforcement learning. BACKGROUND

[0002] Trajectory planning in highly constrained environments is a crucial research direction in the field of mobile robots. Traditional trajectory planning methods, such as elastic band method and dynamic window method, rely on precise environment modeling and controller parameter setting, and can generate feasible trajectories and achieve autonomous navigation of robots in known environments. However, these traditional methods often face limitations in dynamic or complex environments, and are difficult to cope with changes in the environment or unstructured obstacle distribution. In recent years, with the development of artificial intelligence technology, especially learning-based methods such as imitation learning (IL) and reinforcement learning (RL), have gradually been applied to trajectory planning.

[0003] Imitation learning is a method of training models through expert demonstration data, which has shown outstanding performance in the field of autonomous driving. By imitating the operations of experts, imitation learning can generate trajectories or control commands. However, due to the lack of large-scale expert data sets in the field of mobile robots, imitation learning methods are not suitable for highly constrained environments. In addition, imitation learning usually adopts an open-loop framework, which relies on the quality of demonstration data and cannot incorporate the actual execution feedback of the robot during the learning process, so the adaptability of the system is poor when the robot encounters new situations.

[0004] Unlike imitation learning, reinforcement learning is a closed-loop learning method that adjusts the strategy through real-time interaction feedback between the robot and the environment, avoiding the dependence on demonstration data, and can better adapt to complex environments with higher performance limits. Reinforcement learning not only considers the dynamic changes of the environment, but also can guide the robot to explore effectively by setting a reward function, so as to learn more robust navigation strategies. However, there are some challenges in the direct application of reinforcement learning in trajectory planning. For example, reinforcement learning often requires a large number of exploration steps, resulting in low sample efficiency, especially when directly learning speed control strategies. In order to overcome this problem, in recent years, an adaptive planner parameter tuning system based on reinforcement learning has been developed, which can significantly improve the efficiency of trajectory generation and execution by automatically adjusting the key parameters (such as maximum speed, inflation radius, etc.) of traditional planning algorithms, while reducing unnecessary exploration processes.

[0005] The adaptive planner parameter tuning method has the advantage that it can adjust the planner parameters in real time according to environmental changes, without relying on fixed parameters or manual debugging. For example, the Adaptive Planner Parameter Learning (APPL) system proposes an adaptive planner parameter tuning framework based on reinforcement learning, which dynamically selects the optimal parameters through a reinforcement learning strategy and adapts to different navigation scenarios.

[0006] However, although existing adaptive planner parameter tuning methods have made significant progress, they often only focus on optimizing the tuning level and ignore the limitations of the control level. In a closed-loop system, the tuning network relies on the assumption that the controller can accurately execute the planned trajectory, but due to the pure feedforward nature of the control system, there is often a deviation between the executed trajectory and the planned trajectory, resulting in tracking errors. These errors introduce noise and reduce the learning effect of the tuning network. Therefore, simply optimizing the tuning level cannot completely solve the problem, and the overall performance of the system is still limited by the control level. SUMMARY

[0007] In view of the above, the purpose of the present application is to provide an adaptive planner parameter tuning method and device based on reinforcement learning, which designs a hierarchical architecture based on low-frequency parameter tuning, medium-frequency planning, and high-frequency control for reinforcement learning adaptive planner parameter tuning. Not only does it focus on optimizing the tuning level, but it also considers the limitations of the control level, further reducing tracking errors, and combining a reinforcement learning error compensator to correct tracking errors in real time, effectively handling dynamic obstacle avoidance requirements in complex environments. Through an iterative training framework with alternating training, the overall system performance is continuously improved, enabling more accurate, efficient, and reliable mobile robot navigation task execution in complex environments.

[0008] To achieve the above-mentioned application purposes, the technical solutions provided by the present application are as follows:

[0009] In a first aspect, the present application provides an adaptive planner parameter tuning method based on reinforcement learning, comprising the following steps:

[0010] At a low control frequency, the parameter adjustment network is used to adaptively adjust the parameters of the local planner based on the input environmental data using a reinforcement learning algorithm;

[0011] At a medium control frequency, the local planner generates a planned trajectory for the robot to travel based on the adjusted parameters and generates a feedforward speed based on the planned trajectory;

[0012] At a high control frequency, the controller generates a final control command based on the feedforward speed and the feedback speed calculated by the error compensator;

[0013] The parameter adjustment network and the controller are iteratively alternately trained and tested, and the parameters of the local planner and the final control instructions generated by the controller are continuously optimized according to the test results.

[0014] Preferably, the parameter adjustment network adaptively adjusts the parameters of the local planner according to the input environmental data at a low-frequency control frequency, including:

[0015] The parameter adjustment network uses a reinforcement learning algorithm to train the local planner according to the input environmental data at a low-frequency control frequency, and the parameters of the selected local planner are evaluated and adjusted for effect after the mobile robot runs a certain trajectory, so as to adaptively adjust the parameters of the local planner.

[0016] Preferably, the state space in the reinforcement learning algorithm uses the environmental data obtained by the laser radar as input, and embeds the environmental data into a scene vector through a variational autoencoder network; the action space in the reinforcement learning algorithm includes the adjustable parameters in the local planner, including the maximum linear velocity, the maximum angular velocity, the obstacle weight and the inflation radius; the reward function in the reinforcement learning algorithm includes a target progress reward and a collision penalty, and the robot obtains the target progress reward if it moves towards the local target point, and obtains the collision penalty if it collides.

[0017] Preferably, the parameter adjustment network adaptively adjusts the parameters of the local planner according to the input environmental data at a low-frequency control frequency, including:

[0018] The local planner uses a TEB algorithm to generate a planning trajectory according to the adjusted parameters at a medium-frequency control frequency, and the local planner generates a feedforward speed according to the target pose of the planning trajectory and the current speed of the robot, and guides the robot to travel according to the planning trajectory through the feedforward speed.

[0019] Preferably, the parameter adjustment network adaptively adjusts the parameters of the local planner according to the input environmental data at a low-frequency control frequency, including:

[0020] The controller receives the feedforward speed generated by the local planner at a high-frequency control frequency, and simultaneously obtains the state of the robot through sensor feedback, and the error compensator generates a feedback speed according to the environmental data, the state of the robot, the key points of the planning trajectory and the time step, and the feedback speed is used to correct the feedforward speed of the robot in real time, and the controller generates a final control instruction according to the feedforward speed and the feedback speed and sends it to the robot for execution, so that the robot can accurately follow the planning trajectory.

[0021] Preferably, the error compensator is based on reinforcement learning, and the error compensator parameters are optimized by defining a reward function to minimize the tracking error between the actual position of the robot and the planned trajectory, and the feedback velocity is generated by using the optimized error compensator.

[0022] Preferably, the parameter adjustment network and the controller are iteratively alternately trained and tested, and the parameters of the local planner and the final control command generated by the controller are continuously optimized according to the test results, including:

[0023] In the alternately training process, first, the controller is fixed in a round, and the parameter adjustment network is trained based on reinforcement learning to select the optimal local planner parameters, and then the parameter adjustment network is fixed in the next round, and the controller is trained based on reinforcement learning to obtain the optimal final control command.

[0024] After each round of training is completed, the robot trajectory tracking test is performed in a simulation environment or an actual environment, and the parameters of the local planner in the parameter adjustment network are further optimized and the controller is adjusted according to the test results.

[0025] Through multiple rounds of iterative alternately training and robot trajectory tracking test, the tracking error between the actual position of the robot and the planned trajectory is gradually reduced, and the parameters of the local planner and the final control command generated by the controller are continuously optimized.

[0026] In the second aspect, an adaptive planner parameter optimization device based on reinforcement learning is provided, which is implemented by using the adaptive planner parameter optimization method based on reinforcement learning, and includes a low-frequency parameter optimization module, a medium-frequency planning module, a high-frequency control module, and an iterative training module.

[0027] The low-frequency parameter optimization module is configured to adaptively adjust the parameters of the local planner in the parameter adjustment network based on the input environment data using a reinforcement learning algorithm at a low control frequency.

[0028] The medium-frequency planning module is configured to generate a planned trajectory for the robot to travel in the local planner according to the adjusted parameters and generate a feedforward velocity according to the planned trajectory at a medium control frequency.

[0029] The high-frequency control module is configured to generate a final control command in the controller according to the feedforward velocity and the feedback velocity calculated by the error compensator at a high control frequency.

[0030] The iterative training module is configured to iteratively alternately train and test the parameter adjustment network and the controller, and continuously optimize the parameters of the local planner and the final control command generated by the controller according to the test results.

[0031] In a third aspect, an electronic device is provided and includes a memory for storing a computer program and one or more processors for implementing the above-mentioned adaptive planner parameter tuning method based on reinforcement learning when executing the computer program.

[0032] In a fourth aspect, a computer-readable storage medium is provided and has a computer program stored thereon, which, when executed by a computer, implements the above-mentioned adaptive planner parameter tuning method based on reinforcement learning.

[0033] Compared with the prior art, the present application has at least the following beneficial effects:

[0034] (1) Improving the robustness and navigation efficiency of the system: the present application designs a hierarchical architecture based on low-frequency parameter tuning, medium-frequency planning and high-frequency control, so that the parameter tuning of the high layer and the control strategy of the low layer can be optimized cooperatively, improving the stability and robustness of the control system.

[0035] (2) Effectively reducing tracking error: the present application introduces an error compensation mechanism based on an error compensator to calculate the feedback speed, and continuously tunes the parameters of the local planner and the final control command generated by the controller, which can further reduce the trajectory tracking error and ensure the obstacle avoidance ability.

[0036] (3) Enhancing the obstacle avoidance ability of the system: the present application combines the feedback of laser radar with the automatic parameter tuning mechanism of reinforcement learning in complex environments, which can more flexibly avoid dynamic obstacles, thereby realizing more accurate, efficient and reliable navigation task execution. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0038] Figure 1 is a flowchart of the adaptive planner parameter tuning method based on reinforcement learning provided by the embodiments of the present application;

[0039] Figure 2 is a schematic diagram of the hierarchical framework provided by the embodiments of the present application;

[0040] Figure 3 is a schematic diagram of the iterative execution of alternating training provided by the embodiments of the present application;

[0041] Figure 4is a structural schematic diagram of the adaptive planner parameter tuning device based on reinforcement learning provided by the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0043] The inventive concept of the present application is that, in order to solve the problem that the adaptive planner parameter tuning method based on reinforcement learning in the prior art only focuses on optimizing the tuning level and ignores the control layer restriction, the embodiment of the present application provides an adaptive planner parameter tuning method and device based on reinforcement learning, which adaptively adjusts the parameters of the local planner to adapt to different environments at a low control frequency through reinforcement learning, generates a planning trajectory according to the adjusted parameters and provides a feedforward speed for the controller at a medium control frequency, and the controller combines the feedforward speed and a feedback speed to calculate and generate a final control instruction to reduce the trajectory tracking error at a high control frequency. Through the hierarchical design, not only is the parameter tuning of the high layer optimized, but also the robustness of the bottom layer controller is enhanced, and the overall performance of the whole system is improved, so that the robot control system can better adapt to changes in a complex environment. In addition, by introducing an error compensation mechanism based on reinforcement learning, the tracking error caused by the nonlinear dynamic characteristics of the system is further reduced, and the success rate and efficiency of the navigation task are improved.

[0044] Figure 1 is a flowchart of the adaptive planner parameter tuning method based on reinforcement learning provided by the embodiment of the present application. As shown in Figure 1 The embodiment provides an adaptive planner parameter tuning method based on reinforcement learning, which comprises the following steps:

[0045] S1, at a low control frequency, the parameters of the local planner are adaptively adjusted according to the input environment data in the parameter adjustment network through a reinforcement learning algorithm.

[0046] In the embodiment, the control frequency refers to the frequency of executing the control cycle in the control system, i.e. the number of control cycles executed per second, which is expressed in hertz (Hz). The optional range of the low control frequency is 0.1 Hz-1 Hz, and the control frequency is preferably 0.1 Hz, 0.2 Hz, 0.3 Hz, 0.4 Hz, 0.5 Hz, 0.6 Hz, 0.7 Hz, 0.8 Hz, 0.9 Hz or 1 Hz. Figure 2The layered framework shown employs a reinforcement learning algorithm in its parameter tuning network at a low control frequency of 1 Hz. The local planner is trained based on input environmental data to adaptively adjust its parameters. After each trajectory run by the mobile robot, the selected local planner's parameters are evaluated, and the strategy is updated based on the actual execution results. Through repeated parameter adjustments, the system gradually learns how to dynamically select optimal parameters according to environmental changes, thereby optimizing trajectory generation. Key adjustable parameters include maximum linear velocity (max_vel_x), maximum angular velocity (max_vel_theta), obstacle weight (weight_obstacle), and inflation radius (inflation_radius).

[0047] The reinforcement learning algorithm employs the dual-delay deep deterministic policy gradient algorithm (TD3). In the state space, environmental data acquired by LiDAR is used as input. To effectively compress this high-dimensional LiDAR data, this invention uses a variational autoencoder (VAE) network, embedding it into a scene vector. This low-dimensional representation can capture obstacle distribution and path information in the environment. In the action space, actions consist of a set of adjustable parameters, such as maximum speed, rotation speed, obstacle weights, and expansion radius. These parameters are used to adjust the trajectory generation algorithm of the local planner. During each execution, the system adaptively adjusts these parameters based on the current state of the environment. The reward function guides the optimization of the parameter tuning network, including the target progress reward (R). g ) and collision penalty (R) c If the robot successfully avoids obstacles and reaches the target, it receives a progress reward; if the robot collides, it receives a collision penalty. The formula is as follows:

[0048]

[0049] Where r represents arrival The reward value for reaching the goal, where w represents the weight and d is the weight. t r represents the distance between the robot and the target point at time t. collision This represents the penalty value for a collision.

[0050] S2, under the medium frequency control frequency, generates the robot's planned trajectory based on the adjusted parameters in the local planner and generates the feedforward velocity based on the planned trajectory.

[0051] In this embodiment, the selectable range of the intermediate frequency control frequency is 1Hz-10Hz (excluding 1Hz), as shown in the example below. Figure 2The hierarchical framework shown, under the medium frequency control frequency of 10 Hz, the local planner uses the timed elastic band algorithm (TEB algorithm) according to the adjusted parameters to generate a smooth and executable planning trajectory for the robot in the next few seconds by optimizing the distance and time. Then, the local planner generates a feedforward speed according to the target pose of the planning trajectory and the current speed of the robot, and guides the robot to travel along the planning trajectory through the feedforward speed. In this step, the accuracy and obstacle avoidance ability of the trajectory are ensured by combining the lidar data and the robot state information.

[0052] S3, under the high frequency control frequency, the final control command is generated in the controller according to the feedforward speed and the feedback speed calculated by the error compensator.

[0053] In the embodiment, the optional range of the high frequency control frequency is greater than 10 Hz, as shown in the figure Figure 2 The hierarchical framework shown, under the high frequency control frequency of 50 Hz, the controller receives the feedforward speed generated by the local planner, and at the same time, the robot state (such as position, speed and attitude) is obtained through sensor feedback. The goal of the controller is to minimize the tracking error between the actual position of the robot and the planning trajectory as much as possible. The error compensator generates a feedback speed according to the environment data obtained by the lidar, the robot state, the key points of the planning trajectory and the time step, and the feedback speed is used to correct the feedforward speed of the robot in real time. The controller generates a final control command according to the feedforward speed and the feedback speed and sends it to the robot for execution, so that the robot can accurately follow the planning trajectory and optimize the trajectory tracking performance.

[0054] In order to compensate for the error caused by the nonlinear dynamic characteristics of the system during the execution of the robot, the application designs an error compensator based on reinforcement learning. By defining a reward function to minimize the tracking error between the actual position of the robot and the planning trajectory to optimize the error compensator parameters, the optimized error compensator generates a feedback speed, and the reward function is dynamically adjusted according to the distance between the robot and the trajectory, whether a collision occurs, etc. The compensator is not only used to correct the tracking error so that it can follow the planning trajectory more accurately to ensure control accuracy, but also can adjust the control strategy in real time through the combination of high frequency feedback control mechanism and lidar data, and correct the speed of the robot in real time to deal with dynamic obstacles in complex environments, effectively handle dynamic obstacle avoidance requirements in complex environments, flexibly handle obstacles on the path of the robot, and achieve efficient obstacle avoidance ability.

[0055] S4, iteratively alternate training and testing of the parameter adjustment network and the controller, and continuously optimize the parameters of the local planner and the final control command generated by the controller according to the test results.

[0056] In the embodiment, in the alternating training process, first, in one round, the controller is fixed and the parameter adjustment network is trained based on the TD3 algorithm in reinforcement learning to select the optimal local planner parameter, and then in the next round, the parameter adjustment network is fixed and the controller is trained based on the TD3 algorithm in reinforcement learning to obtain the optimal final control instruction. The TD3 algorithm can effectively solve the bias problem in the continuous control task of the robot and improve the stability and robustness of the system. According to the alternating training mode, multiple rounds of iterative training are performed, so that the high-level parameter tuning network and the low-level controller can complement each other, thereby gradually improving the overall performance of the system.

[0057] After each round of training is completed, the robot trajectory tracking test is performed in the simulation environment or the actual environment, and the local planner in the parameter adjustment network is further parameter tuned and the controller is adjusted according to the test result. Through the alternating training and the robot trajectory tracking test in multiple rounds of iteration, the tracking error between the actual position of the robot and the planned trajectory is gradually reduced, and the parameters of the local planner and the final control instruction generated by the controller are continuously optimized, thereby improving the success rate and completion efficiency of the task.

[0058] In summary, the adaptive planner parameter tuning method based on reinforcement learning provided by the embodiment of the present application can improve the robustness and navigation efficiency of the system. The experimental results show that, compared with the traditional 10Hz frequency control system, the combination of the low-frequency parameter tuning with a frequency of 1Hz and the high-frequency control strategy with a frequency of 50Hz can effectively reduce the trajectory tracking error, specifically, the success rate is improved from 70% to 98%, and the task completion time is reduced from 26.7 seconds to 10.2 seconds. This significant improvement is due to the fine error compensation and adaptive parameter tuning of the reinforcement learning controller. The tracking error can also be effectively reduced. In multiple rounds of simulation experiments, compared with the default planner, the control strategy of the present application can effectively reduce the tracking error between the actual trajectory of the robot and the expected trajectory. Through the introduction of the reinforcement learning compensation mechanism, the tracking error is reduced by an average of 30% to 50% in different scenarios. The system can also enhance the obstacle avoidance ability, so that the robot can run smoothly in a more crowded and complex environment without frequent human intervention, reducing the dependence on expert knowledge and trial and error experiments. This adaptability greatly improves the performance of the robot navigation system in complex environments.

[0059] Based on the same inventive concept, as Figure 4 shown, the embodiment of the present application also provides an adaptive planner parameter tuning device 400 based on reinforcement learning, which comprises a low-frequency parameter tuning module 410, a medium-frequency planning module 420, a high-frequency control module 430 and an iterative training module 440.

[0060] The low-frequency parameter tuning module 410 is configured to adaptively tune the local planner in the parameter adjustment network according to the input environmental data using a reinforcement learning algorithm at a low-frequency control frequency.

[0061] The medium-frequency planning module 420 is configured to generate a planning trajectory for the robot to travel in the local planner according to the adjusted parameters and generate a feedforward velocity according to the planning trajectory at a medium-frequency control frequency.

[0062] The high-frequency control module 430 is configured to generate a final control instruction in the controller according to the feedforward velocity and the feedback velocity calculated by the error compensator at a high-frequency control frequency.

[0063] The iterative training module 440 is configured to iteratively and alternately train and test the parameter adjustment network and the controller, and continuously tune the parameters of the local planner and the final control instruction generated by the controller according to the test results.

[0064] Based on the same inventive concept, the embodiments of the present application also provide an electronic device comprising a memory and one or more processors, the memory being configured to store a computer program, and the processor being configured to implement the above-mentioned reinforcement learning-based adaptive planner parameter tuning method when executing the computer program.

[0065] Based on the same inventive concept, the embodiments of the present application also provide a computer-readable storage medium, the storage medium storing a computer program, and the computer program being configured to implement the above-mentioned reinforcement learning-based adaptive planner parameter tuning method when executed by a computer.

[0066] It should be noted that the reinforcement learning-based adaptive planner parameter tuning device, the electronic device and the computer-readable storage medium provided by the above-mentioned embodiments all belong to the same inventive concept as the reinforcement learning-based adaptive planner parameter tuning method, and the specific implementation process is described in detail in the reinforcement learning-based adaptive planner parameter tuning method embodiments, which will not be repeated here.

[0067] The above detailed description of the specific implementation of the technical solutions and advantages of the present application should be understood. It should be understood that the above-mentioned only is the most preferred embodiment of the present application, and is not used to limit the present application, any modification, supplement and equivalent replacement, etc. made within the principle range of the present application, should be included in the protection scope of the present application.

Claims

1. A method for optimizing parameters of an adaptive planner based on reinforcement learning, characterized in that, Includes the following steps: At low frequency control, the parameters of the local planner are adaptively adjusted in the parameter adjustment network based on the input environmental data using a reinforcement learning algorithm. Under the medium frequency control frequency, the local planner generates the robot's planned trajectory based on the adjusted parameters and generates the feedforward velocity based on the planned trajectory. At high frequency control, the final control command is generated in the controller based on the feedforward speed and the feedback speed calculated by the error compensator. The parameter tuning network and controller are iteratively trained and tested alternately, and the parameters of the local planner and the final control commands generated by the controller are continuously optimized based on the test results.

2. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1, characterized in that, The method of adaptively adjusting the parameters of the local planner using a reinforcement learning algorithm based on the input environmental data in the parameter adjustment network at low frequency control includes: At low-frequency control frequencies, the parameter adjustment network uses a reinforcement learning algorithm to train the local planner based on the input environmental data. After the mobile robot runs a certain trajectory, the parameters of the selected local planner are evaluated and adjusted to adaptively adjust the parameters of the local planner.

3. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1 or 2, characterized in that, The state space in the reinforcement learning algorithm uses environmental data acquired by LiDAR as input, and embeds the environmental data into a scene vector through a variational autoencoder network; the action space in the reinforcement learning algorithm includes adjustable parameters in the local planner, including maximum linear velocity, maximum angular velocity, obstacle weight, and expansion radius; the reward function in the reinforcement learning algorithm includes target progress reward and collision penalty. If the robot moves towards the local target point, it will receive the target progress reward; if the robot collides, it will receive the collision penalty.

4. The method for parameter tuning of an adaptive planner based on reinforcement learning according to claim 1, characterized in that, The process of generating a planned trajectory for the robot's movement based on adjusted parameters and generating a feedforward velocity based on the planned trajectory in the local planner at the intermediate frequency control frequency includes: At the intermediate frequency control frequency, the local planner generates a planned trajectory using the TEB algorithm based on the adjusted parameters. The local planner generates a feedforward velocity based on the target pose of the planned trajectory and the robot's current velocity, and guides the robot to travel along the planned trajectory through the feedforward velocity.

5. The method for parameter tuning of an adaptive planner based on reinforcement learning according to claim 1, characterized in that, The process of generating final control commands in the controller based on the feedforward velocity and the feedback velocity calculated by the error compensator at high-frequency control frequencies includes: At high frequency control, the controller receives the feedforward velocity generated by the local planner and obtains the robot state using sensor feedback. The error compensator generates feedback velocity based on environmental data, robot state, key points of the planned trajectory, and time step. The feedback velocity is used to correct the robot's feedforward velocity in real time. The controller generates the final control command based on the feedforward velocity and feedback velocity and sends it to the robot for execution, enabling the robot to accurately follow the planned trajectory.

6. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1 or 5, characterized in that, The error compensator is based on reinforcement learning. It optimizes the error compensator parameters by minimizing the tracking error between the robot's actual position and the planned trajectory by defining a reward function, and then uses the optimized error compensator to generate feedback velocity.

7. The method for parameter tuning of an adaptive planner based on reinforcement learning according to claim 1, characterized in that, The iterative alternating training and testing of the parameter tuning network and controller, and the continuous optimization of the parameters of the local planner and the final control commands generated by the controller based on the test results, include: During the alternating training process, the controller is fixed in one round and the parameter tuning network is trained based on reinforcement learning to select the optimal local planner parameters. Then, in the next round, the parameter tuning network is fixed and the controller is trained based on reinforcement learning to obtain the optimal final control command. After each round of training, robot trajectory tracking tests are conducted in a simulation environment or a real environment. Based on the test results, the parameters of the local planner in the parameter tuning network are further optimized and the controller is adjusted. Through multiple rounds of iterative training and robot trajectory tracking tests, the tracking error between the robot's actual position and the planned trajectory is gradually reduced, and the parameters of the local planner and the final control commands generated by the controller are continuously optimized.

8. A parameter tuning device for an adaptive planner based on reinforcement learning, implemented using the parameter tuning method for an adaptive planner based on reinforcement learning as described in any one of claims 1-7, characterized in that, include: The module includes a low-frequency parameter tuning module, a mid-frequency planning module, a high-frequency control module, and an iterative training module. The low-frequency parameter tuning module is used to adaptively adjust the parameters of the local planner in the parameter tuning network based on the input environmental data using a reinforcement learning algorithm under low-frequency control frequency. The intermediate frequency planning module is used to generate a planned trajectory for the robot's movement in the local planner based on the adjusted parameters under the intermediate frequency control frequency, and to generate a feedforward velocity based on the planned trajectory. The high-frequency control module is used to generate the final control command in the controller based on the feedforward speed and the feedback speed calculated by the error compensator at a high-frequency control frequency. The iterative training module is used to iteratively train and test the parameter adjustment network and the controller, and continuously optimize the parameters of the local planner and the final control instructions generated by the controller based on the test results.

9. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is configured to implement the reinforcement learning-based adaptive planner parameter tuning method according to any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the reinforcement learning-based adaptive planner parameter tuning method according to any one of claims 1-7.

Images