Adaptive planner parameter tuning method and device based on reinforcement learning

Abstract

The invention discloses an adaptive planner parameter tuning method and device based on reinforcement learning, and the method comprises the steps: employing a reinforcement learning algorithm to adaptively carry out the parameter adjustment of a local planner according to the input environment data in a parameter adjustment network under a low-frequency control frequency; under the medium-frequency control frequency, generating a planning track of robot advancing according to the adjusted parameters in the local planner, and generating a feed-forward speed according to the planning track; under the high-frequency control frequency, a final control instruction is generated in the controller according to the feed-forward speed and the feedback speed calculated by the error compensator; and iteratively and alternately training and testing the parameter adjustment network and the controller, and continuously adjusting and optimizing the parameters of the local planner and a final control instruction generated by the controller according to a test result. According to the invention, the robustness and navigation efficiency of the system can be improved, the tracking error is effectively reduced, the obstacle avoidance capability is enhanced, and a more accurate, efficient and reliable robot navigation task can be completed in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a method and device for tuning parameters of an adaptive planner based on reinforcement learning. Background Art

[0002] Trajectory planning in highly constrained environments is a crucial research direction in the field of mobile robots. Traditional trajectory planning methods, such as the elastic band method and the dynamic window method, rely on accurate environment modeling and controller parameter settings to generate feasible trajectories in known environments and achieve autonomous navigation of robots. However, these traditional methods often face limitations in dynamic or complex environments and have difficulty coping with environmental changes or unstructured obstacle distribution. In recent years, with the development of artificial intelligence technology, especially learning-based methods such as imitation learning (IL) and reinforcement learning (RL), they have gradually been applied to trajectory planning.

[0003] Imitation learning is a method of training models through expert demonstration data, which is particularly prominent in the field of autonomous driving. By imitating the actions of experts, imitation learning can generate trajectories or control commands. However, due to the lack of large-scale expert datasets in the field of mobile robots, imitation learning methods are not applicable in highly constrained environments. In addition, imitation learning usually adopts an open loop framework and relies on the quality of demonstration data. The actual execution feedback of the robot cannot be incorporated into the learning process, so when the robot encounters new situations, the system has poor adaptability.

[0004] Unlike imitation learning, reinforcement learning is a closed-loop learning method that adjusts strategies through real-time interactive feedback between the robot and the environment, avoiding dependence on demonstration data, being able to better adapt to complex environments, and having a higher performance ceiling. Reinforcement learning not only takes into account the dynamic changes of the environment, but also guides the robot to conduct effective exploration by setting a reward function, thereby learning a more robust navigation strategy. However, there are some challenges in the direct application of reinforcement learning in trajectory planning. For example, reinforcement learning often requires a large number of exploration steps, resulting in low sample efficiency, especially when directly learning speed control strategies. In order to overcome this problem, an adaptive planner parameter tuning system based on reinforcement learning has been developed in recent years. By automatically adjusting the key parameters of the traditional planning algorithm (such as maximum speed, expansion radius, etc.), it can significantly improve the efficiency of trajectory generation and execution, while reducing unnecessary exploration processes.

[0005] The advantage of the adaptive planner parameter tuning method is that it can adjust the planner parameters in real time according to environmental changes, without relying on fixed parameters or manual debugging. For example, the Adaptive Planner Parameter Learning (APPL) system proposes an adaptive planner parameter tuning framework based on reinforcement learning, which dynamically selects the optimal parameters through reinforcement learning strategies and adapts to different navigation scenarios.

[0006] However, although existing adaptive planner parameter tuning methods have made significant progress, they often focus only on optimization at the tuning level and ignore the limitations of the control layer. In a closed-loop system, the tuning network relies on the assumption that the controller can accurately execute the planned trajectory, but due to the pure feedforward nature of the control system, there is often a deviation between the executed trajectory and the planned trajectory, resulting in tracking errors. This error introduces noise and reduces the learning effect of the tuning network. Therefore, the problem cannot be completely solved by optimizing the tuning level alone, and the overall performance of the system is still limited by the control layer. Summary of the invention

[0007] In view of the above, the purpose of the present invention is to provide a method and device for parameter tuning of an adaptive planner based on reinforcement learning. By designing a hierarchical architecture based on low-frequency parameter tuning, medium-frequency planning and high-frequency control, reinforcement learning adaptive planner parameter tuning is performed. Not only attention is paid to the optimization at the tuning level, but also the limitations of the control layer are taken into account to further reduce the tracking error. At the same time, the reinforcement learning error compensator is combined to correct the tracking error in real time, thereby effectively handling the dynamic obstacle avoidance requirements in complex environments. The iterative training framework of alternating training is used to ultimately achieve continuous improvement of the overall system performance, which is used to realize more accurate, efficient and reliable mobile robot navigation task execution in complex environments.

[0008] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0009] In a first aspect, an embodiment of the present invention provides a method for tuning parameters of an adaptive planner based on reinforcement learning, comprising the following steps:

[0010] At low control frequencies, the reinforcement learning algorithm is used in the parameter adjustment network to adaptively adjust the parameters of the local planner according to the input environmental data;

[0011] Under the medium frequency control frequency, a planned trajectory for the robot to move is generated in the local planner according to the adjusted parameters and a feedforward velocity is generated according to the planned trajectory;

[0012] At high control frequency, a final control command is generated in the controller according to the feedforward speed and the feedback speed calculated by the error compensator;

[0013] The parameter adjustment network and the controller are iteratively trained and tested alternately, and the parameters of the local planner and the final control instructions generated by the controller are continuously tuned according to the test results.

[0014] Preferably, the method of adaptively adjusting the parameters of the local planner using a reinforcement learning algorithm in the parameter adjustment network according to input environmental data under the low-frequency control frequency includes:

[0015] Under low-frequency control frequency, the parameter adjustment network uses a reinforcement learning algorithm to train the local planner according to the input environmental data. Every time the mobile robot runs a trajectory, the parameters of the selected local planner are evaluated and adjusted to adaptively adjust the parameters of the local planner.

[0016] Preferably, the state space in the reinforcement learning algorithm uses environmental data obtained by the lidar as input, and embeds the environmental data into a scene vector through a variational autoencoder network; the action space in the reinforcement learning algorithm includes adjustable parameters in the local planner, and the parameters include maximum linear velocity, maximum angular velocity, obstacle weight and expansion radius; the reward function in the reinforcement learning algorithm includes a target progress reward and a collision penalty. If the robot moves toward the local target point, it obtains a target progress reward, and if the robot collides, it obtains a collision penalty.

[0017] Preferably, generating a planned trajectory for the robot to move according to the adjusted parameters in a local planner under the intermediate frequency control frequency and generating a feedforward speed according to the planned trajectory comprises:

[0018] Under the medium frequency control frequency, the local planner uses the TEB algorithm to generate a planned trajectory according to the adjusted parameters. The local planner generates a feedforward speed according to the target posture of the planned trajectory and the current speed of the robot, and guides the robot to travel along the planned trajectory through the feedforward speed.

[0019] Preferably, generating a final control instruction in a controller according to a feedforward speed and a feedback speed calculated by an error compensator under a high-frequency control frequency comprises:

[0020] Under high-frequency control frequency, the controller receives the feedforward speed generated by the local planner and uses sensor feedback to obtain the robot state. The error compensator generates feedback speed based on environmental data, robot state, key points of the planned trajectory and time step. The robot's feedforward speed is corrected in real time through the feedback speed. The controller generates the final control instruction based on the feedforward speed and feedback speed and sends it to the robot for execution, so that the robot can accurately follow the planned trajectory.

[0021] Preferably, the error compensator is based on reinforcement learning, and a reward function is defined to minimize the tracking error between the actual position of the robot and the planned trajectory to optimize the error compensator parameters, and the optimized error compensator is used to generate a feedback speed.

[0022] Preferably, the iteratively alternating training and testing of the parameter adjustment network and the controller, and continuously tuning the parameters of the local planner and the final control instructions generated by the controller according to the test results, include:

[0023] In the alternating training process, the controller is first fixed in one round and the parameter adjustment network is trained based on reinforcement learning to select the optimal local planner parameters. Then, in the next round, the parameter adjustment network is fixed and the controller is trained based on reinforcement learning to obtain the optimal final control instructions.

[0024] After each round of training, the robot trajectory tracking test is carried out in a simulation environment or an actual environment. According to the test results, the local planner in the parameter adjustment network is further tuned and the controller is adjusted.

[0025] Through multiple rounds of iterative alternating training and robot trajectory tracking tests, the tracking error between the actual position of the robot and the planned trajectory is gradually reduced, and the parameters of the local planner and the final control instructions generated by the controller are continuously optimized.

[0026] In a second aspect, an embodiment of the present invention provides an adaptive planner parameter tuning device based on reinforcement learning, which is implemented by the above-mentioned adaptive planner parameter tuning method based on reinforcement learning, and includes: a low-frequency parameter tuning module, a medium-frequency planning module, a high-frequency control module and an iterative training module;

[0027] The low-frequency parameter tuning module is used to adaptively adjust the parameters of the local planner using a reinforcement learning algorithm in the parameter adjustment network according to the input environmental data at the low-frequency control frequency;

[0028] The intermediate frequency planning module is used to generate a planned trajectory for the robot to move according to the adjusted parameters in the local planner under the intermediate frequency control frequency and generate a feedforward speed according to the planned trajectory;

[0029] The high-frequency control module is used to generate a final control instruction in the controller according to the feedforward speed and the feedback speed calculated by the error compensator at the high-frequency control frequency;

[0030] The iterative training module is used to iteratively and alternately train and test the parameter adjustment network and the controller, and continuously tune the parameters of the local planner and the final control instructions generated by the controller according to the test results.

[0031] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned reinforcement learning-based adaptive planner parameter tuning method when executing the computer program.

[0032] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned reinforcement learning-based adaptive planner parameter tuning method is implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) Improving system robustness and navigation efficiency: The present invention adopts a hierarchical architecture design based on low-frequency parameter tuning, medium-frequency planning and high-frequency control, so that high-level parameter tuning and low-level control strategies can be optimized synergistically, thereby improving the stability and robustness of the control system.

[0035] (2) Effectively reducing tracking errors: The present invention can further reduce trajectory tracking errors and ensure obstacle avoidance capabilities by introducing an error compensation mechanism based on the calculation of feedback speed by an error compensator, and continuously optimizing the parameters of the local planner and the final control instructions generated by the controller.

[0036] (3) Enhance the obstacle avoidance capability of the system: In complex environments, the present invention uses an automated parameter tuning mechanism based on lidar feedback combined with reinforcement learning to more flexibly avoid dynamic obstacles, thereby achieving more accurate, efficient, and reliable navigation task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 It is a flow chart of a method for optimizing parameters of an adaptive planner based on reinforcement learning provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a layered framework provided by an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of iteratively executing alternating training provided by an embodiment of the present invention;

[0041] Figure 4It is a structural diagram of a parameter tuning device for an adaptive planner based on reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0043] The inventive concept of the present invention is: in view of the problem that the adaptive planner parameter tuning method based on reinforcement learning in the prior art only focuses on the optimization of the tuning level and ignores the limitation of the control layer, the embodiment of the present invention provides an adaptive planner parameter tuning method and device based on reinforcement learning, at a low-frequency control frequency, the parameters of the local planner are adaptively adjusted through reinforcement learning to adapt to different environments, at a medium-frequency control frequency, the local planner generates a planning trajectory according to the adjusted parameters and provides a feedforward speed for the controller, at a high-frequency control frequency, the controller combines the feedforward speed and the feedback speed to calculate and generate the final control instruction to reduce the trajectory tracking error, through hierarchical design, not only the high-level parameter tuning is optimized, but also the robustness of the underlying controller is enhanced, and the overall performance of the entire system is improved, so that the robot control system can better adapt to changes in complex environments. In addition, by introducing an error compensation mechanism based on reinforcement learning, the tracking error caused by the nonlinear dynamic characteristics of the system is further reduced, and the success rate and efficiency of the navigation task are improved.

[0044] Figure 1 FIG. 1 is a flow chart of a method for tuning parameters of an adaptive planner based on reinforcement learning provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a method for tuning parameters of an adaptive planner based on reinforcement learning, comprising the following steps:

[0045] S1, under low control frequency, the reinforcement learning algorithm is used in the parameter adjustment network to adaptively adjust the parameters of the local planner according to the input environmental data.

[0046] In the embodiment, the control frequency refers to the frequency of executing the control cycle in the control system, that is, the number of times the control cycle is executed per second, expressed in Hertz (Hz). The optional range of the low-frequency control frequency is 0.1Hz-1Hz, as shown in Figure 2In the hierarchical framework shown in the figure, at a low-frequency control frequency of 1 Hz, the parameter adjustment network uses a reinforcement learning algorithm to train the local planner according to the input environmental data to adaptively adjust the parameters of the local planner. Every time the mobile robot runs a trajectory, the parameters of the selected local planner are evaluated and the strategy is updated according to the actual execution results. By repeatedly adjusting the parameters, the system gradually learns how to dynamically select the optimal parameters according to environmental changes, thereby optimizing trajectory generation. The key adjustable parameters include the maximum linear velocity (max_vel_x), the maximum angular velocity (max_vel_theta), the obstacle weight (weight_obstacle) and the inflation radius (inflation_radius).

[0047] The reinforcement learning algorithm adopts a double-delayed deep deterministic policy gradient algorithm (TD3 algorithm). In the state space, the environmental data obtained by the laser radar (LiDAR) is used as input. In order to effectively compress these high-dimensional laser radar data, the present invention adopts a variational autoencoder (VAE) network to embed it into a scene vector. This low-dimensional representation can capture the obstacle distribution and path information in the environment. In the action space, the action consists of a set of adjustable parameters, such as maximum speed, rotation speed, obstacle weight, and expansion radius. These parameters are used to adjust the trajectory generation algorithm of the local planner. During each execution, the system will adaptively adjust these parameters based on the state of the current environment. The reward function is used to guide the optimization of the parameter adjustment network, including the target progress reward (R g ) and collision penalty (R c ), if the robot successfully avoids obstacles and reaches the target, it will receive a target progress reward, and if the robot collides, it will receive a collision penalty. The formula is:

[0048]

[0049] Among them, it represents r arrival The reward value for reaching the target, w represents the weight, d t represents the distance between the robot and the target point at time t, r collision Indicates the penalty value for a collision.

[0050] S2, under the medium frequency control frequency, generates the planned trajectory of the robot according to the adjusted parameters in the local planner and generates the feedforward velocity according to the planned trajectory.

[0051] In the embodiment, the optional range of the intermediate frequency control frequency is 1 Hz-10 Hz (excluding 1 Hz), referring to Figure 2In the hierarchical framework shown in the figure, at a medium frequency control frequency of 10 Hz, the local planner uses the timed elastic band algorithm (TEB algorithm) according to the adjusted parameters to generate a smooth and executable planning trajectory for the robot in the next few seconds by optimizing the distance and time. Then, the local planner generates a feedforward velocity based on the target pose of the planned trajectory and the current velocity of the robot, and guides the robot to drive along the planned trajectory through the feedforward velocity. In this step, the accuracy of the trajectory and the obstacle avoidance capability are guaranteed by combining the lidar data and the robot status information.

[0052] S3, at a high control frequency, a final control command is generated in the controller according to the feedforward speed and the feedback speed calculated by the error compensator.

[0053] In the embodiment, the optional range of the high frequency control frequency is greater than 10 Hz, as shown in Figure 2 In the hierarchical framework shown in the figure, at a high-frequency control frequency of 50 Hz, the controller receives the feedforward velocity generated by the local planner and uses sensor feedback to obtain the robot state (such as position, velocity, and posture). The goal of the controller is to minimize the tracking error between the actual position of the robot and the planned trajectory. The error compensator generates feedback velocity based on the environmental data obtained by the lidar, the robot state, the key points of the planned trajectory, and the time step, and corrects the robot's feedforward velocity in real time through the feedback velocity. The controller generates the final control command based on the feedforward velocity and feedback velocity and sends it to the robot for execution, so that the robot can accurately follow the planned trajectory, thereby optimizing the trajectory tracking performance.

[0054] In order to compensate for the error caused by the nonlinear dynamic characteristics of the system during the execution of the robot, the present invention designs an error compensator based on reinforcement learning. By defining a reward function to minimize the tracking error between the actual position of the robot and the planned trajectory to optimize the error compensator parameters, the optimized error compensator is used to generate feedback speed, and the reward function will be dynamically adjusted according to factors such as the distance between the robot and the trajectory and whether a collision occurs. The compensator is not only used to correct the tracking error so that it can follow the planned trajectory more accurately to ensure control accuracy, but also can adjust the control strategy in real time and correct the robot speed in real time through the combination of high-frequency feedback control mechanism and laser radar data to cope with dynamic obstacles in complex environments, effectively handle dynamic obstacle avoidance requirements in complex environments, flexibly handle obstacles on the robot's path, and achieve efficient obstacle avoidance capabilities.

[0055] S4, iteratively trains and tests the parameter adjustment network and the controller alternately, and continuously tunes the parameters of the local planner and the final control instructions generated by the controller according to the test results.

[0056] In the embodiment, during the alternating training process, the controller is first fixed in one round and the parameter adjustment network is trained based on the TD3 algorithm in reinforcement learning to select the optimal local planner parameters, and then the parameter adjustment network is fixed in the next round and the controller is trained based on the TD3 algorithm in reinforcement learning to obtain the optimal final control instructions. The TD3 algorithm can effectively solve the deviation problem in the continuous control task of the robot and improve the stability and robustness of the system. Multiple rounds of iterative training are performed in this alternating training method, so that the high-level parameter tuning network and the low-level controller can complement each other, thereby gradually improving the overall performance of the system.

[0057] After each round of training, the robot trajectory tracking test is carried out in a simulation environment or an actual environment. According to the test results, the parameters of the local planner in the parameter adjustment network are further tuned and the controller is adjusted. Through multiple rounds of iterative alternating training and robot trajectory tracking tests, the tracking error between the actual position of the robot and the planned trajectory is gradually reduced, and the parameters of the local planner and the final control instructions generated by the controller are continuously optimized to improve the success rate and completion efficiency of the task.

[0058] In summary, the embodiment of the present invention provides a method for parameter tuning of an adaptive planner based on reinforcement learning, which can improve the robustness and navigation efficiency of the system. The experimental results show that the combination of low-frequency parameter tuning with a frequency of 1 Hz and a high-frequency control strategy with a frequency of 50 Hz can effectively reduce the trajectory tracking error compared with the traditional control system with a frequency of 10 Hz, which is specifically manifested in that the success rate is increased from 70% to 98%, and the task completion time is reduced from 26.7 seconds to 10.2 seconds. This significant improvement is due to the refined error compensation and adaptive parameter tuning of the reinforcement learning controller. It can also effectively reduce the tracking error. In multiple rounds of simulation experiments, the control strategy of the present invention can effectively reduce the tracking error between the actual trajectory of the robot and the expected trajectory compared with the default planner. In different scenarios, by introducing the reinforcement learning compensation mechanism, the tracking error is reduced by an average of 30% to 50%. It can also enhance the obstacle avoidance capability of the system, so that the robot can run smoothly in a more crowded and complex environment without frequent human intervention, reducing the dependence on expert knowledge and trial and error experiments. This adaptability greatly improves the performance of the robot navigation system in a complex environment.

[0059] Based on the same inventive concept, Figure 4 As shown, an embodiment of the present invention also provides an adaptive planner parameter tuning device 400 based on reinforcement learning, including: a low-frequency parameter tuning module 410, a medium-frequency planning module 420, a high-frequency control module 430 and an iterative training module 440.

[0060] The low-frequency parameter tuning module 410 is used to adaptively adjust the parameters of the local planner using a reinforcement learning algorithm in the parameter adjustment network according to input environmental data at a low-frequency control frequency.

[0061] The intermediate frequency planning module 420 is used to generate a planned trajectory for the robot to move according to the adjusted parameters in the local planner under the intermediate frequency control frequency and to generate a feedforward speed according to the planned trajectory.

[0062] The high frequency control module 430 is used to generate a final control instruction in the controller according to the feedforward speed and the feedback speed calculated by the error compensator at the high frequency control frequency.

[0063] The iterative training module 440 is used to iteratively and alternately train and test the parameter adjustment network and the controller, and continuously tune the parameters of the local planner and the final control instructions generated by the controller according to the test results.

[0064] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory and one or more processors, the memory is used to store a computer program, and the processor is used to implement the above-mentioned reinforcement learning-based adaptive planner parameter tuning method when executing the computer program.

[0065] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned reinforcement learning-based adaptive planner parameter tuning method is implemented.

[0066] It should be noted that the adaptive planner parameter tuning device based on reinforcement learning, the electronic device and the computer-readable storage medium provided in the above embodiments all belong to the same inventive concept as the adaptive planner parameter tuning method based on reinforcement learning. The specific implementation process is detailed in the embodiment of the adaptive planner parameter tuning method based on reinforcement learning, which will not be repeated here.

[0067] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for tuning parameters of an adaptive planner based on reinforcement learning, characterized in that: The following steps are involved: At low control frequencies, the reinforcement learning algorithm is used in the parameter adjustment network to adaptively adjust the parameters of the local planner according to the input environmental data; Under the medium frequency control frequency, a planned trajectory for the robot to move is generated in the local planner according to the adjusted parameters and a feedforward velocity is generated according to the planned trajectory; At high control frequency, a final control command is generated in the controller according to the feedforward speed and the feedback speed calculated by the error compensator; The parameter adjustment network and the controller are iteratively trained and tested alternately, and the parameters of the local planner and the final control instructions generated by the controller are continuously tuned according to the test results.

2. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1, characterized in that: The method of adaptively adjusting the parameters of the local planner using a reinforcement learning algorithm in the parameter adjustment network according to input environmental data under the low-frequency control frequency includes: Under low-frequency control frequency, the parameter adjustment network uses a reinforcement learning algorithm to train the local planner according to the input environmental data. Every time the mobile robot runs a trajectory, the parameters of the selected local planner are evaluated and adjusted to adaptively adjust the parameters of the local planner.

3. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1 or 2, characterized in that: The state space in the reinforcement learning algorithm uses environmental data obtained by the lidar as input, and embeds the environmental data into a scene vector through a variational autoencoder network; the action space in the reinforcement learning algorithm includes adjustable parameters in the local planner, including maximum linear velocity, maximum angular velocity, obstacle weight, and expansion radius; the reward function in the reinforcement learning algorithm includes target progress reward and collision penalty. If the robot moves toward the local target point, it will receive a target progress reward, and if the robot collides, it will receive a collision penalty.

4. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1, characterized in that: The method of generating a planned trajectory of the robot according to the adjusted parameters in the local planner under the intermediate frequency control frequency and generating a feedforward speed according to the planned trajectory comprises: Under the medium frequency control frequency, the local planner uses the TEB algorithm to generate a planned trajectory according to the adjusted parameters. The local planner generates a feedforward speed according to the target posture of the planned trajectory and the current speed of the robot, and guides the robot to travel along the planned trajectory through the feedforward speed.

5. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1, characterized in that: The method of generating a final control instruction in a controller according to a feedforward speed and a feedback speed calculated by an error compensator under a high-frequency control frequency includes: Under high-frequency control frequency, the controller receives the feedforward speed generated by the local planner and uses sensor feedback to obtain the robot state. The error compensator generates feedback speed based on environmental data, robot state, key points of the planned trajectory and time step. The robot's feedforward speed is corrected in real time through the feedback speed. The controller generates the final control instruction based on the feedforward speed and feedback speed and sends it to the robot for execution, so that the robot can accurately follow the planned trajectory.

6. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1 or 5, characterized in that: The error compensator is based on reinforcement learning. The reward function is defined to minimize the tracking error between the actual position of the robot and the planned trajectory to optimize the error compensator parameters, and the optimized error compensator is used to generate feedback speed.

7. The method for optimizing parameters of an adaptive planner based on reinforcement learning according to claim 1, characterized in that: The parameter adjustment network and the controller are iteratively and alternately trained and tested, and the parameters of the local planner and the final control instructions generated by the controller are continuously tuned according to the test results, including: In the alternating training process, the controller is first fixed in one round and the parameter adjustment network is trained based on reinforcement learning to select the optimal local planner parameters. Then, in the next round, the parameter adjustment network is fixed and the controller is trained based on reinforcement learning to obtain the optimal final control instructions. After each round of training, the robot trajectory tracking test is carried out in a simulation environment or an actual environment. According to the test results, the local planner in the parameter adjustment network is further tuned and the controller is adjusted. Through multiple rounds of iterative alternating training and robot trajectory tracking tests, the tracking error between the actual position of the robot and the planned trajectory is gradually reduced, and the parameters of the local planner and the final control instructions generated by the controller are continuously optimized.

8. A device for optimizing parameters of an adaptive planner based on reinforcement learning, implemented by the method for optimizing parameters of an adaptive planner based on reinforcement learning according to any one of claims 1 to 7, characterized in that: include: Low-frequency parameter tuning module, medium-frequency planning module, high-frequency control module and iterative training module; The low-frequency parameter tuning module is used to adaptively adjust the parameters of the local planner using a reinforcement learning algorithm in the parameter adjustment network according to the input environmental data at the low-frequency control frequency; The intermediate frequency planning module is used to generate a planned trajectory for the robot to move according to the adjusted parameters in the local planner under the intermediate frequency control frequency and generate a feedforward speed according to the planned trajectory; The high-frequency control module is used to generate a final control instruction in the controller according to the feedforward speed and the feedback speed calculated by the error compensator at the high-frequency control frequency; The iterative training module is used to iteratively and alternately train and test the parameter adjustment network and the controller, and continuously tune the parameters of the local planner and the final control instructions generated by the controller according to the test results.

9. An electronic device comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the reinforcement learning-based adaptive planner parameter tuning method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the method for optimizing parameters of an adaptive planner based on reinforcement learning according to any one of claims 1 to 7 is implemented.

Images