Hierarchical reinforcement learning based autonomous driving method and device

Abstract

The present disclosure provides an automatic driving method and device based on hierarchical reinforcement learning, the method comprising: obtaining driving data, the driving data comprising background data of a simulation driving platform; training an encoder according to the driving data to obtain a multi-dimensional feature vector; performing unsupervised training on the multi-dimensional feature vector to obtain a plurality of initial skills, the unsupervised training comprising training with information entropy constraint; performing hierarchical reinforcement training on the multi-dimensional feature vector to obtain a controller, the hierarchical reinforcement training comprising adjusting the loss of a deep learning network; and combining the plurality of initial skills through the controller to perform an automatic driving action. The method of the present disclosure can train a plurality of general skills, the training of the skills is objective and efficient, the skills are combined through the controller to realize automatic driving, the method has strong interpretability, and the automatic driving can cope with complex environments.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to an autonomous driving method and device based on hierarchical reinforcement learning. Background Art

[0002] With the rapid development of autonomous driving in recent years, a growing number of autonomous driving algorithms have emerged. Currently, there are two frameworks for autonomous driving algorithms: modular and end-to-end. The end-to-end approach treats driving as a task. The input is raw data collected by various sensors, and neural networks handle all intermediate processes. The output then outputs the vehicle's underlying control, such as steering wheel angle and speed.

[0003] However, existing autonomous driving algorithms have problems such as poor interpretability, the need for manual setting of rewards for each action, and sparse rewards. Summary of the Invention

[0004] (1) Technical issues to be solved

[0005] In response to existing technical problems, the present disclosure provides an autonomous driving method based on hierarchical reinforcement learning, which is used to at least partially solve the above technical problems.

[0006] (2) Technical solution

[0007] The present disclosure provides an autonomous driving method based on hierarchical reinforcement learning, comprising: acquiring driving data, the driving data including background data of a simulated driving platform; training an encoder based on the driving data to obtain a multidimensional feature vector; performing unsupervised training on the multidimensional feature vector to obtain a plurality of initial skills, the unsupervised training including training with information entropy constraints; performing hierarchical reinforcement training on the multidimensional feature vector to obtain a controller, the hierarchical reinforcement training including adjusting the loss of a deep learning network; and combining the plurality of initial skills through the controller to execute autonomous driving actions.

[0008] Optionally, training an encoder according to the driving data to obtain a multidimensional feature vector includes: fusing the background data based on a rule of the background data to obtain a bird's-eye view; and extracting features of the bird's-eye view to obtain a multidimensional feature vector.

[0009] Optionally, the driving data also includes image data and lidar data. Training an encoder based on the driving data to obtain a multi-dimensional feature vector further includes: performing bird's-eye view fusion on the image data and the lidar data to obtain a bird's-eye view with supplementary information.

[0010] Optionally, performing unsupervised training on the multidimensional feature vector to obtain multiple initial skills includes: selecting partial dimensional feature vectors from the multidimensional feature vector; obtaining multiple original skills based on the prior distribution of the partial dimensional feature vectors; training the partial dimensional feature vectors based on information entropy constraints to obtain endogenous pseudo rewards; training multiple original skills based on the endogenous pseudo rewards to obtain multiple initial skills.

[0011] Optionally, training multiple original skills according to the endogenous pseudo-reward to obtain multiple initial skills includes: adding a speed reward to the endogenous pseudo-reward; training multiple original skills according to the endogenous pseudo-reward after adding the speed reward to obtain multiple initial skills; wherein the speed reward includes: when the speed is positive, positively rewarding the multiple original skills; and when the speed is 0 or negative, negatively rewarding the multiple original skills.

[0012] Optionally, performing unsupervised training on the multi-dimensional feature vector to obtain multiple initial skills also includes: obtaining partial dimensional feature vectors of multiple consecutive frames; determining the relationship between the partial dimensional feature vectors of each frame based on the self-attention mechanism of the converter to obtain time series information; and training multiple original skills according to the time series information and endogenous pseudo rewards to obtain multiple initial skills.

[0013] Optionally, performing layered reinforcement training on the multidimensional feature vector to obtain a controller includes: superimposing the multidimensional feature vectors of multiple consecutive frames to obtain a state representation; determining the environmental reward of the simulated driving platform; using an action-value function learning method combined with a deep learning network, and training the state representation according to the environmental reward to obtain the controller; wherein the deep learning network is set with criticism loss and terminal loss.

[0014] Optionally, the autonomous driving method based on hierarchical reinforcement learning also includes: conducting supervised training on multiple initial skills to obtain multiple optimized skills, wherein the supervised training includes training with the addition of environmental rewards; and combining multiple optimized skills through a controller to perform autonomous driving actions.

[0015] Optionally, the autonomous driving method based on hierarchical reinforcement learning also includes: scoring multiple initial skills through a controller to obtain the action value of each initial skill.

[0016] On the other hand, the present disclosure provides an autonomous driving device based on hierarchical reinforcement learning, including: an acquisition module for acquiring driving data, wherein the driving data includes background data of a simulated driving platform; an encoding module for training an encoder based on the driving data to obtain a multidimensional feature vector; a skill training module for performing unsupervised training on the multidimensional feature vector to obtain multiple initial skills, wherein the unsupervised training includes training with information entropy constraints; a control training module for performing hierarchical reinforcement training on the multidimensional feature vector to obtain a controller, wherein the hierarchical reinforcement training includes adjusting the loss of a deep learning network; and a driving module for combining multiple initial skills through the controller to perform autonomous driving actions.

[0017] (3) Beneficial effects

[0018] This disclosure provides an autonomous driving method based on hierarchical reinforcement learning. This method trains multiple skills (e.g., a combination of multiple actions) based on background data in a simulated driving platform environment. Then, a controller is trained to flexibly combine these skills to achieve autonomous driving, tailored to driving needs. Skill training utilizes an unsupervised learning approach, using information entropy constraints to drive the agent's skill learning. This avoids the subjectivity and inefficiency of manually designed rewards, allows for the simultaneous learning of multiple skills, and enables autonomous driving to cope with complex environments.

[0019] The skills disclosed in this paper are universal. By training the upper-level meta-controller for different problems, task requirements can be quickly met, solving the problem of difficult policy changes. They also summarize previously learned skills into skills, allowing for reuse of previously learned knowledge, allowing the learned skills to be applied to the next similar scenario.

[0020] This disclosure breaks down the process of reinforcement learning decision-making into skill training and skill combination, which can more intuitively observe the decision-making steps in autonomous driving and more conveniently perform error correction, modification and interpretation of intelligent systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1 The flowchart of the autonomous driving method based on hierarchical reinforcement learning according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 2 The schematic diagram shows the algorithm structure of the autonomous driving method based on hierarchical reinforcement learning;

[0024] Figure 3Schematically shows a bird's-eye view obtained by data fusion according to an embodiment of the present disclosure;

[0025] Figure 4 The following schematically shows a skill learning flow chart according to an embodiment of the present disclosure;

[0026] Figure 5 Schematically shows a skill learning interface diagram according to an embodiment of the present disclosure;

[0027] Figure 6 The system framework of the autonomous driving method based on hierarchical reinforcement learning according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 7 Schematically shows a total reward curve diagram for skill training using a relevant algorithm according to an embodiment of the present disclosure;

[0029] Figure 8 Schematically shows a total reward curve diagram for skill training using an optimization algorithm according to an embodiment of the present disclosure;

[0030] Figure 9 A block diagram of an autonomous driving device based on hierarchical reinforcement learning according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0032] It should be noted that in the drawings or descriptions of the specification, similar or identical parts use the same figure numbers. The technical features in the various embodiments exemplified in the specification can be freely combined to form new solutions without conflict. In addition, each claim can be used as an embodiment alone or the technical features in each claim can be combined as a new embodiment. In the drawings, the shape or thickness of the embodiment can be expanded and simplified or conveniently indicated. Furthermore, the elements or implementations not shown or described in the drawings are forms known to ordinary technicians in the relevant technical field. In addition, although this article may provide demonstrations of parameters containing specific values, it should be understood that the parameters do not need to be exactly equal to the corresponding values, but can be approximated to the corresponding values ​​within an acceptable error tolerance or design constraint.

[0033] Unless there are technical obstacles or contradictions, the above-mentioned various embodiments of the present disclosure can be freely combined to form additional embodiments, and these additional embodiments are all within the protection scope of the present disclosure.

[0034] Although the present disclosure is described in conjunction with the accompanying drawings, the embodiments disclosed in the drawings are intended to illustrate preferred embodiments of the present disclosure and are not to be construed as limiting the present disclosure. The dimensional ratios in the drawings are merely illustrative and are not to be construed as limiting the present disclosure.

[0035] Although some embodiments of the present general inventive concept have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the present general inventive concept, the scope of which is defined in the claims and their equivalents.

[0036] Figure 1 A flowchart of an autonomous driving method based on hierarchical reinforcement learning according to an embodiment of the present disclosure is schematically shown.

[0037] According to the embodiments of the present disclosure, Figure 1 As shown, the autonomous driving method based on hierarchical reinforcement learning includes, for example:

[0038] S110, obtaining driving data, where the driving data includes background data of the simulation driving platform.

[0039] Figure 2 The figure schematically shows an algorithm structure diagram of an autonomous driving method based on hierarchical reinforcement learning according to an embodiment of the present disclosure.

[0040] According to the embodiments of the present disclosure, Figure 2 As shown, the autonomous driving algorithm disclosed in the present invention mainly includes two parts, a multi-scale feature vector (i.e., multi-dimensional feature vector) generation part based on encoder learning and a strategy learning part including skill learning and controller training. Through encoder learning, the multi-dimensional feature vector required for training skills and controllers can be obtained. Through strategy learning, multiple skills and controllers can be trained, and multiple skills can be flexibly combined through the controller to cope with complex autonomous driving environments. Among them, the strategy learning part, for example, first uses an unsupervised skill learning method to learn skills, and then applies the trained skills to hierarchical reinforcement learning. Finally, the skills are fine-tuned according to the environmental rewards, and the turning angle and speed of the autonomous driving are output to control the action of the autonomous driving vehicle.

[0041] For example, the input data for encoder learning is the background data of the driving simulation platform. This data is easily accessible and has a comprehensive set of data usage rules. By training with the driving simulation platform's background data, the multidimensional feature vectors required for subsequent skill learning can be efficiently generated. To increase data richness and better adapt to the autonomous driving environment, image data and lidar data can also be used as supplementary or independent input data. Image data can be images captured by the vehicle's cameras, and lidar data can be data obtained by the vehicle's lidar for ranging and object detection.

[0042] S120: Train an encoder based on the driving data to obtain a multi-dimensional feature vector.

[0043] According to embodiments of the present disclosure, when the input data is background data from a simulated driving platform, for example, the background data can be fused based on the background data rules to obtain a bird's-eye view. Features of the bird's-eye view can be extracted, for example, by performing convolution operations on the bird's-eye view using an encoder to obtain a multidimensional feature vector.

[0044] For example, the input data may also include image data and lidar data, and a bird's-eye view fusion (BEVfusion) method based on image data and lidar data may be used to obtain a bird's-eye view with supplementary information.

[0045] Figure 3 The figure schematically shows a bird's-eye view obtained by data fusion according to an embodiment of the present disclosure.

[0046] According to the embodiment of the present disclosure, after data fusion, the following can be obtained: Figure 3 A grayscale image of a bird's-eye view is shown. The white rectangle represents the test vehicle, the gray rectangle behind it represents background vehicles, the light-colored dots represent pedestrians, the light-gray path represents the navigation path, the dark-gray path represents the drivable road, and black represents the non-drivable road. Drivable roads may have lane markings on both sides, and intersections may have traffic lights.

[0047] S130, performing unsupervised training on the multidimensional feature vector to obtain multiple initial skills, wherein the unsupervised training includes training with information entropy constraints.

[0048] According to an embodiment of the present disclosure, for example, unsupervised training is performed on the multi-dimensional feature vector through steps S131 to S134 to obtain a plurality of initial skills.

[0049] Step S131: Selecting partial dimensional feature vectors from the multi-dimensional feature vectors.

[0050] For example, the multidimensional feature vectors extracted from a bird's-eye view image include features of the driving area, test vehicle, background vehicles, pedestrians, navigation road, lane markings, and traffic lights. Feature vectors of a subset of dimensions, such as those of the driving area and test vehicle, are selected for skill learning. Learning a single skill only requires feature vectors of a subset of dimensions, rather than all dimensions simultaneously. This reduces training complexity and facilitates unsupervised skill learning.

[0051] Step S132: obtaining a plurality of original skills based on the prior distribution of the feature vectors of some dimensions.

[0052] Figure 4 The flowchart of skill learning according to an embodiment of the present disclosure is schematically shown.

[0053] For example, Figure 4 As shown, set the skill number to z, and according to the prior distribution p(z) of the skill corresponding to the eigenvector of this part of the dimension, the corresponding original skill (also called skill strategy) π can be obtained. θ (a t |s t , z), where a t For action, s t Then, action a is executed in the simulated driving platform environment. t After that, the environment will feedback the reward r at time t+1 to the original skill. t+1 and state s t+1 , and generate a pseudo reward through the discriminator This is the probability value obtained by the softmax function. The autonomous driving algorithm learns skills with probability values ​​greater than a certain threshold, thereby obtaining multiple initial skills. The prior distribution can be a mean distribution or a Gaussian distribution.

[0054] For example, for reinforcement learning input state representations t This is a bird's-eye view created using backend data from CARLA (an open-source simulation simulator). It includes drivable paths, navigation paths, background vehicles, pedestrians, and the vehicle itself. Vehicles and pedestrians are added to the time series, and data from five time frames, t-16, t-12, t-8, t-4, and t-1, are extracted. T represents the current moment. These five frames are superimposed to form the state representation of the entire graph, serving as input to the transformer (also known as the self-attention model) encoder.

[0055] For example, for reinforcement learning action representation a t, there are only two parameters, namely the steering wheel angle and the throttle. The range of both parameters is between -1 and 1. When the throttle value is less than 0, it means that the car is braking.

[0056] For example, the design of rewards for environmental (env) feedback includes speed, position, rotation, path accuracy or deviation, collision (hitting people, hitting cars, and hitting other objects all have different rewards), blockage, and running red lights. It is necessary to design environmental rewards based on prior information (for example, based on traffic rules, such as giving negative rewards for running red lights and collisions) and multiple trials. Otherwise, the collision penalty will be too large, and other rewards will be too small, resulting in the car not moving at the end of training to get the most rewards (for example, if the termination condition is block, if it is 80 seconds, only speed is negative for 80 seconds, if |r speed *80s|<|r 碰撞 |, it will fall into a local optimum and cause the test vehicle to stop moving). Therefore, the reward design needs to be very carefully planned and requires multiple experiments to be designed correctly.

[0057] For example, the design of environmental rewards can be used to train an autonomous driving system implemented by the PPO algorithm as a baseline, and can also be used for subsequent fine-tuning training of skills.

[0058] Step S133: Train the feature vectors of some dimensions according to the information entropy constraint to obtain intrinsic pseudo rewards.

[0059] According to an embodiment of the present disclosure, a pseudo reward replaces the real reward given by the environment. The pseudo reward is endogenous and generated based on information entropy constraints.

[0060] For example, by using methods similar to the intrinsic curiosity module, information entropy can be used to drive the agent to learn skills. At the same time, different numbers of skills can be trained to cope with more complex environments.

[0061] Specifically, the pseudo reward is derived from information entropy, for example.

[0062]

[0063] Where, I(s;z)=H(z)-H(z|s), I(a;z|s)=H(a|s)-H(a|s,z), which can be substituted into formula (1) to obtain

[0064]

[0065] Among them, I(·;·) represents mutual information, H(·) represents information entropy, s represents state, z represents skill (basic action), a represents action (low-dimensional action), and F(θ) is to be maximized. According to Jensen's inequality, we can get Among them, H[a|s,z] is the regularized entropy, As an endogenous pseudo-reward, since p(z) is a constant, the pseudo-reward mainly follows Related. The first term I(s;z) represents the mutual information between s and z. This mutual information should be as large as possible, indicating that the distance between s and z is very close, that is, it is easy to know s when z is known. Skill z can determine which state s the agent is visiting. Similarly, after knowing state s, it is also easy to know which skill z the agent is performing in that state. The second term H(a|s) represents the information entropy when knowing state s and action a. Making this information entropy as large as possible means that the uncertainty of action a is higher, that is, the choice of action a is freer. The third term I(a;z|s) represents the mutual information between action a and skill z after knowing state s. Making this mutual information as small as possible means that after a given state s, the skill z cannot be determined by action a, nor can the specific action a be determined by skill z.

[0066] Furthermore, the first item states that different states correspond to different skills, making them distinguishable. The combination of the first and third items demonstrates that skill z can be determined by state s, rather than by action a. The second item, entropy maximization, aims to create more randomness and diversity in actions, encouraging the agent to explore as much as possible. Diversity and distinguishability are contradictory. If maximum distinguishability is desired, skills will be very far apart, resulting in a small number of skills. If the goal is to maximize the number of skills, skills will be very close, potentially leading to indistinguishability. Combining the first and third items effectively limits this. Therefore, based on information theory, the function to be optimized is defined according to the objective. This function is then transformed, and different optimization methods are used to determine the subsequent reward representation. By using information theory to constrain the agent's actions, entropy optimization methods are integrated into the rewards of reinforcement learning.

[0067] Step S134: Train multiple original skills based on the endogenous pseudo-rewards to obtain multiple initial skills.

[0068] According to the embodiments of the present disclosure, to ensure that all skill learning is done with speed, a speed reward can be added to the intrinsic pseudo-reward. Multiple original skills can then be trained based on the added pseudo-reward to obtain multiple initial skills. The speed reward includes: positive rewards for multiple original skills when speed is positive, and negative rewards for multiple original skills when speed is zero or negative. By adding speed rewards, the practicality of skill learning is improved, making it more suitable for autonomous driving.

[0069] For example, you can refer to the work of DIAYN (Diversity is All You Need), use the SAC (soft actor critic) method, and modify the reward to include a speed reward.

[0070] For example, the expected sum of rewards is The optimal strategy is expressed as follows,

[0071]

[0072] in,

[0073] r(τ t , a t )=r speed +r e (4)

[0074]

[0075]

[0076] According to embodiments of the present disclosure, to enhance the semantic information of skills, temporal sequence information can also be incorporated into the skill learning process. For example, this includes obtaining partial dimensional feature vectors for multiple consecutive frames, determining the relationship between the partial dimensional feature vectors for each frame based on the self-attention mechanism of the transformer to obtain temporal sequence information, and training multiple original skills based on the temporal sequence information and endogenous pseudo-rewards to obtain multiple initial skills.

[0077] For example, according to the formula

[0078] τ t =Encoder(s t , s t-1 ,...,s t-T+1 ) (7)

[0079] The relationship between different frame data τ can be learned t . Where T is the number of frames. s t , s t-1,...,s t-T+1 is a time series.

[0080] Figure 5 The following schematically shows a skill learning interface diagram according to an embodiment of the present disclosure.

[0081] According to the embodiments of the present disclosure, Figure 5 As shown, the skill learning interface is divided into three parts. The bird's-eye view on the left shows the current driving situation. The middle part is some background data used to monitor various data in real time. The right part is the value function of all skills. Since each skill has certain semantic information, by viewing the value of each skill at the current moment, the decision-making steps in autonomous driving can be more intuitively observed, making it easier to perform error correction, modification and interpretation of the intelligent system. For example, Figure 5 The current action selected is Skill 2. By visualizing Skill 2, we can determine that Skill 2 slightly steers the car to the right. The bird's-eye view indicates that the vehicle is currently positioned to the left of the lane line. This corresponds to the currently selected skill, indicating that the policy has been learned and interpreted correctly. If the two results do not match, it indicates a problem with the policy learning process, which can be corrected promptly.

[0082] S140, performing layered reinforcement training on the multi-dimensional feature vector to obtain a controller, where the layered reinforcement training includes adjusting the loss of the deep learning network.

[0083] According to an embodiment of the present disclosure, for example, hierarchical reinforcement training is performed on the multi-dimensional feature vector through steps S141 to S143 to obtain a controller.

[0084] Step S141 , superimposing multi-dimensional feature vectors of multiple consecutive frames to obtain a state representation.

[0085] According to an embodiment of the present disclosure, when training a controller, a state representation is obtained by superimposing feature vectors of the entire bird's-eye view as input data for a transformer.

[0086] Step S142: Determine the environmental reward of the simulated driving platform.

[0087] Step S143: adopting an action-value function learning method combined with a deep learning network, and training the state representation according to the environment reward to obtain a controller, wherein the deep learning network is set with a critic loss and a terminal loss.

[0088] Figure 6 The system framework of the hierarchical reinforcement learning-based autonomous driving method according to an embodiment of the present disclosure is schematically shown.

[0089] According to the embodiments of the present disclosure, Figure 6As shown, by learning multiple skills, high-dimensional actions, such as left and right lane changes, and lane keeping, are converted into low-dimensional actions, such as turning angles and speeds. Based on the high-dimensional action requirements, the controller controls the skill combination and outputs low-dimensional actions. During skill learning, the discriminator generates pseudo-rewards, which are used to learn the skills. During controller training (e.g., a meta-controller), the state representation is trained based on environmental rewards.

[0090] For example, the decision-making training part uses reinforcement learning-related content including PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), option-critic and other algorithms.

[0091] Specifically, for example, the meta controller is trained using a hierarchical reinforcement learning approach, and the option-critic framework is used to modify the terminal loss, actor loss, and critic loss. Since the underlying skills have been trained and the initial state of the underlying skill network has been determined, there is no need to use actor loss for training. Therefore, this disclosure simplifies actor-critic into a q-learning (action value function learning), using the DQN (Deep Q-Network, combined with the action value function learning of the deep learning network) method, and adds a terminal loss to reduce the complexity of the neural network. The td_loss in DQN is where y t is the target value function at time t,

[0092]

[0093] The π φ (a|τ t+1 ,ω t ) is a sub-strategy, i.e., a skill strategy. Since skill training has been completed, y t is known, and we can substitute td_loss into the above formula to get critic loss. The state value function obtained when skill ω is selected is U(τ t ,ω),

[0094] U(τ t ,ω)=(1-β μ (τ t ,ω))Q(τ t ,ω)+β μ V(τt ) (9)

[0095] The terminal loss can be obtained by finding the gradient of this function.

[0096] Figure 7 A total reward curve diagram of skill training using a relevant algorithm according to an embodiment of the present disclosure is schematically shown. Figure 8 A total reward curve diagram of skill training using an optimization algorithm according to an embodiment of the present disclosure is schematically shown.

[0097] According to the embodiments of the present disclosure, Figure 7 As shown in the figure, for example, when using the PPO algorithm to train skills, the PPO algorithm begins to converge after 3M and has not yet converged at 5M. Figure 8 As shown, using the optimization algorithm of the present disclosure to train skills, for example, the 30skill method has begun to converge after about 1M steps, and after fine-tuning the skills, even better results are achieved. It can be seen that the training speed of the PPO strategy is much slower than the technical solution of the present disclosure. Compared with the PPO algorithm, the technical solution of the present disclosure can solve the problem more quickly.

[0098] It should be noted that, under the same conditions, a greater number of skills yields greater total rewards, but also slower convergence. Because a small number of samples are collected at each data step, the curve itself exhibits some oscillation; the main focus is on the overall trend. While the world environment for skill training differs somewhat from that for task testing, the skills trained using this approach are universal. Training prior to 1M steps shows that even untuned skills can initially complete driving tasks.

[0099] S150, combining multiple initial skills through a controller to perform autonomous driving actions.

[0100] Since there is a difference between the skill learning environment and the real environment, in order to make up for this difference and make the skills more practical, after completing the controller training, the present invention can make some fine-tuning to the skills to better adapt to the new task scenario.

[0101] According to an embodiment of the present disclosure, after learning multiple initial skills, to improve the matching of the skills with the driving environment, the autonomous driving method based on hierarchical reinforcement learning, for example, further includes: supervised training of the multiple initial skills to obtain multiple optimized skills, wherein the supervised training includes training with environmental rewards, and the multiple optimized skills are combined by a controller accordingly to execute autonomous driving actions. The reward for fine-tuning the skills in a supervised manner in real-world scenarios is the environmental reward, and intrinsic rewards are also taken into account.

[0102] According to embodiments of the present disclosure, a controller can also score multiple initial skills to obtain the action value of each initial skill. By analyzing the action value of a skill, it is possible to make a certain judgment about the road conditions. For example, if the value of skill 20, which is the deceleration skill, suddenly increases at a certain moment, it indicates that there is a vehicle ahead, providing a certain degree of explanation for the autonomous driving algorithm.

[0103] In summary, the disclosed embodiments propose a hierarchical reinforcement learning-based autonomous driving method. This method trains multiple skills based on background data in a simulated driving platform environment. Then, a controller is trained to flexibly combine these skills based on driving needs to achieve autonomous driving. This method can train multiple general skills, resulting in objective and efficient skill training. The controller combines these skills to achieve autonomous driving, providing strong interpretability and enabling autonomous driving to cope with complex environments.

[0104] Figure 9 A block diagram of an autonomous driving device based on hierarchical reinforcement learning according to an embodiment of the present disclosure is shown.

[0105] The present disclosure also proposes an autonomous driving device 900 based on hierarchical reinforcement learning, such as Figure 9 As shown, for example, it includes: an acquisition module 910, an encoding module 920, a skill training module 930, a control training module 940 and a driving module 950.

[0106] The acquisition module 910 is used to acquire driving data, which includes background data of the simulated driving platform. In one embodiment, the acquisition module 910 can be used to perform the operation S110 described above, which will not be repeated here.

[0107] The encoding module 920 is configured to train an encoder based on the driving data to obtain a multi-dimensional feature vector. In one embodiment, the encoding module 920 may be configured to perform the operation S120 described above, which will not be described in detail herein.

[0108] Skill training module 930 is used to perform unsupervised training on the multidimensional feature vector to obtain multiple initial skills. The unsupervised training includes training with information entropy constraints. In one embodiment, skill training module 930 can be used to perform operation S130 described above, which will not be repeated here.

[0109] The control training module 940 is used to perform layered reinforcement training on the multi-dimensional feature vector to obtain a controller. The layered reinforcement training includes adjusting the loss of the deep learning network. In one embodiment, the control training module 940 can be used to perform the operation S140 described above, which will not be repeated here.

[0110] The driving module 950 is configured to combine multiple initial skills through a controller to perform an autonomous driving action. In one embodiment, the driving module 950 may be configured to perform the operation S150 described above, which will not be described in detail herein.

[0111] According to an embodiment of the present disclosure, any multiple modules among the acquisition module 910, the encoding module 920, the skill training module 930, the control training module 940 and the driving module 950 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 910, the encoding module 920, the skill training module 930, the control training module 940 and the driving module 950 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 910, encoding module 920, skill training module 930, control training module 940 and driving module 950 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0112] The details not included in the product embodiment section are similar to those in the method embodiment section. Please refer to the method embodiment section and will not be repeated here.

[0113] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in an exemplary order and are not intended to be limited to a specific order or hierarchy.

[0114] It should also be noted that directional terms such as "upper," "lower," "front," "back," "left," and "right" mentioned in the embodiments are merely references to the directions in the accompanying drawings and are not intended to limit the scope of protection of this disclosure. Throughout the drawings, identical elements are represented by identical or similar reference numerals. Conventional structures or configurations that may cause confusion in understanding this disclosure will be omitted. Furthermore, the shapes, sizes, and positional relationships of the components in the drawings do not reflect their actual sizes, proportions, or actual positional relationships.

[0115] In the foregoing detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that embodiments of the claimed subject matter require more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the disclosure comprises less than all features of any individual disclosed embodiment. The appended claims are therefore hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the disclosure.

[0116] In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. With respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", as explained in terms of "including" used as a transitional word in the claims. Any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0117] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above are only specific embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An autonomous driving method based on hierarchical reinforcement learning, characterized in that: include: Acquiring driving data, wherein the driving data includes background data of a simulated driving platform; Training an encoder based on the driving data to obtain a multi-dimensional feature vector; Performing unsupervised training on the multidimensional feature vector to obtain a plurality of initial skills, including: selecting a partial dimensional feature vector from the multidimensional feature vector; obtaining a plurality of original skills based on a prior distribution of the partial dimensional feature vector; training the partial dimensional feature vector based on an information entropy constraint to obtain an endogenous pseudo-reward; training the plurality of original skills based on the endogenous pseudo-reward to obtain the plurality of initial skills, wherein the unsupervised training includes training with an information entropy constraint; Performing layered reinforcement training on the multidimensional feature vector to obtain a controller, wherein the layered reinforcement training includes adjusting the loss of a deep learning network; The controller combines the multiple initial skills to perform autonomous driving actions.

2. The method according to claim 1, characterized in that The step of training an encoder according to the driving data to obtain a multi-dimensional feature vector comprises: fusing the background data based on the rules of the background data to obtain a bird's-eye view; The features of the bird's-eye view are extracted to obtain the multi-dimensional feature vector.

3. The method according to claim 2, characterized in that The driving data further includes image data and lidar data, and the step of training an encoder based on the driving data to obtain a multidimensional feature vector further includes: The image data and the lidar data are fused to obtain a bird's-eye view with supplementary information.

4. The method according to claim 1, wherein The training of the plurality of original skills according to the endogenous pseudo-reward to obtain the plurality of initial skills comprises: Adding speed rewards to the intrinsic pseudo rewards; Training the multiple original skills according to the intrinsic pseudo-reward after adding the speed reward to obtain the multiple initial skills; The speed bonus includes: When the speed is positive, positively reward the plurality of original skills; and When the speed is 0 or negative, negative rewards are given to the multiple original skills.

5. The method according to claim 1, wherein The performing unsupervised training on the multidimensional feature vector to obtain a plurality of initial skills further comprises: Obtaining the partial dimensional feature vectors of multiple consecutive frames; Based on the self-attention mechanism of the converter, the relationship between the feature vectors of the partial dimensions of each frame is determined to obtain the temporal sequence information; The multiple original skills are trained according to the time series information and the endogenous pseudo reward to obtain the multiple initial skills.

6. The method according to claim 1, characterized in that The multi-dimensional feature vector is subjected to layered reinforcement training to obtain a controller, comprising: superimposing the multidimensional feature vectors of multiple consecutive frames to obtain a state representation; determining an environmental reward for the simulated driving platform; The controller is obtained by adopting an action-value function learning method combined with a deep learning network and training the state representation according to the environmental reward; The deep learning network is provided with criticism loss and terminal loss.

7. The method according to claim 1, characterized in that Also includes: Performing supervised training on the multiple initial skills to obtain multiple optimized skills, wherein the supervised training includes training with environmental rewards; The controller combines the multiple optimization skills to perform autonomous driving actions.

8. The method according to claim 1, characterized in that Also includes: The controller scores the multiple initial skills to obtain an action value of each initial skill.

9. An autonomous driving device based on hierarchical reinforcement learning, characterized in that: include: An acquisition module, configured to acquire driving data, wherein the driving data includes background data of a simulated driving platform; an encoding module, configured to train an encoder based on the driving data to obtain a multi-dimensional feature vector; A skill training module is configured to perform unsupervised training on the multidimensional feature vector to obtain a plurality of initial skills, comprising: selecting a partial dimensional feature vector from the multidimensional feature vector; obtaining a plurality of original skills based on a prior distribution of the partial dimensional feature vector; training the partial dimensional feature vector based on an information entropy constraint to obtain an endogenous pseudo-reward; and training the plurality of original skills based on the endogenous pseudo-reward to obtain the plurality of initial skills, wherein the unsupervised training includes training with an information entropy constraint. A control training module, configured to perform layered reinforcement training on the multidimensional feature vector to obtain a controller, wherein the layered reinforcement training includes adjusting the loss of a deep learning network; A driving module is used to combine the multiple initial skills through the controller to perform autonomous driving actions.

Images