Autonomous driving reinforcement learning method and device fusing human driving prior data

Abstract

The application discloses an automatic driving reinforcement learning method and device fusing human driving prior data, and belongs to the technical field of automatic driving and reinforcement learning, and comprises the following steps: firstly, converting an offline driving data set or a human driving log into offline state transition samples consistent with automatic driving reinforcement learning, and writing the offline state transition samples into an offline prior experience pool; then, collecting online state transition samples in a closed-loop simulation environment or a controlled road test environment, and writing the online state transition samples into an online experience pool; and finally, jointly sampling from the offline prior experience pool and the online experience pool according to unified state space, action space and reward rules, and performing mixed training and updating on a strategy network and a value network. The application can fully utilize high-quality driving prior data, improve the sample efficiency, training stability and closed-loop performance of automatic driving reinforcement learning, and is suitable for automatic driving low-level control strategy and end-to-end strategy training and deployment integrated with control.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving and reinforcement learning technology, specifically relating to an autonomous driving reinforcement learning method and apparatus that integrates prior human driving data. Background Technology

[0002] The learning methods for autonomous driving planning and control strategies mainly include rule-driven methods, imitation learning methods, and reinforcement learning methods. Rule-driven methods rely on manually designed decision logic and perform stably in structured road environments, but they struggle to cover long-tail problems in complex traffic scenarios. Imitation learning methods learn basic driving behaviors using human driving data through supervised learning, but because the training phase typically uses open-loop supervision, error accumulation and distribution shifts are prone to occur during closed-loop deployment. Reinforcement learning methods directly optimize long-term rewards through closed-loop interaction between the agent and the environment. With reasonable reward settings, they can learn autonomous driving strategies with higher upper limits, making them particularly suitable for decision-making and planning in complex interaction scenarios.

[0003] However, traditional online reinforcement learning methods suffer from low sample efficiency in practical applications. Closed-loop training of autonomous driving strategies heavily relies on complex simulation environments or high-cost real-world testing environments. The strategy requires extensive interactions to obtain a sufficient number of effective samples, resulting in long training cycles, high computational resource consumption, and difficulty in quickly developing stable basic driving capabilities in the early stages of training. Meanwhile, the autonomous driving field has accumulated massive amounts of high-quality driving log data, including publicly available datasets and data collected by individual companies. This log data naturally records rich behavioral patterns of drivers in interactions with multiple traffic participants, adherence to traffic rules, and route selection, containing valuable prior knowledge and possessing the potential to serve as an initial source of experience for reinforcement learning.

[0004] Currently, mainstream methods for introducing offline data into online reinforcement learning processes are primarily geared towards general continuous control domains, such as robot operation or game agent training. Their assumptions and data structures differ significantly from those of autonomous driving tasks. Directly migrating these methods to autonomous driving scenarios often encounters the following problems: First, autonomous driving logs are mostly stored as time-series sensor data or trajectory point sequences. Their organization does not follow the "state, action, reward, termination flag" four-tuple structure required by reinforcement learning algorithms, lacking clear action labels and reward feedback, making it difficult to directly write log data into the experience replay pool for policy updates. Second, the coordinate system, velocity unit, steering angle definition, and observation space encoding method used in offline logs may be inconsistent with the state representation and action space caliber in the online simulation environment. Forced mixing of these elements during training introduces inconsistently distributed samples, leading to reduced policy performance. Third, autonomous driving tasks involve complex traffic rules, navigation constraints, and safety constraints. It is necessary to convert the trajectory information in the driving logs into reinforcement learning supervision signals that reflect path completion and violations in order to effectively apply them to the training objective.

[0005] In summary, existing methods are insufficient to fully utilize the prior knowledge in massive driving logs, and there is an urgent need for a reinforcement learning scheme that can effectively integrate offline data and online learning to solve the problems of data heterogeneity and distribution conflicts. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide an autonomous driving reinforcement learning method and apparatus that integrates prior human driving data. By converting offline driving datasets or human driving logs into offline state transition samples consistent with autonomous driving reinforcement learning, and realizing the hybrid sampling and updating of offline prior samples and online state transition samples under a unified state space, action space, and reward rules, this method fully utilizes prior human driving data while maintaining the closed-loop optimization capability of reinforcement learning, significantly improving the sample efficiency, initial stability, and final closed-loop performance of autonomous driving policy training. It is suitable for efficient training and safe deployment of autonomous driving policies in complex traffic scenarios.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides an autonomous driving reinforcement learning method that integrates prior human driving data, comprising the following steps: Acquire prior data including the vehicle's state, the states of surrounding traffic participants, road topology information, traffic rule information, and navigation path information. Generate continuous control actions through inverse kinematics solution and optimization correction. Generate reward and termination flags based on path completion and violation judgment. Based on the current state and next state constructed from the prior data, combine the continuous control actions, rewards, and termination flags to construct offline state transition samples that are consistent with the autonomous driving strategy in terms of state space, action space, and reward rules, and store them in the offline prior experience pool. The autonomous driving strategy to be trained is deployed in a closed-loop simulation environment or a controlled road test environment. Online state transition samples are collected according to the state space, action space and reward rules consistent with the offline prior experience pool and stored in the online experience pool. A hybrid training batch is obtained by jointly sampling from the offline prior experience pool and the online experience pool. This batch is used to jointly update the parameters of the policy network and the value network in reinforcement learning, resulting in an optimized autonomous driving policy. The optimized autonomous driving strategy is deployed on autonomous vehicles and outputs continuous control actions based on the current scene status.

[0008] Preferably, the step of generating continuous control actions through inverse kinematics solution and optimization correction includes: Using a vehicle kinematics model, the initial values ​​of actions at each moment are obtained by inverse solving based on the vehicle's position, heading angle, and velocity at two or more consecutive moments, which are consistent with the action space form of the reinforcement learning policy. With the goal of minimizing the deviation between the predicted state calculated based on the initial action value and the observed state at the corresponding time in the prior data, the initial action value at each time is optimized and corrected by a linear quadratic regulator or an iterative linear quadratic regulator to obtain the optimized action value at each time. The optimized action values ​​at each moment are arranged in chronological order to form a continuous control action sequence.

[0009] Preferably, generating rewards based on path completion includes: The path completion increment is calculated based on navigation path information at adjacent times. The path completion increment is used as the main body of positive reward. When soft constraint violations such as speeding or decreased comfort occur, a multiplicative penalty or weight reduction is applied to the path completion increment.

[0010] Preferably, generating a termination flag based on the violation determination includes: Based on road topology information and traffic rule information, violations are determined. When a collision, boundary crossing, or violation of traffic signals is determined, a termination sign is generated to terminate the current training round.

[0011] Preferably, the step of jointly sampling from the offline prior experience pool and the online experience pool to obtain the mixed training batch includes: According to a preset sampling ratio or an adaptive sampling ratio, samples are taken from the offline prior experience pool and the online experience pool respectively and merged into a mixed training batch. In the early stage of training, the sampling ratio of the offline prior experience pool is increased to accelerate the acquisition of basic driving capabilities by the strategy. In the middle and later stages of training, the sampling ratio of the offline prior experience pool is dynamically adjusted according to the closed-loop performance index of the strategy to enhance online exploration capabilities and the upper limit of the strategy.

[0012] Preferably, the joint parameter update of the policy network and value network in reinforcement learning is performed based on mixed training batches, including: The mixed training batches are input into the value network, and a normalization layer is used to suppress value overestimation. A multi-value network ensemble structure is used for value estimation to reduce training variance. The policy gradient of the policy network is updated based on the output of the value network, and the parameters of the policy network and the value network are updated synchronously.

[0013] Preferably, the offline state transition samples undergo quality screening or sample weighting before being stored in the offline prior experience pool, including: The sample quality score is determined based on at least one of the following: trajectory continuity, degree of compliance with traffic rules, consistency of navigation path, driving comfort, perception completeness, and scene difficulty. When storing samples in the offline prior experience pool, low-quality samples with quality scores below a preset threshold are removed, or the sampling probability of high-quality samples is increased during joint sampling, or the loss weight corresponding to high-quality samples is increased during joint parameter updates.

[0014] Secondly, embodiments of the present invention also provide an autonomous driving reinforcement learning device that integrates prior human driving data, for implementing the above-mentioned autonomous driving reinforcement learning method integrating prior human driving data, including: The offline prior data construction module is used to acquire prior data of human driving to generate continuous control actions, rewards and termination flags; based on the current state and the next state constructed from the prior data, and combined with the continuous control actions, rewards and termination flags, it constructs offline state transition samples that are consistent with the autonomous driving strategy in terms of state space, action space and reward rules, and stores them in the offline prior experience pool. The online interactive data construction module is used to deploy the autonomous driving strategy to be trained in a closed-loop simulation environment or a controlled road test environment, collect online state transition samples according to the state space, action space and reward rules consistent with the offline prior experience pool, and store them in the online experience pool. The hybrid training optimization module is used to obtain hybrid training batches by jointly sampling from the offline prior experience pool and the online experience pool. It is used to jointly update the parameters of the policy network and the value network in reinforcement learning to obtain the optimized autonomous driving policy. The strategy deployment output module is used to deploy the optimized autonomous driving strategy to the autonomous vehicle and output continuous control actions according to the current scene state.

[0015] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described method for autonomous driving reinforcement learning that integrates prior human driving data when executing the computer program.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the aforementioned autonomous driving reinforcement learning method that integrates prior human driving data.

[0017] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) This invention utilizes prior data from offline driving datasets or human driving logs to generate continuous control actions through inverse kinematics and optimization correction. It also generates reward and termination flags through path completion increments and violation determination, constructing offline state transition samples consistent with online reinforcement learning. This allows the large-scale, high-quality driving data already available in the autonomous driving field to directly serve reinforcement learning training, significantly alleviating the problem of low efficiency in purely online training samples. Furthermore, this invention is compatible with publicly available driving datasets, proprietary driving logs, and various closed-loop simulation platforms, demonstrating good versatility.

[0018] (2) By keeping the offline state transition samples and the online state transition samples consistent in terms of state space, action space and reward rules, the present invention reduces the distribution conflict between offline prior data and online interactive data, avoids training performance degradation due to inconsistencies in state representation, action definition or reward caliber, and thus improves the stability and effectiveness of hybrid training.

[0019] (3) By adopting a reward generation rule that uses the path completion increment as the positive reward subject and the violation event as the basis for termination and punishment, the present invention keeps the offline reward labeling process consistent with the online closed-loop optimization objective, avoiding the local optimum trap and heavy parameter tuning burden introduced by manually designing complex and dense reward functions.

[0020] (4) This invention uses a hybrid sampling mechanism with a preset sampling ratio or an adaptive sampling ratio to increase the sampling ratio of the offline prior experience pool in the early stage of training, so that the strategy can quickly form basic driving ability with the help of human driving prior. In the middle and later stages of training, the sampling ratio is dynamically adjusted according to the closed-loop performance index, and the proportion of the online experience pool is gradually increased to enhance the online exploration ability and hard example adaptability of the strategy. It can take into account the stability in the early stage of training and the performance limit in the later stage of training. It is suitable for the training and deployment of low-level control strategies for autonomous driving and end-to-end strategies integrated with control. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the autonomous driving reinforcement learning method that integrates prior human driving data, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the overall framework of the autonomous driving reinforcement learning method that integrates prior human driving data provided in this embodiment of the invention. Figure 3 This is a schematic diagram of the construction process of the offline state transition sample provided in the embodiment of the present invention; Figure 4 This is a schematic diagram of the hybrid sampling and update process of offline prior experience pool and online experience pool provided in the embodiments of the present invention; Figure 5 This is a schematic diagram of the autonomous driving strategy deployment process after training, provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of the structure of the autonomous driving reinforcement learning device that integrates prior human driving data provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0024] The inventive concept of this invention is as follows: Addressing the problems of heterogeneous formats between offline driving logs and reinforcement learning training data in existing technologies, the difficulty in effectively utilizing prior human driving knowledge, and the low sample efficiency and cold-start difficulties faced by purely online reinforcement learning, this invention provides an autonomous driving reinforcement learning method and apparatus that integrates prior human driving data. It generates continuous control actions through inverse kinematics and optimization correction, and generates rewards and termination flags through path completion calculation and violation determination, converting unlabeled trajectory data into offline state transition samples that can be directly used by the reinforcement learning strategy. By unifying the consistency of offline and online samples in state space, action space, and reward rules, it eliminates distribution conflicts between offline prior data and online interactive data. Based on this, a hybrid sampling mechanism is used to jointly utilize the offline prior experience pool and the online interactive experience pool for policy training. This allows the autonomous driving strategy to quickly form basic driving capabilities in the early stages with the help of prior human driving knowledge, and continuously improve its performance ceiling through online exploration in the mid-to-late stages, ultimately significantly improving the efficiency, stability, and closed-loop performance of autonomous driving strategy training.

[0025] like Figure 1 and Figure 2 As shown in the figure, the embodiment provides an autonomous driving reinforcement learning method that integrates prior human driving data, specifically including the following steps: S1. Acquire prior data including the vehicle's state, the states of surrounding traffic participants, road topology information, traffic rule information, and navigation path information. Generate continuous control actions through inverse kinematics solution and optimization correction. Generate rewards and termination flags based on path completion and violation judgment. Based on the current state and next state constructed from the prior data, combine the continuous control actions, rewards, and termination flags to construct offline state transition samples that are consistent with the autonomous driving strategy in terms of state space, action space, and reward rules, and store them in the offline prior experience pool.

[0026] In the embodiments, such as Figure 3 As shown, the construction of offline state transition samples specifically includes the following sub-steps: S1.1 performs scene slicing and time alignment on offline driving datasets or human driving logs to obtain driving segments of fixed or variable length. The vehicle state, surrounding traffic participant states, road topology information, traffic rule information, and navigation path information at each time step are extracted as prior data.

[0027] S1.2, the above prior data is converted into a scenario representation usable by the policy network, and states are constructed according to a preset time interval. State at the next moment The scene representation can employ bird's-eye view semantic representation, object-level scene representation, or a combination of both. If bird's-eye view semantic representation is used, road areas, navigation paths, lane lines, vehicles, pedestrians, and traffic light states can be rendered into a multi-channel semantic graph, and combined with scalar measurements such as vehicle speed, historical control values, and gear status, which are then input into the policy network. If object-level scene representation is used, the vehicle, surrounding traffic participants, and road elements are encoded into a vectorized object list. The above representation methods can be flexibly selected according to specific application scenarios and network architectures, and this invention does not impose any limitations on this.

[0028] S1.3, Based on the vehicle's position, heading angle, and velocity at consecutive time points, and combined with a simplified vehicle kinematics model, the human driving action labels are inversely derived. In this embodiment, the action space is preferably defined as longitudinal acceleration and steering angle. The following simplified vehicle kinematics model relationship is adopted: , , , , in, Indicates time The planar position, Indicates the heading angle. Indicates vehicle speed. Indicates the vehicle's wheelbase. Indicates longitudinal acceleration. Indicates the steering angle. Indicates the time interval between adjacent intervals.

[0029] Based on the above model, the initial values ​​of the actions are first calculated from the prior data: , in, The longitudinal acceleration in the initial value of the motion, The steering angle in the initial values ​​of the action. To prevent stable terms with a denominator of zero, the initial action values ​​are further optimized and corrected with the objective of minimizing the deviation between the predicted state and the log observation state. , in, , , , Based on the vehicle kinematics model and the current log status and candidate control actions The predicted state of the vehicle at the next moment. The next moment's vehicle status is recorded in the offline driving log. The longitudinal acceleration and steering angle are obtained after optimization and correction. and This is the weight matrix, with superscripts... This is a transpose. This optimization correction process can be implemented using a linear quadratic regulator (LQR) or an iterative linear quadratic regulator (iLQR) to obtain a continuous sequence of control actions consistent with the action space of the reinforcement learning policy.

[0030] In S1.4, during the offline reward generation phase, the path completion increment between adjacent time steps is calculated based on the navigation path information and used as the primary positive reward. Furthermore, if violations of soft constraints such as speeding, sudden acceleration / deceleration, or sharp turns are detected, a multiplicative penalty or weight decay is applied to the path completion increment. Simultaneously, violation events are determined based on road topology and traffic rule information: if major violations such as collisions, serious boundary crossings, or serious red-light violations are detected, that time step is marked as terminated, and a termination flag is generated. This design ensures that the reward generation in the offline phase is consistent with the reward criteria used in online reinforcement learning.

[0031] S1.5, the state corresponding to each time step. ,action ,award Next state and termination mark The offline state transition samples are generated and stored in the offline prior experience pool. .

[0032] Furthermore, offline state transition samples undergo quality screening or sample weighting before being stored in the offline prior experience pool. Specifically, a sample quality score is determined based on at least one of the following: trajectory continuity, traffic rule compliance, navigation path consistency, driving comfort, perception completeness, and scenario difficulty. Low-quality samples with quality scores below a preset threshold are removed during storage, or the sampling probability of high-quality samples is increased during joint sampling, or the loss weight corresponding to high-quality samples is increased during joint parameter updates. This approach can further improve the utilization efficiency of offline prior data.

[0033] S2 deploys the autonomous driving strategy to be trained in a closed-loop simulation environment or a controlled road test environment, collects online state transition samples according to the state space, action space and reward rules consistent with the offline prior experience pool, and stores them in the online experience pool.

[0034] In this embodiment, the autonomous driving policy to be trained is deployed in a closed-loop simulation environment, interacting with the environment using scene representations and action spaces consistent with offline state transition samples. The environment provides reward signals according to the same reward rules as in the offline data generation phase, and returns a termination flag upon collision, boundary crossing, or violation of traffic signals. Through the interaction between the policy and the environment, the state corresponding to each time step is... ,action ,award Next state and termination mark Offline state transition samples are generated and stored in the online experience pool. .

[0035] S3 obtains a mixed training batch by jointly sampling from the offline prior experience pool and the online experience pool. This batch is used to jointly update the parameters of the policy network and the value network in reinforcement learning, resulting in an optimized autonomous driving policy.

[0036] In the embodiments, such as Figure 4 As shown, during training updates, data is retrieved from the offline prior experience pool. and online experience pool The samples are collected according to a preset sampling ratio or an adaptive sampling ratio, with the proportions of offline state transition samples and online state transition samples being respectively... and and in proportion Form mixed training batches. Preferably, in the early stages of training, the proportion of samples transitioning from offline state is increased. The value is set to 0.6~0.9; in this embodiment, it is taken as... To improve sample efficiency during the cold start phase of the strategy, the strategy can quickly acquire basic following, steering, intersection crossing, and rule-following capabilities by leveraging prior human driving experience. Once the strategy has acquired basic driving capabilities, the proportion of offline state transition samples is gradually reduced and the proportion of online state transition samples is increased based on the strategy's success rate, collision rate, violation rate, path completion rate, or value function convergence in the closed-loop scenario. In this embodiment, the average path completion rate of the most recent 20 closed-loop test rounds is used as the adjustment basis: when the average path completion rate is below 20%, the offline sample proportion is 0.8; after reaching 20%, 40%, 60%, and 80%, the offline sample proportion is adjusted to 0.7, 0.6, 0.5, and 0.4, respectively, to enhance exploration capabilities and increase the strategy's upper limit. Preferably, each training batch includes samples from both the offline prior experience pool and the online experience pool.

[0037] In terms of network structure, the policy network can adopt the same or compatible model structure as existing autonomous driving reinforcement learning controllers, including a bird's-eye view encoding subnetwork, a scalar measurement encoding subnetwork, a feature fusion subnetwork, and an action output head, which directly outputs longitudinal acceleration and steering angle. The value network receives the state and action as input and uses a multi-value network ensemble structure for value estimation to reduce training variance. To suppress overestimation of value and training instability, a normalization layer structure can be introduced into the value network. The policy network and value network are updated jointly using a mixed training batch of offline and online samples: the mixed training batch is input into the value network, and the policy gradient is updated in the policy network based on the output of the value network, updating the parameters of the policy network and the value network simultaneously until training converges.

[0038] Furthermore, the sampling ratio of offline samples to online samples is not fixed, but dynamically adjusted according to performance metrics during training. When the loop closure success rate is lower than a preset threshold, the collision rate is higher than a preset threshold, or the path completion rate is at a low level, the sampling ratio of offline prior samples is increased to utilize human driving priors to constrain and guide the strategy. When the loop closure performance of the strategy reaches a preset level, the proportion of offline samples is reduced and the proportion of online samples is increased to enhance the strategy's autonomous exploration capability for difficult scenarios.

[0039] S4 deploys the optimized autonomous driving strategy to autonomous vehicles and outputs continuous control actions based on the current scene state.

[0040] In the embodiments, such as Figure 5 As shown, the trained autonomous driving strategy is deployed in the controller of the autonomous vehicle. The strategy network outputs continuous control actions such as longitudinal acceleration and steering angle in real time based on the current scene state, which the vehicle then executes. After the vehicle executes the actions, the vehicle state and the surrounding traffic environment change accordingly. Onboard sensors collect environmental perception data to update the current scene state and feed it back to the input layer of the strategy network, forming a closed-loop control loop of perception-decision-execution-feedback. Optionally, state transition samples generated during vehicle operation can be transmitted back and stored in an online experience pool as an online data source for subsequent offline-online hybrid training, used for continuous optimization and adaptive iteration of the strategy, enabling the strategy to continuously adapt to new driving scenarios and traffic modes in actual operation.

[0041] Based on the same inventive concept, such as Figure 6 As shown, this embodiment of the invention also provides an autonomous driving reinforcement learning device 600 that integrates prior human driving data, including: an offline prior data construction module 610, an online interactive data construction module 620, a hybrid training optimization module 630, and a policy deployment output module 640.

[0042] The offline prior data construction module 610 is used to acquire prior data of human driving to generate continuous control actions, rewards and termination flags; based on the current state and the next state constructed from the prior data, it combines the continuous control actions, rewards and termination flags to construct offline state transition samples that are consistent with the autonomous driving strategy in terms of state space, action space and reward rules, and stores them in the offline prior experience pool.

[0043] The online interactive data construction module 620 is used to deploy the autonomous driving strategy to be trained in a closed-loop simulation environment or a controlled road test environment, collect online state transition samples according to the state space, action space and reward rules consistent with the offline prior experience pool, and store them in the online experience pool.

[0044] The hybrid training optimization module 630 is used to obtain hybrid training batches by jointly sampling from the offline prior experience pool and the online experience pool, and is used to jointly update the parameters of the policy network and the value network in reinforcement learning to obtain the optimized autonomous driving policy.

[0045] The strategy deployment output module 640 is used to deploy the optimized autonomous driving strategy to the autonomous vehicle and output continuous control actions according to the current scene state.

[0046] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described autonomous driving reinforcement learning method that integrates prior human driving data when executing the computer program.

[0047] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for autonomous driving reinforcement learning that integrates prior human driving data.

[0048] It should be noted that the autonomous driving reinforcement learning device, electronic device, and computer-readable storage medium that integrate prior human driving data provided in the above embodiments all belong to the same inventive concept as the autonomous driving reinforcement learning method that integrates prior human driving data. For details of their specific implementation process, please refer to the embodiments of the autonomous driving reinforcement learning method that integrates prior human driving data, which will not be repeated here.

[0049] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A reinforcement learning method for autonomous driving that integrates prior human driving data, characterized in that, Includes the following steps: Acquire prior data including vehicle status, status of surrounding traffic participants, road topology information, traffic rule information, and navigation route information; The system generates continuous control actions through inverse kinematics and optimization correction, and generates rewards and termination flags based on path completion and violation judgment. This includes: using a vehicle kinematics model, obtaining initial action values ​​at each moment consistent with the action space form of the reinforcement learning strategy based on the vehicle's position, heading angle, and velocity at two or more consecutive moments; minimizing the deviation between the predicted state calculated from the initial action values ​​and the observed state at the corresponding moment in the prior data; optimizing and correcting the initial action values ​​at each moment using a linear quadratic regulator or an iterative linear quadratic regulator to obtain optimized action values ​​at each moment; constructing a continuous control action sequence by arranging the optimized action values ​​at each moment in chronological order; calculating the path completion increment between adjacent moments based on navigation path information, using the path completion increment as the main body of positive rewards; applying multiplicative penalties or weight decay to the path completion increment when soft constraint violations such as speeding or decreased comfort occur; and judging violation events based on road topology information and traffic rule information. When a collision, boundary crossing, or traffic signal violation is judged, a termination flag is generated to terminate the current training round. Based on the current state and the next state constructed from prior data, and combined with continuous control actions, rewards and termination flags, an offline state transition sample that is consistent with the autonomous driving strategy in terms of state space, action space and reward rules is constructed and stored in the offline prior experience pool. The autonomous driving strategy to be trained is deployed in a closed-loop simulation environment or a controlled road test environment. Online state transition samples are collected according to the state space, action space and reward rules consistent with the offline prior experience pool and stored in the online experience pool. A hybrid training batch is obtained by jointly sampling from the offline prior experience pool and the online experience pool. This batch is used to jointly update the parameters of the policy network and the value network in reinforcement learning, resulting in an optimized autonomous driving policy. The optimized autonomous driving strategy is deployed on autonomous vehicles and outputs continuous control actions based on the current scene status.

2. The autonomous driving reinforcement learning method that integrates prior human driving data according to claim 1, characterized in that, The process of obtaining a hybrid training batch by jointly sampling from the offline prior experience pool and the online experience pool includes: According to a preset sampling ratio or an adaptive sampling ratio, samples are taken from the offline prior experience pool and the online experience pool respectively and merged into a mixed training batch. In the early stage of training, the sampling ratio of the offline prior experience pool is increased to accelerate the acquisition of basic driving capabilities by the strategy. In the middle and later stages of training, the sampling ratio of the offline prior experience pool is dynamically adjusted according to the closed-loop performance index of the strategy to enhance online exploration capabilities and the upper limit of the strategy.

3. The autonomous driving reinforcement learning method that integrates prior human driving data according to claim 1, characterized in that, Joint parameter updates for the policy network and value network in reinforcement learning are performed based on mixed training batches, including: The mixed training batches are input into the value network, and a normalization layer is used to suppress value overestimation. A multi-value network ensemble structure is used for value estimation to reduce training variance. The policy gradient of the policy network is updated based on the output of the value network, and the parameters of the policy network and the value network are updated synchronously.

4. The autonomous driving reinforcement learning method that integrates prior human driving data according to claim 1, characterized in that, Offline state transition samples undergo quality screening or sample weighting before being stored in the offline prior experience pool, including: The sample quality score is determined based on at least one of the following: trajectory continuity, degree of compliance with traffic rules, consistency of navigation path, driving comfort, perception completeness, and scene difficulty. When storing samples in the offline prior experience pool, low-quality samples with quality scores below a preset threshold are removed, or the sampling probability of high-quality samples is increased during joint sampling, or the loss weight corresponding to high-quality samples is increased during joint parameter updates.

5. An autonomous driving reinforcement learning device that integrates prior human driving data, used to implement the method described in any one of claims 1 to 4, characterized in that, include: The offline prior data construction module is used to acquire prior data on human driving to generate continuous control actions, rewards, and termination flags; Based on the current state and the next state constructed from prior data, and combined with continuous control actions, rewards and termination flags, an offline state transition sample that is consistent with the autonomous driving strategy in terms of state space, action space and reward rules is constructed and stored in the offline prior experience pool. The online interactive data construction module is used to deploy the autonomous driving strategy to be trained in a closed-loop simulation environment or a controlled road test environment, collect online state transition samples according to the state space, action space and reward rules consistent with the offline prior experience pool, and store them in the online experience pool. The hybrid training optimization module is used to obtain hybrid training batches by jointly sampling from the offline prior experience pool and the online experience pool. It is used to jointly update the parameters of the policy network and the value network in reinforcement learning to obtain the optimized autonomous driving policy. The strategy deployment output module is used to deploy the optimized autonomous driving strategy to the autonomous vehicle and output continuous control actions according to the current scene state.

6. An electronic device comprising a memory and one or more processors, the memory being used to store a computer program, characterized in that, The processor is used to implement the autonomous driving reinforcement learning method according to any one of claims 1 to 4, which integrates prior human driving data, when executing a computer program.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the autonomous driving reinforcement learning method that integrates prior human driving data as described in any one of claims 1 to 4.

Images