A reward-free autonomous driving control method and system based on online distillation of experts

Abstract

The present application relates to a kind of based on expert online distillation no reward automatic driving control method and system, belong to automatic driving, imitation learning and model predictive control technical field.The method obtains expert data and the behavior data generated by vehicle online interaction, generates latent state-action representation by encoder;Using fixed random target mapping set, expert predictor and behavior predictor construct internal imitation signal, and train no decoder latent world model, Q network integration and Lipschitz regularization random strategy according to it;When online planning, Sobol low difference sample is generated in low-dimensional control point space, is developed into smooth candidate action sequence by B spline base matrix, action distribution is updated by latent space rolling, value evaluation and elite sample weighting, first action is executed and sequence is translated for next time hot start.The present application can improve the foresight of driving decision, sample utilization efficiency and the continuity of acceleration and steering control.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving, imitation learning, and model predictive control, and specifically to a rewardless autonomous driving control method and system based on expert online distillation. Background Technology

[0002] Autonomous vehicles need to continuously perform environmental perception, behavioral decision-making, and vehicle control in dynamic traffic environments, while simultaneously meeting requirements for safety, comfort, and generalization ability. Reinforcement learning can optimize driving strategies through the interaction between the agent and the environment, and is therefore widely used in autonomous driving tasks such as lane keeping, lane changing, ramp merging, and intersection navigation.

[0003] Existing reinforcement learning methods for autonomous driving typically require pre-designing reward functions for specific driving scenarios. For example, in lane keeping scenarios, the reward function usually includes evaluation items such as vehicle travel distance and lane departure degree; in lane changing scenarios, the reward function usually includes rewards for successful lane changes and penalties for collisions; in ramp merging or intersection passage scenarios, it is also necessary to further encode factors such as traffic interaction risks, collision risks, and passage time into the reward function.

[0004] However, autonomous driving objectives such as safety, driving efficiency, and passenger comfort have different physical meanings and evaluation scales, and there may be mutual constraints between these objectives, making the construction and weight adjustment of the artificial reward function extremely complex. An unreasonable reward function design can easily lead to the autonomous driving agent receiving high numerical rewards but exhibiting unexpected driving behavior.

[0005] Imitation learning utilizes expert demonstrations instead of manual rewards, providing an autonomous driving learning method that eliminates the need to directly design environmental reward functions. Existing offline imitation learning methods typically rely solely on pre-collected, fixed expert datasets to train driving policies, which can easily lead to covariate bias during closed-loop operation when the learner accesses states not covered by the expert data.

[0006] Online imitation learning allows learners to interact with the traffic environment and update driving strategies using online-generated behavioral data, which can mitigate covariate bias to some extent. However, existing online imitation learning methods primarily focus on single-step state transitions or state-action matching, making it difficult to adequately predict multi-step traffic state evolution.

[0007] Latent world models, by encoding environmental observations into a compact latent space and predicting future states within that space, enable imaginative scrolling and multi-step evaluation of candidate action sequences, thereby improving the foresight of autonomous driving decisions and the efficiency of online sample utilization.

[0008] However, existing latent world models and sampling-based model predictive control methods typically sample motion disturbances independently at each time step within the planning time domain. For continuous driving actions that include acceleration control and steering control, independent sampling at each time step can easily introduce high-frequency variations, causing acceleration and steering commands to jump between adjacent time steps, and may further cause speed fluctuations, vehicle heading oscillations, and a decrease in ride comfort.

[0009] Furthermore, typical neural network strategies do not explicitly constrain the local sensitivity of the mapping between latent states and driving actions. Small changes in the latent state can lead to large changes in the policy output, further increasing the discontinuity of autonomous driving actions.

[0010] Therefore, there is a need for an autonomous driving method that does not require manual design of environmental reward functions, can utilize expert data and online behavioral data to form stable imitation signals, can predict multi-step traffic state evolution, and can suppress high-frequency changes in continuous driving actions. Summary of the Invention

[0011] The purpose of this invention is to solve the problems of existing autonomous driving reinforcement learning methods, such as reliance on manual reward functions, the tendency of fixed expert data to produce covariate bias, the lack of long-term predictive ability of single-step imitation signals, and the tendency of continuous driving actions to cause high-frequency jumps. The invention provides a rewardless autonomous driving control method, system, and device based on online expert distillation.

[0012] This invention learns and imitates signals from expert data and online behavioral data, using these imitated signals as scalar bases for value learning and potential space planning, without the need for manually designed environmental reward functions; simultaneously, it constructs a decoder-free potential world model, predicting future traffic conditions through potential space imagination; furthermore, it sets smoothing constraints at the policy and planning layers respectively to improve the continuity of acceleration control and steering control quantities for autonomous driving.

[0013] To achieve the above objectives, the present invention provides a rewardless autonomous driving control method based on expert online distillation, comprising the following steps: Acquire an expert dataset and a behavior dataset formed by the interaction between the autonomous driving agent and the traffic environment. Both the expert dataset and the behavior dataset include observations, driving actions corresponding to the observations, and the next observation obtained after the driving action is performed. The driving actions include acceleration control variables and steering control variables.

[0014] The observations in the expert dataset and the behavior dataset are encoded into latent states using an encoder, and the latent states are concatenated with the corresponding driving actions to form a latent state-action representation.

[0015] Set up a random target mapping set consisting of multiple fixed random target mappings. Train an expert predictor and a behavior predictor using expert data and behavioral data respectively. Construct an intrinsic imitation signal based on the prediction deviation of the expert predictor and the behavior predictor relative to the random target mapping set.

[0016] The intrinsic imitation signal assigns a higher value to the potential state-action representation that conforms to the expert data distribution, while inhibiting the autonomous driving agent from staying in the state-action region that has been fully covered by behavioral data. Thus, it forms the scalar learning basis required for value learning and online planning without setting an artificial environment reward function.

[0017] A decoder-free latent world model is trained based on a behavioral dataset. The decoder-free latent world model includes an encoder, a latent state transition model, and a Q-network ensemble. The latent state transition model predicts the next latent state based on the current latent state and driving actions, while the Q-network ensemble estimates the latent state-action value based on intrinsic imitation signals.

[0018] By training a Lipschitz regularized stochastic policy using a Q-network ensemble, we can suppress abrupt changes in driving actions caused by small changes in the latent state by limiting the local sensitivity of the mapping from the latent state to the driving action.

[0019] During the online driving process of autonomous vehicles, low-difference control point samples are generated in the low-dimensional control point space, and the control point samples are expanded into time-related candidate driving action sequences in the planning time domain using the B-spline basis matrix constructed with the planning time domain.

[0020] The candidate driving action sequence is evaluated by using a latent state transition model to perform latent spatial imagination rolling, based on the intrinsic imitation signal obtained during the rolling process and the Q-network estimation value at the end of the rolling process.

[0021] Based on the evaluation results of the candidate driving action sequences, select elite candidate driving action sequences, update the mean and standard deviation of the driving action sequence distribution, execute the first driving action in the updated mean sequence, and then shift the updated mean sequence for the hot start of the next planning time.

[0022] The present invention also provides a rewardless autonomous driving control system based on expert online distillation, including a data acquisition and encoding module, an intrinsic imitation signal construction module, a latent world model module, a smoothing strategy module, a smoothing planning module, and a vehicle control module.

[0023] The data acquisition and encoding module is used to acquire expert datasets and behavior datasets, encode observations into latent states, and concatenate latent states with driving actions to form a latent state-action representation.

[0024] The intrinsic imitation signal construction module is used to calculate the expert prediction bias and the behavior prediction bias based on a fixed random target mapping set, an expert predictor, and a behavior predictor, and to generate an intrinsic imitation signal that does not depend on an artificial environmental reward function.

[0025] The latent world model module includes an encoder, a latent state transition model, and a Q-network ensemble, used to predict the evolution of traffic environment states in the latent space and to perform value estimation based on intrinsic imitation signals.

[0026] The smoothing strategy module is used to generate driving action strategies using a Lipschitz regularized stochastic strategy and to limit the local sensitivity of the mapping from potential states to driving actions.

[0027] The smoothing planning module is used to generate low-difference control point samples, expand the low-difference control point samples into candidate driving action sequences using B-spline basis matrices, evaluate the candidate driving action sequences using the latent world model, and update the distribution of driving action sequences based on the evaluation results.

[0028] The vehicle control module is used to execute the first driving action in the updated driving action sequence, and then shifts the updated driving action sequence to the smooth planning module to hot-start the driving action planning for the next planning moment.

[0029] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that can run on the processor, and when the processor executes the computer program, it implements the above-described rewardless automatic driving control method based on expert online distillation.

[0030] The beneficial effects of this invention are as follows: (1) Based on expert data and online behavioral data, an intrinsic simulation signal is constructed, eliminating the need to manually set environmental reward functions for different driving scenarios such as highways, ramp merging or intersections, thereby reducing the design difficulty of reward engineering.

[0031] (2) By sharing a fixed random target mapping set, the prediction biases of the expert predictor and the behavior predictor are made comparable. The intrinsic imitation signal constructed by the expert prediction bias and the behavior prediction bias can both attract the autonomous driving agent to approach the expert data distribution and inhibit the agent from staying too long in the area that has been fully covered by the behavior data.

[0032] (3) Use online behavioral data to continuously update the behavioral dataset so that the driving strategy can be trained according to the actual state of the vehicle, reducing the covariate bias that may be caused by relying solely on fixed expert data.

[0033] (4) Using a decoderless latent world model, environmental dynamics and state-action values ​​are learned directly in the latent space without the need to reconstruct the original observations, making the latent representation more focused on semantic information related to autonomous driving control.

[0034] (5) By using the latent state transition model for imaginary rolling, the multi-step traffic state evolution corresponding to candidate driving actions can be predicted with less reliance on real-world environmental interactions, thereby improving the efficiency of online sample utilization and the foresight of autonomous driving decisions. The latent world model provides long-term value guidance for rolling time-domain planning through dynamic prediction and value integration.

[0035] (6) A Lipschitz regularized stochastic strategy is adopted to normalize the policy output based on the Jacobian matrix of the policy network with respect to the latent state and to constrain the state-related scale, thereby reducing the possibility of sudden changes in driving actions caused by latent state disturbances from the policy level.

[0036] (7) Instead of sampling actions independently at each discrete moment in the planning time domain, this invention generates samples in a lower-dimensional control point space and then expands them into a complete candidate driving action sequence using a B-spline basis matrix. The same control point simultaneously affects adjacent planning steps, making adjacent acceleration control and steering control quantities time-dependent.

[0037] (8) By using B-spline parameterization to reduce the high-frequency degrees of freedom of the candidate action sequence, the first-order and second-order differences of the candidate driving actions can be reduced without the need to set additional artificial smoothing rewards in the candidate sequence scoring function. This invention uses the optimal action sequence obtained from the previous planning as the hot start sequence for the next planning, so that the continuity of actions is reflected not only between adjacent actions in a single planning, but also between adjacent rolling planning cycles.

[0038] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a general framework diagram of the rewardless autonomous driving control system based on expert online distillation of the present invention; Figure 2 This is a flowchart illustrating the overall process of the rewardless autonomous driving control method based on expert online distillation of the present invention. Figure 3 This is a flowchart illustrating the construction of the intrinsic analog signal of the present invention; Figure 4 This is a flowchart illustrating the training process of the decoder-free latent world model of this invention. Figure 5 This is a flowchart illustrating the construction process of the Lipschitz regularized random strategy of this invention. Figure 6 This is a flowchart of the smooth planning process based on B-spline structured low-difference sampling in this invention. Figure 7 This is a schematic diagram of the continuous rolling planning and action sequence hot start of the present invention. Detailed Implementation

[0040] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0041] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures, and should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0042] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0043] This implementation models autonomous driving as an online imitation learning problem. The autonomous driving agent obtains observations from the traffic environment, generates driving actions based on the current driving strategy, and obtains the next observation from the traffic environment.

[0044] Figure 1The overall framework shown includes a data section, a training section, an inference section, and a traffic environment section. Expert data and behavioral data are encoded as latent states. The training section includes a no-reward latent world model, a smoothing policy, and a distributed intrinsic imitation signal. The inference section includes a smoothing model predicting a path integral control planner and outputting acceleration control and steering control variables.

[0045] In this embodiment, the observation employs semantic observation related to driving control. The driving action is a continuous action, including acceleration control variables and steering control variables. The acceleration control variables and steering control variables can be represented using normalized numerical values.

[0046] Example 1: Overall Training and Online Control Process like Figure 2 As shown, a rewardless autonomous driving control method based on expert online distillation includes S101 to S107.

[0047] S101, Obtain the expert dataset and initialize the behavior dataset. .

[0048] Expert dataset This includes multiple expert driving transfer samples, each of which is represented as:

[0049] in, The expert vehicle is in t Observation of time, The expert vehicle is in t Driving actions performed at all times This indicates the next observation obtained after performing the driving action.

[0050] The autonomous driving agent interacts with the traffic environment, and the interaction transfer samples are added to the behavior dataset.

[0051] in, Indicates that the autonomous driving agent is in the first t Observations obtained at any time This refers to the driving actions performed by the autonomous driving agent. This indicates the next observation after the driving action is performed.

[0052] In the initial training phase, behavior transfer samples can be collected through preheating exploration; after the smooth planner has usable planning capabilities, behavior transfer samples can be collected through B-spline-based action planning.

[0053] S102, perform latent representation encoding on expert data and behavioral data.

[0054] Using encoder Observation o Encoding as latent states z :

[0055] potential state z and corresponding driving actions a By splicing the data together, a latent state-action representation is formed:

[0056] Expert samples and behavioral samples are encoded into the same latent state-action space using a shared encoder, enabling the distribution coverage of the two types of data to be compared in a unified representation space.

[0057] S103, constructs an internal analog signal.

[0058] S104, training a latent world model without a decoder and an ensemble of Q-networks.

[0059] S105, training a Lipschitz regularized stochastic policy.

[0060] S106 generates candidate control point disturbances in the low-dimensional control point space, and uses the B-spline basis matrix to expand the candidate control point disturbances into a sequence of candidate driving actions in the planning time domain.

[0061] S107 uses a decoderless latent world model to evaluate candidate driving action sequences, updates the driving action sequence distribution, executes the current driving action, and translates the action sequence to hot-start the next planning.

[0062] Example 2: Construction of Intrinsic Imitation Signal like Figure 3 As shown, the intrinsic analog signal construction module includes a fixed random target mapping set, an action predictor, and an expert predictor.

[0063] For latent state-action representation ,set up A fixed random target mapping:

[0064] in, Represents the potential state dimension. Indicates the dimension of driving actions. This represents the output dimension of the fixed random target mapping. The parameters of each fixed random target mapping remain fixed after initialization.

[0065] Calculate the first moment based on a fixed set of random target mappings:

[0066] Calculate the second moment:

[0067] Calculate the element-wise variance:

[0068] Among them, symbols This means squaring each element of the vector.

[0069] Set up a behavior predictor and expert predictors Behavior predictor Using behavioral datasets Training expert predictors Using expert datasets Conduct training.

[0070] for The prediction deviation is calculated using the following formula:

[0071] in, Indicates behavioral prediction bias. This indicates expert prediction bias; This is the balance coefficient between the first and second deviation terms; Predictor The first in the output vector One element; and Let the mean and variance of the random target mapping set be the first and second values, respectively. Each element.

[0072] The first bias term measures the squared distance between the predictor output and the mean of the fixed random target mapping set.

[0073] The second bias term is used to correct for data coverage. For latent state-action representations with insufficient coverage, the prediction bias has a relatively large value; for latent state-action representations that are frequently covered in the corresponding dataset, the prediction bias has a relatively small value.

[0074] The ideal latent state-action representation should have a small expert prediction bias and a large behavioral prediction bias; therefore, the intrinsic imitation signal is constructed according to the following formula:

[0075] in, Representing the potential state z and driving actions aThe corresponding intrinsic imitation signal; For signal scaling coefficients; This is the balance coefficient between the expert attraction term and the behavioral inhibition term.

[0076] For potential state-action representations that conform to the expert data distribution, the expert predictor exhibits smaller prediction bias; for potential state-action representations not yet fully covered by behavioral data, the behavioral predictor exhibits larger prediction bias. Therefore, the intrinsic imitation signal can encourage the autonomous driving agent to approach the expert distribution while reducing its excessive lingering within familiar behavioral areas.

[0077] When training the behavior predictor and the expert predictor, for each training sample, from the index set Uniformly extract target mapping index I Mapping fixed random targets The output of is used as the regression target for stopping the gradient.

[0078] The fitting loss of the behavior predictor and the expert predictor is:

[0079] in, This indicates the cessation of gradient operations. The intrinsic imitation signals derived from expert and behavioral data can serve as a learning basis for value learning and potential space planning.

[0080] Example 3: Decoder-less Latent World Model like Figure 4 As shown, the decoder-less latent world model includes an encoder. Potential state transition model Integration with Q network.

[0081] Sample trajectory segments with a length corresponding to the planning time domain from the behavior dataset or experience replay buffer. t + k The observation code for time is:

[0082] Simultaneously construct the target latent state that stops the gradient:

[0083] Use the current potential state as the initial state for potential rolling:

[0084] The latent state transition model predicts the next latent state recursively according to the following formula:

[0085] To align the latent states predicted by the latent state transition model with the latent states encoded by actual future observations, the following decoder-free consistency loss is adopted:

[0086] in, H Indicates the potential rolling length or the planning time domain length; Represents the time-domain weighting coefficients; k This represents a discrete moment in the planning time domain.

[0087] By performing a stopping gradient operation on the target latent state branch, unconstrained drift of both the predicted latent state and the target latent state is avoided.

[0088] Q network integration includes Two Q-evaluators are randomly selected from the target Q-network ensemble when constructing the reward-free temporal difference objective. and And take the smaller of the two values:

[0089] Sample the next driving action from a randomized strategy:

[0090] Construct a reward-free temporal difference objective based on the intrinsic imitation signal:

[0091] in, This represents the discount factor.

[0092] The training loss of the Q-network ensemble is expressed as:

[0093] in, This represents the regression loss between the Q evaluator output and the unrewarded time-series difference objective.

[0094] The overall training objective of the decoder-free latent world model is:

[0095] in, , and These are the weight coefficients corresponding to the potential dynamics loss, Q-network loss, and intrinsic imitation signal fitting loss, respectively.

[0096] During the random policy training phase, two Q evaluators are randomly selected from the Q network set, and the smaller value of the two evaluators' outputs is taken as the conservative Q estimate:

[0097] During the online planning phase, the average of the outputs of the two Q-evaluators is used as the final value of the candidate action sequence:

[0098] The strategy training phase employs a relatively conservative pairwise minimum Q estimation, which helps reduce value overestimation; the online planning phase employs pairwise average Q estimation, which enables candidate action sequences to obtain relatively less pessimistic end value evaluations.

[0099] Example 4: Lipschitz Regularized Stochastic Strategy like Figure 5 As shown, the random policy employs a Lipschitz policy network that has been normalized using Jacobian.

[0100] make Indicates the output of the unconstrained policy, let This represents the state-related scale generated by an auxiliary network with a Softplus output layer.

[0101] Output based on the unconstrained policy Regarding potential states z Jacobian matrix Construct a normalization strategy mapping:

[0102] in, This represents a positive number used to ensure the stability of normalization calculations; Let represent the 2-norm of the Jacobian matrix.

[0103] Obtain the mean and log-standard deviation of the Gaussian random policy based on the normalized policy mapping:

[0104] The driving motion is sampled and compressed using the hyperbolic tangent function according to the following formula:

[0105] in, This indicates that the system follows a pattern with a mean of zero and a covariance matrix equal to the identity matrix. I Gaussian random variables; This indicates element-wise multiplication.

[0106] Based on the latent states corresponding to the expert data, regularization is applied to the state-related scales to obtain:

[0107] in, This represents the potential state obtained from expert data.

[0108] The conservative Q estimate is normalized by running scale to obtain the normalized Q estimate. And define the single-step policy loss:

[0109] in, This represents the weighting coefficient of the stochastic policy entropy term.

[0110] For length of H The potential rolling, the policy training objective is:

[0111] in:

[0112] This represents the weight coefficients of the Lipschitz regularization loss.

[0113] Through the constraint state related scale It can suppress unnecessary excessive local changes in the mapping from latent state to driving action without the need to generate additional latent state perturbation samples.

[0114] Example 5: B-spline structured low-discrepancy programming like Figure 6 As shown, the smooth planning method based on B-spline structured low-difference sampling does not sample each driving action independently in the planning time domain. Instead, it generates disturbances in the low-dimensional control point space and then continuously expands the control point disturbances to the complete planning time domain.

[0115] The planning time domain includes H There are discrete driving actions, with the number of control points being... C The B-spline degree is p And satisfy:

[0116] Define the normalized programming time:

[0117] Construct on the interval [0,1] p Secondary clamping node vector:

[0118] The corresponding zero-order B-spline basis function is:

[0119] p The B-spline basis functions are obtained through the Cox-de Boor recurrence relation:

[0120] Terms with a denominator of zero are treated as zero, and in... The last basis function is set to 1.

[0121] Construct the B-spline basis matrix based on the B-spline basis function values ​​at each normalized programming time step:

[0122]

[0123] The B-spline basis matrix satisfies:

[0124] as well as:

[0125] Therefore, the motion disturbance at each discrete planning time is a convex combination of disturbances at adjacent control points. The same control point can affect multiple adjacent planning times, thus establishing the temporal correlation between candidate driving actions structurally.

[0126] 1. Example of a linear B-spline basis matrix When the planning time domain length H =3. Number of control points C =2, B-spline degree p When = 1, the linear B-spline basis matrix is:

[0127] At this point, the three driving action disturbances are generated sequentially from the first control point, the average of the two control points, and the second control point, which can avoid the independent jump of action disturbances at intermediate planning moments.

[0128] 2. Example of a quadratic B-spline basis matrix When the planning time domain length H =5. Number of control points C =3, B-spline degree p When = 2, the basis matrix of the quadratic B-spline is:

[0129] The matrix integrates three control points in each row, enabling a smooth transition of candidate driving actions from the initial control trend to the final control trend.

[0130] Let the three control points correspond to the normalized acceleration control quantity and the steering control quantity, respectively:

[0131] control point matrixP By expanding the quadratic B-spline basis matrix, the driving action sequence in the planning time domain is obtained:

[0132] The first column indicates that the acceleration control quantity increases smoothly from moderate acceleration and then gradually decreases. The second column indicates that the steering control quantity transitions continuously from negative steering to positive steering. There are no abrupt changes caused by independent sampling at each adjacent planning time.

[0133] 3. Sampling of low-difference control points make represents the mean and standard deviation of the distribution of candidate driving action sequences, respectively.

[0134] For the n From the candidate driving action sequences, first generate Sobol low-discrepancy samples:

[0135] The Sobol low-discrepancy samples are transformed into standard normal control point samples using the inverse cumulative distribution function of the standard normal distribution.

[0136] The standard normal control point samples are reconstructed into a control point noise matrix:

[0137] When driving actions include acceleration control and steering control, the driving action dimension The two columns of the control point noise matrix represent the acceleration control point noise and the steering control point noise, respectively.

[0138] The control point noise is expanded into planning time-domain noise using the B-spline basis matrix:

[0139] Candidate driving action sequences are generated based on the mean and standard deviation of the driving action sequence distribution:

[0140] in, This indicates that the acceleration control and steering control values ​​are limited to the normalized action range [-1, 1].

[0141] Each candidate driving action sequence contains only Each independent control point sample, rather than including Each time-series independent sample can reduce the search dimensionality and the high-frequency degrees of freedom in candidate driving action sequences.

[0142] make and Let represent the first-order difference operator and the second-order difference operator along the planning time domain, respectively. Then:

[0143]

[0144] The rate of change of candidate driving actions and the change of the rate of change are jointly determined by the difference between adjacent B-spline basis functions, rather than directly inheriting time-by-time independent noise.

[0145] The higher the degree of the B-spline, the more gently the basis matrix changes along the planning time domain, which can further reduce the high-frequency components in the acceleration and steering control variables. This smoothing effect is generated by the parameterization of the candidate action sequence, without the need to add an additional manual smoothing reward to the planning evaluation function.

[0146] Example 6: Candidate Action Evaluation and Distribution Update For each candidate driving action sequence From the current potential state We begin by imagining rolling using a latent state transition model.

[0147] In the k At each planning moment, candidate driving actions will be considered. Input the latent state transition model to obtain the next predicted latent state:

[0148] The candidate driving action sequences are evaluated as follows:

[0149] in:

[0150] The first part of the evaluation function is the sum of the discounted intrinsic imitation signals corresponding to each predicted potential state-action representation in the planning time domain, and the second part is the average value of the Q-network at the end of the planning process.

[0151] An elite set was selected from all candidate driving action sequences based on the evaluation scores. And determine the maximum evaluation value in the elite set:

[0152] The normalized weights for the elite candidate driving action sequences are:

[0153] in, The temperature coefficient represents the weight of the evaluation value.

[0154] Update the mean of the candidate driving action sequence distribution based on the normalized weights:

[0155] Update the standard deviation of the candidate driving action sequence distribution:

[0156] in, and These represent the lower and upper limits of the standard deviation, respectively.

[0157] Within a planning cycle, candidate action sampling, potential space rolling, elite candidate screening, and mean and standard deviation updates can be performed repeatedly a preset number of times.

[0158] After the update is complete, the mean sequence will be... M The first driving action As the action to be performed at the current planning moment:

[0159] The vehicle control module controls the autonomous vehicle based on the acceleration control and steering control quantities during the execution of actions.

[0160] Example 7: Action Sequence Translation Warm Start like Figure 7 As shown, in the continuous rolling planning process, the mean sequence obtained from the previous planning is shifted forward by one discrete time along the planning time domain, and the shifted sequence is used as the initial mean for the next planning time.

[0161] The first driving action already executed in the previous planning is removed from the mean sequence, and the remaining driving actions are moved forward; for the end gaps created after the shift, the end control trend of the previous mean sequence is used for initialization.

[0162] By shifting the action sequence, the B-spline time structure formed in the previous planning can be transferred to the next planning cycle.

[0163] Within a single planning cycle, the B-spline basis matrix constrains adjacent driving actions by sharing control points; between adjacent planning cycles, the action sequence translation and hot start maintain control consistency during continuous replanning.

[0164] Furthermore, the Lipschitz regularized stochastic strategy will be used. The sampled driving action trajectories are added to the candidate action sequence set as policy priors, so that the training results of stochastic policies can be connected with online model predictive control planning.

[0165] Example 8: Model Training Process The training process for a rewardless autonomous driving control model based on expert online distillation includes: Initialize the behavior dataset, the target Q-network, and the action sequence distribution statistics of the smooth planner.

[0166] In each environmental interaction step, a behavior transfer sample is obtained through preheating exploration or smooth action planning based on B-splines, and the behavior transfer sample is stored in the behavior dataset.

[0167] In each gradient update step, expert sample batches and behavior sample batches are sampled from the expert dataset and behavior dataset, respectively.

[0168] The encoder is used to generate expert latent states and behavioral latent states, and the latent state transition model is used to generate latent space imaginary scrolling.

[0169] Based on behavioral prediction bias and expert prediction bias, an intrinsic imitation signal is constructed, and reward-free temporal difference objective, latent dynamic loss, Q-network loss, predictor fitting loss, and stochastic policy loss are constructed.

[0170] The encoder, latent state transition model, behavior predictor, expert predictor, Q-network ensemble, Lipschitz regularized stochastic policy, and corresponding auxiliary network are updated based on the corresponding loss.

[0171] Perform a soft update on the target Q network.

[0172] Repeatedly perform environment interaction and gradient updates until the preset training termination condition is met to obtain the trained decoderless latent world model, intrinsic imitation signal construction module, Q network ensemble, and Lipschitz regularized stochastic policy.

[0173] Example 9: Online Inference Process During the vehicle's online operation t At each planning moment, obtain the current observation. Previous planning mean series Planning time domain length H Number of control points C B-spline degree p and the number of planning iterations .

[0174] Obtain the current potential state using the encoder:

[0175] Construct the B-spline basis matrix based on the planning time domain length, the number of control points, and the degree of B-splines. .

[0176] Initialize the mean during the initial planning. Mand standard deviation In subsequent planning, the mean sequence of the previous planning will be used. The result after translation is the current mean of the hot start. M .

[0177] The policy prior driving action trajectory is sampled from the Lipschitz regularized random policy, and the policy prior driving action trajectory is added to the candidate action sequence set.

[0178] In each planning iteration, Sobol low-discrepancy control point samples are generated. The control point samples are expanded into a smooth driving action sequence using the B-spline basis matrix, and the potential state transition model is used to perform potential space rolling prediction.

[0179] The evaluation value of each candidate driving action sequence is calculated based on the intrinsic imitation signal and the Q-value of the planning terminal. Elite candidate driving action sequences are selected, and the mean is updated. M and standard deviation .

[0180] After completing a preset number of planning iterations, the first driving action in the mean sequence is executed, and the updated mean sequence is saved for the next planning hot start.

[0181] Example 10: Automated Driving System This embodiment provides a rewardless autonomous driving control system based on expert online distillation, including a data acquisition and encoding module, an intrinsic imitation signal construction module, a latent world model module, a smoothing strategy module, a smoothing planning module, and a vehicle control module.

[0182] The data acquisition and encoding module acquires expert datasets and online behavioral datasets, and calls the encoder to encode expert observations and behavioral observations into latent states.

[0183] The intrinsic imitation signal construction module includes a fixed random target mapping set, a behavior predictor, and an expert predictor, which outputs an intrinsic imitation signal based on the behavior prediction bias and the expert prediction bias.

[0184] The latent world model module includes an encoder, a latent state transition model, and a Q-network ensemble for performing latent state prediction, latent dynamic alignment, and latent state-action value estimation.

[0185] The smoothing strategy module includes an unconstrained policy network, a state-dependent scale auxiliary network, and a random action sampling unit, which are used to generate driving strategies with local sensitivity constraints.

[0186] The smoothing planning module includes a control point sampling unit, a B-spline basis matrix construction unit, a candidate action sequence generation unit, a potential rolling unit, a candidate action evaluation unit, and an action distribution update unit.

[0187] The vehicle control module reads the first acceleration control value and the first steering control value from the updated mean sequence, and controls the vehicle based on the acceleration control value and the steering control value.

[0188] The system can be applied to continuous control driving scenarios such as highway driving, ramp merging, and intersection passage.

[0189] Example 11: Electronic Equipment This embodiment provides an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor.

[0190] When the processor executes the computer program, it acquires expert data and behavioral data, encodes latent states, constructs intrinsic imitation signals, trains latent world models, trains Lipschitz regularization policies, generates B-spline candidate actions, evaluates the latent rolling performance of candidate actions, and controls vehicle actions.

[0191] The processor can be a processing unit with data processing capabilities, and the memory is used to store expert data, behavioral data, model parameters, candidate action sequence distribution parameters, and computer programs.

[0192] This invention can also be implemented using a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described rewardless autonomous driving control method based on expert online distillation.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A rewardless automatic driving control method based on expert online distillation, characterized in that, include: Acquire an expert dataset and a behavior dataset formed by the interaction between the autonomous driving agent and the traffic environment. Both the expert dataset and the behavior dataset include observations, driving actions corresponding to the observations, and the next observation obtained after executing the driving actions. The driving actions include acceleration control variables and steering control variables. The observations in the expert dataset and the behavior dataset are encoded into latent states using an encoder. The latent states are then concatenated with the corresponding driving actions to obtain a latent state-action representation. Set up a random target mapping set consisting of multiple fixed random target mappings. Train an expert predictor and a behavior predictor based on the expert dataset and the behavior dataset, respectively. Based on the prediction deviation of the expert predictor and the behavior predictor relative to the random target mapping set, construct an intrinsic imitation signal that does not depend on the artificial environment reward function, so that the latent state-action representation that conforms to the expert data distribution and is not fully covered by the behavior data has a high intrinsic imitation signal value. A decoder-free latent world model is trained based on the behavior dataset. The decoder-free latent world model includes the encoder, a latent state transition model, and a Q-network ensemble. The latent state transition model is used to predict the next latent state based on the current latent state and driving action. The Q-network ensemble is used to estimate the value of the latent state-action representation based on the intrinsic imitation signal. Based on the Q-network ensemble training of the Lipschitz regularized stochastic policy, the sudden changes in the driving action are suppressed by limiting the local sensitivity of the mapping from the latent state to the driving action to changes in the latent state. During the vehicle's online driving process, low-difference control point samples are generated in the low-dimensional control point space. The low-difference control point samples are expanded into time-related candidate driving action sequences within the planning time domain using the B-spline basis matrix constructed with the planning time domain. The latent state transition model is used to perform latent space imaginary rolling on each candidate driving action sequence. Each candidate driving action sequence is evaluated based on the intrinsic imitation signal during the rolling process and the Q-network estimation value at the end of the rolling process. The driving action sequence distribution is updated based on the evaluation results. The first driving action in the updated driving action sequence is executed, and the updated driving action sequence is translated for hot start at the next planning time.

2. The rewardless automatic driving control method based on expert online distillation according to claim 1, characterized in that, The intrinsic imitation signal is constructed in the following manner: Let the potential state-action representation be: in, z As a potential state, a For driving actions; The random target mapping set includes A fixed random target mapping The first moment, second moment, and element-wise variance are calculated based on the fixed random target mapping: for The prediction deviation is calculated using the following formula: in, For the behavior predictor, For the expert predictor, To fix the output dimension of random target mapping, This is the balance coefficient between the two prediction bias terms; The intrinsic imitation signal is constructed according to the following formula: in, For signal scaling coefficients, This is the balance coefficient between the expert attraction term and the behavioral inhibition term.

3. The rewardless automatic driving control method based on expert online distillation according to claim 2, characterized in that, When training the behavior predictor and the expert predictor, target mapping indices are uniformly drawn from the index set corresponding to the random target mapping set. I The output corresponding to the fixed random target mapping is used as the regression target for stopping the gradient, and training is performed according to the following loss function: in, For the aforementioned behavior dataset, For the expert dataset, This indicates that the gradient operation is stopped.

4. The rewardless automatic driving control method based on expert online distillation according to claim 1, characterized in that, Training the latent state transition model includes: The first segment of the trajectory Observation of time Encoding as latent states: And copy the stopping gradient of the potential state: As the potential state of the target; by Given the initial potential state, the predicted potential state is obtained recursively through the potential state transition model: The following decoder-free consistency loss is used to align the predicted latent state with the target latent state in the planning time domain: in, For the encoder, For the potential state transition model, H To plan the time domain length, These are the time-domain weighting coefficients.

5. The rewardless automatic driving control method based on expert online distillation according to claim 4, characterized in that, Training the Q-network ensemble includes: Two evaluators are extracted from the target Q-network ensemble, and the smaller value output by the two evaluators is taken to obtain the conservative target value: Construct a reward-free temporal difference target based on the intrinsic imitation signal: in: Each evaluator in the Q-network ensemble regresses the reward-free temporal difference objective, and the latent dynamics loss, Q-network loss, and intrinsic imitation signal loss are weighted and combined to obtain the training objective of the decoder-free latent world model: in, As a discount factor, For the random strategy, , and These are the weighting coefficients for the corresponding loss terms.

6. The rewardless automatic driving control method based on expert online distillation according to claim 5, characterized in that, The Lipschitz regularized stochastic strategy is constructed as follows: make For the output of an unconstrained policy, let To normalize the unconstrained policy output, which is generated by an auxiliary network with a Softplus output layer to a state-related scale, the Jacobian matrix of the unconstrained policy output with respect to the latent state is used as follows: The mean and log-standard deviation of the Gaussian strategy are obtained from the output of the normalization strategy: And generate the driving motion compressed by hyperbolic tangent according to the following formula: Based on the latent states corresponding to the expert data, the relevant scales of the states are regularized: The regularization loss of the state-related scale is then added to the training objective of the random policy.

7. The rewardless automatic driving control method based on expert online distillation according to claim 1, characterized in that, Expanding the low-difference control point samples using the B-spline basis matrix includes: The planning time domain includes H There are discrete driving actions, with the number of control points being... C The B-spline degree is p And satisfy: The first k The discrete programming time-time normalization is as follows: Construct on the interval [0,1] p The B-spline basis functions are obtained by using the second-order clamping node vector and the Cox-de Boor recursive method. And construct the B-spline basis matrix: In this context, the elements in the B-spline basis matrix are non-negative, and the sum of the basis function values ​​corresponding to the same discrete programming time is 1, so that the action disturbance at each discrete programming time is generated by a convex combination of multiple adjacent control points.

8. The rewardless automatic driving control method based on expert online distillation according to claim 7, characterized in that, Generating, evaluating, and updating the candidate driving action sequences includes: Generation dimension is Sobol low difference samples It is converted into control point noise using the inverse cumulative distribution function of the standard normal distribution: The control point noise is expanded into planning time-domain noise using the B-spline basis matrix: Based on the mean of the distribution of driving action sequences M and standard deviation , obtained the n Candidate driving action sequences: The candidate driving action sequence is evaluated according to the following evaluation function: in, This represents the average of the outputs of the two evaluators extracted from the Q-network ensemble. Elite candidate driving action sequences are selected based on the evaluation values. The normalized weights of each elite candidate driving action sequence are calculated, and the mean is updated based on these normalized weights. M and standard deviation The updated mean M The first driving action is taken as the action to be executed at the current planned moment, and the updated mean is used. M The mean value is shifted along the planning time domain and used as the initial mean value for the next planning time.

9. A rewardless autonomous driving control system based on expert online distillation, characterized in that, include: The data acquisition and encoding module is used to acquire expert datasets and behavioral datasets, encode observations into latent states, and concatenate latent states with driving actions to form a latent state-action representation. The intrinsic imitation signal construction module is used to calculate the behavior prediction bias and expert prediction bias based on a shared set of fixed random target maps, behavior predictors and expert predictors, and generate an intrinsic imitation signal that does not depend on the artificial environment reward function. The latent world model module, including an encoder, a latent state transition model, and a Q-network ensemble, is used to predict the evolution of traffic environment states in the latent space and to perform value estimation based on the intrinsic imitation signal. The smoothing strategy module is used to generate driving action policies using Lipschitz regularized stochastic policies and to limit the local sensitivity of the mapping from latent states to driving actions. The smoothing planning module is used to generate low-difference control point samples in a low-dimensional control point space, expand the low-difference control point samples into time-related candidate driving action sequences in the planning time domain using the B-spline basis matrix, evaluate the candidate driving action sequences using the latent world model, and update the driving action sequence distribution based on the evaluation results. as well as The vehicle control module is used to execute the first driving action in the updated driving action sequence, and to provide the updated driving action sequence to the smooth planning module after translation, so as to hot-start the driving action planning for the next planning moment.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that can run on the processor, and when the processor executes the computer program, it implements the rewardless autonomous driving control method based on expert online distillation as described in any one of claims 1 to 8.

Images