Trolley imitation learning method and device based on expert state, and storage medium

Abstract

The invention discloses a trolley imitation learning method based on expert state learning. Under the condition of not depending on external rewards and expert actions, firstly, a state, action and next state triple is collected, and an environment dynamic model is trained and frozen after convergence; and collecting a sequence only containing an expert state, and constructing a strategy network output action. Respectively acting the same action on an agent state and an expert state, obtaining two groups of predicted next states through a freezing model, and forming an error compensation signal according to the difference between the two states; and performing variance reduction by taking an expected difference given by the difference network as a reference, and minimizing a correction error signal on a track so as to update a strategy from end to end. The method is compatible with image or vector states and continuous or discrete actions, is suitable for being deployed on a universal computing platform, an edge device or a vehicle-mounted controller, and provides a corresponding computing device and a computer readable storage medium.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle decision-making and control technology, and relates to reinforcement learning and imitation learning. More specifically, it relates to a car imitation learning method based on expert state learning, a computing device, and a computer-readable storage medium. Background Technology

[0002] Data-driven vehicle decision-making and control are rapidly developing in simulation and real-world scenarios. Traditional imitation learning often uses expert-provided states and actions as supervisory signals for behavior cloning. This approach is simple to implement and has relatively fast training convergence. However, in complex and long-tailed conditions, it is difficult to safely and scalably collect expert actions. Inconsistent training and deployment distribution can easily lead to error accumulation, limiting the application of strategies in open roads and diverse environments.

[0003] To reduce reliance on expert actions, industry researchers are attempting to learn policies using demonstrations containing only expert state sequences and exploring generative prediction and distribution matching approaches. For example, inferring missing actions through inverse dynamics before cloning is significantly affected by discriminability and sensor noise; aligning policy-induced state or state transition distributions with expert distributions using adversarial discriminators is sensitive to game equilibrium, limiting stability and sample efficiency; using next-time observations or state predictions as intermediate supervision can weaken the dependence on action labeling, but if the prediction model has systematic biases, error signals may be amplified or directionally misleading, affecting policy quality and generalization ability.

[0004] Another important approach is to first learn a differentiable forward dynamics model of the environment, and then perform planning or policy optimization on that model. This type of method has the potential for end-to-end training and knowledge transfer, but it still faces several key challenges in practice: the model's bias towards environmental dynamics can easily accumulate during multi-step development, thus affecting the update direction obtained through backpropagation; without external rewards, directly using prediction error or distribution differences as optimization objectives often results in large gradient variance, early oscillations, and unstable convergence; in the absence of observed expert actions, it is necessary to construct training signals that are both action-related and differentiable to avoid degeneracy; simultaneously, it is also necessary to consider different state modes such as images and vectors, and different action spaces such as continuous and discrete, to meet real-time requirements on computationally limited platforms.

[0005] In summary, existing technologies still have significant gaps in the following aspects: reliance on expert actions leads to high costs and security risks, and is susceptible to distribution drift; state-only demonstration methods are still insufficient in terms of stability and identifiability; policy optimization based on world models is easily constrained by systematic biases and multi-step error accumulation; high variance and oscillations in training signals are common in the absence of external rewards; and supporting multimodal states and multiple types of actions while meeting engineering deployment requirements remains challenging.

[0006] Therefore, there is an urgent need for a method that can provide a stable gradient path through a differentiable forward model without using external rewards or expert actions, suppress model bias and reduce optimization variance by using well-designed error signals, and support multimodal states and multi-type action spaces while ensuring sample efficiency and training stability, thereby improving the feasibility and implementation of vehicle policy learning in complex and dynamic environments. Summary of the Invention

[0007] In complex and dynamic vehicle control environments, traditional imitation learning relies on expert actions for behavior cloning, which suffers from problems such as difficulty in designing reward functions, high cost and safety risks in expert action acquisition, instability and large variance in training based solely on state demonstration, and error accumulation in the world model during multi-step expansion. This invention aims to construct a differentiable and low-variance policy optimization path without external rewards or expert action constraints. This path suppresses the distortion of the learning direction by world model bias and adapts to image / vector states and continuous / discrete actions within a unified framework, meeting engineering deployment requirements.

[0008] This invention proposes a car imitation learning method based on expert state learning, and provides the corresponding system and storage medium. The method uses the dynamic environment model fθ as a differentiable link, employs error compensation (EC) to offset model bias, and uses a difference network (DN) to provide an expected difference benchmark to reduce optimization variance, thereby achieving end-to-end training without using rewards or expert actions. Its core process is as follows: S1. Interact with the environment in the vehicle control environment, collect triplets consisting of the current state, the executed action, and the next state, and store them in the interaction data buffer; train the environment dynamic model fθ based on this data, so that it can predict the next state given s_t and a_t. and with Supervised learning is performed on the loss, and the model parameters are frozen after convergence for subsequent differentiable inference and gradient propagation.

[0009] S2. Without recording expert actions, collect the expert driving state sequence e_t and construct an expert state buffer containing only the state.

[0010] S3. In the policy learning phase, a policy network πϕ is constructed to output actions or their distribution parameters based on the current state; continuous actions are sampled using differentiable reparameters (deterministic actions can be formed using the mean during the evaluation phase), and discrete actions are sampled using Gumbel-Softmax differentiable sampling.

[0011] S4. Construct the difference network DN in parallel, and give an estimate b_ψ(s_t) of the expected state difference caused by all possible actions in the current state at state s_t, as the variance benchmark.

[0012] S5. To offset the systematic bias of the world model, error compensation EC is introduced: In the same frozen state fθ, the same action a_t is applied to the agent state s_t and the expert state e_t respectively, resulting in two sets of predictions. and their differences As an instantaneous signal; a correction signal is constructed by subtracting from the reference. Minimize along the trajectory And update πϕ by backpropagating the gradient through the frozen differentiable path of fθ.

[0013] S6, Difference Measurement The default value is mean squared error, but cosine distance, Chamfer distance, or perceptual distance can be selected according to the state mode. To improve sample efficiency and stability, training strategies such as K-step multi-step expansion, importance weighting, learning rate warm-up, and gradient clipping can be combined.

[0014] S7. The corresponding system includes a processor and a memory, in which a computer program is stored. When the program is executed by the processor, it completes the above-mentioned data acquisition, environmental dynamic model training and freezing, error compensation and difference benchmark calculation, and policy network optimization and update.

[0015] S8. The program is stored on a computer-readable storage medium, and when the program is executed by a processor, it implements all or part of the steps of the method.

[0016] Beneficial effects Compared to behavior cloning that relies on expert actions or adversarial state-only demonstrations, this invention constructs an error compensation signal through a "dual prediction comparison of the same action on the same frozen fθ," which can explicitly offset world model bias and suppress multi-step error accumulation. By using a difference network to estimate the expected difference in the state domain as a benchmark, variance reduction and stable training are achieved, thus obtaining convergent and generalizable policies even without rewards or expert actions. The method is compatible with image and vector states, and continuous and discrete actions within an integrated framework, facilitating deployment on general computing platforms, edge devices, or vehicle controllers, reducing data acquisition and annotation costs, and improving robustness and sample efficiency in complex dynamic traffic scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings are now briefly described. The following drawings are used to assist in understanding the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0018] Figure 1 This is a schematic diagram of the task scenario.

[0019] Figure 2 The overall block diagram of the method shows the relationships between the environment, policy network (πϕ), error compensation (EC), difference network (DN), frozen environment dynamic model (fθ), and data buffer.

[0020] Figure 3 This is a schematic diagram of the structure and training process of the environmental dynamic model, including modules such as state feature extraction, action injection, temporal modeling and future state prediction, as well as the training process using mean squared error.

[0021] Figure 4 (a) Performance comparison curves in the first simulation environment; Figure 4 (b) shows the performance comparison curves in the second simulation environment.

[0022] Figure 5 This is the curve showing the results of the ablation experiment.

[0023] Figure 6 This is a graph comparing training time with average reward.

[0024] Figure 7 This is a schematic diagram of the running results in a real environment. Detailed Implementation

[0025] Terminology and Symbols Comparison Terminology and Symbols: Frozen Environment Dynamics Model fθ; Policy Network πϕ; Difference Network DN; Error Compensation EC; Interactive Data Buffer BuffEC; Expert State Buffer ExpECt BuffEC; Difference Measure The default value is mean squared error; agent state s_t; expert state e_t; action a_t; prediction of the next state ŝ_{t+1}; expectation operator E; environment dynamic model loss L_dyn; policy loss L_π; error compensation difference Δ_t; difference network baseline b_ψ(s_t); corrected optimization signal g_t; continuous action distribution parameters μ_ϕ(s), σ_ϕ(s); discrete action sampling temperature τ; Gumbel-Softmax (differentiable approximation sampling method for discrete actions).

[0026] Example 1 See Figures 1 to 3 This embodiment uses a car control task within a built-in autonomous driving simulation environment as an example. It learns strategies for stable driving in diverse road conditions without relying on external rewards or recording expert actions. The method flow is described in [link to method details]. Figure 2For the structure and training process of the environmental dynamic model, please refer to [link / reference]. Figure 3 .

[0027] S1 Data Acquisition and Buffer Construction: Interact with the environment in the vehicle control environment, collect samples consisting of the current state, executed actions, and the next state, and store them in the interactive data buffer in chronological order for model training and validation; S2 Environment Dynamic Model Construction: Based on interactive data, a differentiable environment dynamic model fθ is constructed to predict the next state given the current state and action. The model includes a state feature extraction module, an action injection module, a temporal modeling module, and a future state prediction module to support end-to-end differentiable reasoning. In one embodiment, the state feature extraction module uses multi-layer convolutional downsampling to obtain high-level spatial features for image-type states and multi-layer perceptron to obtain compact representations for vector-type states; the action injection module uses two learnable masks for modulation in the high-level feature space, one for element-wise multiplicative scaling and the other for element-wise additive biasing, to achieve feature fusion for action perception; the temporal modeling module uses a Transformer encoder to capture global dependencies, which can be replaced by a recurrent network or a temporal convolutional network when resources are limited; the future state prediction module uses upsampling or deconvolution to restore spatial resolution when images are input, and uses fully connected layers to map to the same-dimensional space of the vector state at the next time step when vectors are input. S3 Environment Dynamic Model Training and Freezing: Supervised learning is performed using the mean squared error between the predicted next state and the actual next state as the loss. After convergence, the model parameters are frozen and used only for forward inference in subsequent policy optimization to provide differentiable gradient paths. S4 Expert State Data Acquisition: Without recording expert actions, acquire the state sequence of the expert's driving process to form an expert state buffer containing only the state; the data can come from online interaction of expert strategies, or be obtained by parsing demonstration videos frame by frame and performing time alignment and jitter reduction processing. S5 Policy Network and Difference Network Construction: Construct a policy network to output actions or action distribution parameters based on the current state; construct a difference network to estimate the expected state difference of the action distribution in the current state and use it as a baseline to reduce optimization variance. S6 Error Compensation: At time step t, the policy network generates actions based on the agent's state. The same action is input into the same frozen environmental dynamic model along with the agent's state and the expert state, respectively, to obtain two sets of predicted next states. The difference between the two is used as an instantaneous error signal to offset the systematic bias of the model. Continuous actions can be sampled using a differentiable reparameterization method, and discrete actions can be approximated using the differentiable Gumbel-Softmax method. Deterministic actions can be used in the evaluation phase. S7 Differentiation Benchmark and Optimization Signal Construction: Input the current state into the difference network to obtain the desired difference benchmark, and subtract the benchmark value from the instantaneous error signal to form an optimization signal with lower variance; S8 Policy Optimization and Backpropagation. The policy network parameters are updated with the goal of minimizing the optimization signal accumulated along the trajectory, and end-to-end backpropagation is performed through the differentiable path of the frozen fθ. To improve sample efficiency and training stability, training strategies such as multi-step unfolding, importance weighting, learning rate warm-up, gradient pruning, and exponential moving average can be combined. The difference metric uses mean squared error by default, but cosine distance, Chamfer distance, or perceptual distance can also be used depending on the state mode.

[0028] In one embodiment, actions in the continuous action space are sampled in a reparameterized manner, while actions in the discrete action space are sampled using Gumbel-Softmax differentiability and annealed with temperature parameter τ. To suppress early oscillations and improve sample efficiency, learning rate warm-up and gradient clipping can be combined. In multi-step expansion, K is set to a rolling window of 1 to 10 and importance weighting is added. The difference network approximates the expected difference with a least-squares objective, and its update is alternated with the policy network at a fixed step size.

[0029] Example 2 See Figure 1 and Figures 3 to 7 This embodiment illustrates the experimental environment and evaluation process, and compares it with a baseline method using only expert states. It also conducts an ablation study of the difference network and error compensation. Performance convergence curves are shown in [reference needed]. Figure 4 (a) / Figure 4 (b) Ablation results are shown in [reference]. Figure 5 For a comparison of training time and average return, see [link to training time and average return]. Figure 6 For real-world examples, please refer to [link / reference]. Figure 7 .

[0030] Experimental environment In the MetaDrive environment, the state space uses LidarStateObservation with approximately 259 dimensions; the action space consists of continuous actions, including steering angle and speed, and can be configured with multiple road conditions for policy stability evaluation. In the Drift (CarlaEnv) environment, the state is derived from LiDAR observations with approximately 42 dimensions; the action space consists of two-dimensional continuous actions, including steering angle and speed.

[0031] Comparison Methods Under the unified setting of no external rewards and no expert actions, four state-only demonstration (action-free) methods are compared: BCO learns inverse dynamics models to supplement expert actions and then clones; GAIFO uses an adversarial discriminator matching strategy to induce distributions and expert distributions; MobILE combines forward dynamics and stable distance metrics (IPM / DDM) to replace adversarial learning; and DILO relies on a utility function stability extraction strategy.

[0032] Training and evaluation process All methods are trained under conditions of no external rewards and no expert actions, using a consistent training budget and evaluation protocol; evaluation is performed every 10 training iterations in MetaDrive and every 50 training iterations in Drift (CarlaEnv) (see [link]). Figure 4 (a) / Figure 4 (b) The primary metric is the average return (dimensionless), and the secondary metrics are the actual training time (minutes) and training stability (curve variance / oscillation). To ensure fair comparison, a unified data collection protocol and random seed are used, and the model width and depth are executed according to the public default or conventional settings. Ablation studies only change the components under study.

[0033] Results and Analysis See average return and training efficiency. Figure 4 (a) / Figure 4 (b) and Figure 6 Under a representative set of configurations, our method (fθ frozen, EC, DN frozen) achieved an average return of 95.73 on MetaDrive; the comparative methods were BCO: 32.93, GAIFO: 20.52, MobILE: 43.29, and DILO: 14.60. In terms of actual training time, our method took approximately 60 minutes; the comparative methods were BCO: 83 minutes, MobILE: 73 minutes, GAIFO: 223 minutes, and DILO: 156 minutes (see [link to training program]). Figure 6 For easy comparison, Table 1 presents a set of results under the same hardware and training budget settings.

[0034] Table 1. Comparison of training time and average reward (Time: minutes; Reward: dimensionless) Training process behavior Taking MetaDrive as an example, the reward fluctuates in the early stages of training, then gradually increases and tends to stabilize (see...). Figure 4 (b)).

[0035] Ablation Research See Figure 5The study compared four scenarios: "retaining EC and DN", "removing DN", "removing EC", and "removing both DN and EC". In a representative set of configurations, the average end-point returns were 76.13, 9.57, 88.13, and 60.27, respectively.

[0036] The results show that removing the difference network increases the gradient variance and causes significant curve fluctuations; removing the error compensation makes it difficult to offset model biases and results in more pronounced early oscillations; removing both at the same time leads to poor overall performance. Table 2 summarizes the relative performance of the four scenarios.

[0037] Table 2. Relative performance of ablation studies (returns: dimensionless) Training time comparison Under unified hardware and budget, this method results in a shorter wall clock training time, which facilitates reducing computing power costs and shortening the parameter tuning cycle (see [link]). Figure 6 ).

[0038] Real-world examples See Figure 7 It displays typical states and trajectory segments under working conditions such as turning and straight driving.

[0039] Example 3 This embodiment provides a computing device and a computer-readable storage medium for implementing the above method steps.

[0040] computing devices A computing device includes a processor and a memory; the memory stores a computer program that can run on the processor; when the program is executed by the processor, it causes the processor to perform all or part of the steps of the method described above, including interactive data acquisition and buffer construction, training and freezing of an environmental dynamic model, construction of an expert state dataset, establishment of a policy network and a difference network, calculation of error compensation, and construction of optimized signals and policy updates. The computing device can be a personal computer, a server, an edge computing unit, an in-vehicle controller, or other data processing equipment.

[0041] Computer-readable storage media A computer-readable storage medium storing a computer program thereon; when the program is executed by a processor, it implements all or part of the steps of the above-described method. The storage medium can be a USB flash drive, portable hard drive, read-only memory, random access memory, magnetic disk, optical disk, flash memory, solid-state drive, or other media capable of storing program code; the program can also be provided to a computing device via network transmission and executed after local storage. Any combination of the above-described hardware, software, and firmware implementations can be used; without departing from the basic idea of ​​this invention, equivalent substitutions, mergers, or splits of modules should be included within the scope of protection of this invention.

[0042] Industrial applicability The method and system described in this invention can be implemented on general computing platforms, edge devices, or vehicle controllers, and can be applied to online or offline training and deployment in autonomous driving scenarios such as park logistics vehicles and low-speed unmanned vehicles; this invention can be manufactured or used, and can be used in industry.

Claims

1. A car imitation learning method based on expert state learning, characterized in that, include: S1. Collect triplets consisting of the current state, action, and next state in the vehicle control environment to construct an interactive data buffer. S2. Construct and train an environmental dynamic model based on the triplet to predict the next state given the current state and action. After the training converges, freeze the parameters of the environmental dynamic model. S3. Collect the state sequence generated by expert driving without recording expert actions, and construct an expert state buffer that only contains expert states. S4. Establish a policy network for outputting actions based on the current state under the condition of no external reward and no expert action, and input the current state into the policy network to obtain the predicted action. S5. Input the current state and the predicted action into the frozen environmental dynamic model to obtain the agent's predicted next state; input the expert state and the predicted action into the frozen environmental dynamic model to obtain the expert's predicted next state, and obtain an instantaneous error signal based on the difference between the two. S6. Estimate the expected state difference caused by all possible actions in the current state through the difference network as a reference value, and subtract the reference value from the instantaneous error signal to obtain the error signal for optimization; S7. The policy network is trained with the goal of minimizing the error signal accumulated along the trajectory for optimization, and the gradient is backpropagated to the parameters of the policy network through the differentiable path of the frozen environment dynamic model to complete the update.

2. The method according to claim 1, characterized in that, The environmental dynamic model includes: a state feature extraction module, an action injection module, a temporal modeling module, and a future state prediction module; wherein, the state feature extraction module is used to convert the state in the form of an image or vector into a high-level feature representation; the action injection module is used to introduce action information into the high-level features; the temporal modeling module is used to perform temporal dependency modeling on the action perception features; and the future state prediction module is used to generate a state prediction for the next moment.

3. The method according to claim 2, characterized in that, The action injection module includes a first mask for performing element-wise multiplicative scaling on the high-level features and a second mask for performing element-wise additive bias on the high-level features, wherein both the first mask and the second mask are learnable parameters.

4. The method according to claim 2, characterized in that, The future state prediction module is adaptively implemented for different state types: when the input state is an image, upsampling or deconvolutional networks are used to restore spatial resolution; when the input state is a vector, fully connected layers are used to map to the vector state at the next time step.

5. The method according to claim 1, characterized in that, The training of the environmental dynamic model includes: using the mean squared error between the predicted next state of the environmental dynamic model based on the current state and the action output and the corresponding real next state in the interaction data as the loss function for supervised learning, until the convergence condition is met, and then freezing the parameters of the environmental dynamic model.

6. The method according to claim 1, characterized in that, The collection of expert states includes at least one of the following: collecting expert state sequences by loading expert model weights and interacting with the environment; collecting expert state sequences by parsing expert demonstration videos frame by frame and performing time alignment and decluttering; wherein, the expert state buffer does not store expert action data.

7. The method according to claim 1, characterized in that, The action generation of the policy network includes: for continuous action space, outputting the mean and variance of the action distribution and sampling it using a differentiable reparameterization method, and using the mean to form deterministic actions in the evaluation stage; for discrete action space, outputting the log probability of each action and using the Gumbel-Softmax approximation for differentiable sampling.

8. The method according to claim 1, characterized in that, The difference network is trained to approximate the expected state difference for all possible actions in the current state, and its output is used as a variance benchmark in the construction of the error signal for optimization. During policy optimization, the error signal for optimization is accumulated and minimized over a multi-step trajectory, and the end-to-end gradient backpropagation and update of the policy network parameters are achieved through the frozen environment dynamic model.

9. A computing device, characterized in that, The method includes a processor and a memory, wherein the memory stores a computer program, and the processor, when executing the computer program, is configured to perform all or part of the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to perform all or part of the steps of the method according to any one of claims 1 to 8.

Images