An imitation learning method and device based on flow matching and dynamic reward scheduling

Abstract

The present application relates to deep learning technology, disclose a kind of based on flow matching and dynamic reward scheduling's imitation learning method, comprising: obtaining expert demonstration trajectory, and extracting expert feature sequence in expert demonstration trajectory using flow matching model;Obtain the policy state sequence of agent policy network executing task and convert to policy feature;Expert feature sequence and policy feature are used to construct reward function;Reward result is calculated and the value function network in agent policy network is optimized;According to the action sequence of the current policy of the value function network optimized after sampling agent policy network;According to action sequence, generate trajectory segment, and trajectory segment is used as training sample to iteratively optimize agent policy network, and the target agent policy network is obtained after iterative optimization is completed.The present application also proposes a kind of based on flow matching and dynamic reward scheduling's imitation learning device, equipment and storage medium.The present application can improve the state modeling efficiency in imitation learning and the stability of reward structure.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and particularly relates to an imitation learning method and apparatus based on stream matching and dynamic reward scheduling. Background Technology

[0002] Reinforcement learning (RL) interacts with the environment through trial and error, learning the optimal policy by maximizing cumulative rewards. In the field of autonomous driving, reward functions typically consider multiple factors, including safety, efficiency, and comfort. In recent years, deep reinforcement learning, a combination of deep learning and reinforcement learning, has become a new paradigm for end-to-end autonomous driving. Despite its successes in autonomous driving, reinforcement learning still faces the challenge of low efficiency.

[0003] While reinforcement learning doesn't require labeled data like supervised learning, it heavily relies on the reward function. Sometimes, even minor changes to the reward function can drastically alter the trained policy. In the context of autonomous driving, reward signals are extremely sparse (for example, for the regulation of an autonomous vehicle agent, the observation is the 3D local environment reconstructed from current environmental perception, and the action is the vehicle's specific path planning for the next few seconds. So what is the reward? If the reward is simply +1 for normal driving without a collision and -100 for a collision, the agent's learning result is likely to be simply stopping somewhere. Specific reward functions that help autonomous vehicles with regulation often require careful design and tuning by experts). In this situation, randomly designing a reward function cannot guarantee that the policy trained by reinforcement learning will meet actual needs.

[0004] Suppose there exists an expert agent whose policy can be considered optimal. A policy can be trained by directly imitating the expert's state-action data in the environment, without needing reward signals provided by the environment. Imitation learning studies this type of problem. Within the framework of imitation learning, the expert provides a series of state-action pairs, representing the actions the expert performs in environment S. The imitator's task is to use this expert data for training, achieving a policy close to the expert's without requiring reward signals.

[0005] Generative adversarial imitation learning (GAIL) is an important branch of imitation learning. It introduces the idea of ​​generative adversarial networks (GANs) into reinforcement learning, training a discriminator to distinguish between expert and imitator policies. As the adversarial process progresses, the data distribution generated by the imitator policy will approximate the distribution of real expert data. However, this learning paradigm shares the same problem as GANs: unstable training and a tendency for model collapse. In contrast, rating-based diffusion models are more stable in training and can produce higher-quality samples in fields such as audio and video. Intuitively, diffusion models corrupt samples by adding noise to the target distribution and then train a generative model to reverse this corruption. Crucially, these models are trained through rating matching: a non-adversarial, purely regressive process where the network is trained to match the ratings of the target distribution (i.e., the gradient of the log probability).

[0006] To further improve the stability of imitation learning, diffusion models, a non-adversarial, score-matching-based generative modeling approach, have been introduced into imitation learning tasks. Unlike adversarial methods, diffusion models construct a disruptive process by progressively adding Gaussian noise to expert samples, and then train a model to reverse this process, i.e., restore the "clean" state samples. The model's goal is to predict the clean signal or noise at each noise step, thereby approximating the gradient of the expert state distribution via regression. This process is essentially a non-adversarial scorematching method, avoiding the instability introduced by the discriminator and making training more stable.

[0007] The SMILING framework was proposed to address the instability introduced by discriminators in non-adversarial score matching. It uses the score matching process in the diffusion model as a training method for the score function in inverse reinforcement learning. By fitting the score function of the expert state distribution, it constructs the cost of policy search using this function. Compared to GAIL, SMILING does not rely on a discriminator, is more stable in training, and has stronger distribution representation capabilities. However, SMILING still suffers from low computational efficiency and unstable rewards in the early stages of training, especially in high-dimensional state spaces where the progressive sampling and denoising of the diffusion process itself constitute a significant computational burden. Furthermore, its reward design still relies on a single MSE fitting error, lacking dynamic adjustment capabilities, which can easily lead to training stagnation or poor convergence when the policy deviates significantly in the early stages.

[0008] Based on the above problems, there is an urgent need for an imitation learning method based on flow matching and dynamic reward scheduling to solve the problems of low efficiency in state modeling and unstable reward structure in imitation learning. Summary of the Invention

[0009] This invention provides an imitation learning method and apparatus based on flow matching and dynamic reward scheduling, which can improve the efficiency of state modeling and the stability of the reward structure in imitation learning.

[0010] To achieve the above objectives, this invention provides an imitation learning method based on flow matching and dynamic reward scheduling, comprising:

[0011] The expert demonstration trajectory is obtained, and the dynamic features of each expert state in the expert demonstration trajectory are extracted using a flow matching model. The dynamic features of each expert state are then integrated to obtain the expert feature sequence.

[0012] Obtain the policy state sequence of the agent policy network executing the task, and use the flow matching model to convert the state sequence into policy features;

[0013] Based on the training stage of the agent policy network, a reward function is constructed using expert feature sequences and policy features;

[0014] The reward result is calculated based on the reward function, and the reward result is used to optimize the median function network of the agent's policy network;

[0015] The agent policy network is optimized based on the optimized value function network, and the action sequence of the current policy is sampled from the optimized agent policy network.

[0016] Trajectory segments are generated based on the action sequence, and these trajectory segments are used as training samples to iteratively optimize the agent's policy network. After the iterative optimization is completed, the target agent's policy network is obtained.

[0017] Optionally, the step of extracting the dynamic features of each expert state in the expert demonstration trajectory using a flow matching model includes:

[0018] Transform the expert demonstration trajectory into a continuous state sequence;

[0019] State transition pairs are constructed using continuous state sequences and time steps of expert demonstration trajectories. Each state transition pair includes adjacent time steps and the state sequences corresponding to those adjacent time steps.

[0020] Dynamic features are obtained by extracting feature vectors of state transition pairs using a flow matching model.

[0021] Optionally, before extracting the feature vectors of state transition pairs using the flow matching model to obtain dynamic features, the flow matching model is further optimized.

[0022] Optionally, the convection matching model is optimized, including:

[0023] A vector field function is constructed based on the state transition pair, and a loss function for the flow matching model is constructed based on the feedforward neural network and the vector field function.

[0024] The flow matching model's state flow direction is trained using its loss function, and the network parameters are updated through backpropagation. After the network parameters are updated, the flow matching model is optimized.

[0025] Optionally, obtaining the policy state sequence of the agent policy network executing the task includes: obtaining the initial state of policy execution, and generating the policy state sequence by interacting with the environment using the agent policy network for a preset time step length.

[0026] Optionally, constructing a reward function using expert feature sequences and policy features based on the training stage of the agent's policy network includes:

[0027] Identify the current stage of the training process, which includes the initial stage, the middle stage, and the later stage.

[0028] In the initial stage, the first reward function is constructed using the cosine scheduling function and the mean square error function with the first weight.

[0029] In the intermediate stage, a second reward function is constructed using the mean squared error;

[0030] In the later stage, a third reward function is constructed using the sample contrast loss function and the mean squared error function with a second weight.

[0031] Based on the current training stage, the reward value for the initial stage is calculated using the first reward function, the reward value for the middle stage is calculated using the second reward function, and the reward value for the later stage is calculated using the third reward function.

[0032] Optionally, the optimization of the median function network of the agent's policy network using reward results includes:

[0033] Calculate the estimated value of each Q network in the dual-Q network structure based on the reward results;

[0034] Compare the value estimates of each Q-network and extract the smaller value estimate from the value estimates of each Q-network to obtain the target value estimate;

[0035] Optimize the median function network of the agent's policy network using the target value estimate.

[0036] To address the aforementioned problems, the present invention also provides an imitation learning device based on stream matching and dynamic reward scheduling, the device comprising:

[0037] The expert feature sequence extraction module is used to obtain the expert demonstration trajectory and use the flow matching model to extract the dynamic features of each expert state in the expert demonstration trajectory. The dynamic features of each expert state are integrated to obtain the expert feature sequence.

[0038] The policy feature extraction module is used to obtain the policy state sequence of the agent policy network executing the task, and to convert the state sequence into policy features using the flow matching model.

[0039] The target agent policy network acquisition module is used to construct a reward function based on the training stage of the agent policy network using expert feature sequences and policy features; calculate the reward result based on the reward function and optimize the median function network of the agent policy network using the reward result; optimize the agent policy network based on the optimized median function network and sample the action sequence of the current policy from the optimized agent policy network; generate trajectory segments based on the action sequences and use the trajectory segments as training samples to iteratively optimize the agent policy network. After iterative optimization, the target agent policy network is obtained.

[0040] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0041] At least one processor; and,

[0042] A memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the imitation learning method based on stream matching and dynamic reward scheduling described above.

[0044] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the above-described imitation learning method based on stream matching and dynamic reward scheduling.

[0045] This invention utilizes a flow matching model to extract dynamic features of each expert state in an expert demonstration trajectory, efficiently extracting feature representations of expert states and improving feature extraction efficiency. Furthermore, by acquiring the policy state sequence of the agent policy network executing the task and converting it into policy features using the flow matching model, comparability between policy states and expert states in a unified feature space is achieved. Using the same flow matching model to process the policy-generated state sequence and mapping it to the same feature space as the expert feature sequence ensures that expert features and policy features are in a comparable metric space, avoiding distribution drift problems. Additionally, based on the training stage of the agent policy network, a reward function is constructed using the expert feature sequence and policy features, introducing a dynamic reward scheduling mechanism to improve training stability and adaptability. The reward result is calculated based on the reward function, and the reward result is used to optimize the value function network in the agent policy network. Dynamic reward signals can guide value function learning. Furthermore, the value function guides policy network updates, and trajectory fragments are generated based on action sequences. These trajectory fragments are used as training samples to iteratively optimize the agent policy network. After iterative optimization, the target agent policy network is obtained, improving the efficiency of state modeling and the stability of the reward structure in imitation learning. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an imitation learning method based on flow matching and dynamic reward scheduling provided in an embodiment of the present invention.

[0047] A comparison of the reconstruction results of the present invention with existing techniques of imitation learning methods based on flow matching and dynamic reward scheduling;

[0048] Figure 2 A functional block diagram of an imitation learning device based on flow matching and dynamic reward scheduling provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the imitation learning method based on flow matching and dynamic reward scheduling, according to an embodiment of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] This application provides an imitation learning method based on stream matching and dynamic reward scheduling. The execution entity of this imitation learning method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the imitation learning method based on stream matching and dynamic reward scheduling can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0053] Reference Figure 1 The diagram shown is a flowchart illustrating an imitation learning method based on flow matching and dynamic reward scheduling according to an embodiment of the present invention. In this embodiment, the imitation learning method based on flow matching and dynamic reward scheduling includes:

[0054] S1. Obtain the expert demonstration trajectory and use the flow matching model to extract the dynamic features of each expert state in the expert demonstration trajectory. Integrate the dynamic features of each expert state to obtain the expert feature sequence.

[0055] In this embodiment of the invention, the expert demonstration trajectory refers to the state-action sequence generated by an expert (such as a human or a high-performance intelligent agent) during the execution of a driving task. The expert demonstration trajectory records how the expert gradually executes the optimal action from the initial state to complete the task.

[0056] In this embodiment of the invention, the flow matching model is a generative modeling framework based on continuous normalized flow. It learns the probabilistic flow path between the data distribution and the simple prior distribution by directly learning the data distribution. Its core is to describe the evolutionary trajectory using differential equations.

[0057] As an embodiment of the present invention, the dynamic features of each expert state in the expert demonstration trajectory are extracted using a flow matching model, including:

[0058] Transform the expert demonstration trajectory into a continuous state sequence;

[0059] State transition pairs are constructed using continuous state sequences and time steps of expert demonstration trajectories. Each state transition pair includes adjacent time steps and the state sequences corresponding to those adjacent time steps.

[0060] Dynamic features are obtained by extracting feature vectors of state transition pairs using a flow matching model.

[0061] Furthermore, before extracting the feature vectors of state transition pairs using the flow matching model to obtain dynamic features, the flow matching model is further optimized.

[0062] Furthermore, the convection matching model is optimized, including:

[0063] A vector field function is constructed based on the state transition pair, and a loss function for the flow matching model is constructed based on the feedforward neural network and the vector field function.

[0064] The flow matching model's state flow direction is trained using its loss function, and the network parameters are updated through backpropagation. After the network parameters are updated, the flow matching model is optimized.

[0065] For example, obtaining the expert demonstration trajectory and extracting the dynamic features of each expert state in the expert demonstration trajectory using a flow matching model, and integrating the dynamic features of each expert state to obtain the expert feature sequence, can be achieved using the following implementation steps:

[0066] Step 101: Describe the expert demonstration trajectory S expert Input stream matching model.

[0067] The expert demonstration trajectory consists of a continuous sequence of states, representing the behavioral path of a human expert in the task, denoted as .

[0068]

[0069] in, This represents the expert state at the first time step, where T is the time step length.

[0070] Step 102: Extract dynamic features from the expert state using a flow matching model.

[0071] Flow matching originates from the Probability Flow Ordinary Differential Equation (ODE), which describes the evolution of a trajectory by constructing a "dynamic consistency" function between states. This method does not directly model the distribution of states, but rather models the "flow trend" of states changing over time.

[0072] Constructing state transition pairs from expert state trajectories

[0073]

[0074] Where t is the time step number.

[0075] The goal is to learn a vector field function. The goal is to make the predicted state increments (i.e., the changes in state over time) as close as possible to the actual state transitions.

[0076]

[0077] Therefore, the flow matching loss function is defined as follows:

[0078]

[0079] here, It is a feedforward neural network (multilayer perceptron MLP) used to approximate the "flow direction" at each state point.

[0080] The physical meaning of this loss function is: if the expert state changes from s expert,t Real evolution to s expert,t+1 Then we hope that the network will predict the direction of flow. Consistent with the actual direction as much as possible.

[0081] Step 103: Train the flow matching network to obtain the dynamic features of each state.

[0082] The optimization process updates the network parameters θ through backpropagation, ultimately resulting in:

[0083]

[0084] After training, for any expert state s expert,t Each of these can be used to extract a feature vector consistent with its dynamic behavior:

[0085]

[0086] in, For the feature space, d s For the feature dimension, feature f expert,t It is a "dynamic behavioral feature" extracted from the original state, which can capture the trend information of the expert in that state (i.e. which direction the expert intends to go).

[0087] Ultimately, the entire expert trajectory is converted into a corresponding feature sequence:

[0088] S2. Obtain the policy state sequence of the agent policy network executing the task, and use the flow matching model to convert the state sequence into policy features.

[0089] In this embodiment of the invention, the agent policy network is the core neural network that drives agent decision-making in reinforcement learning (RL). Its function is to map the environmental state to the optimal action (or action probability distribution). The environmental state includes, but is not limited to, the raw sensor data such as the vehicle's position, speed, heading angle, acceleration, and steering angle in the autonomous driving scenario. The agent policy network includes, but is not limited to, the Soft Actor-Critic (SAC) network.

[0090] In this embodiment of the invention, the policy in the agent policy network is the rule for selecting actions in the autonomous driving scenario, the action is the specific path planning of the vehicle in the autonomous driving scenario, and the reward is a scalar signal that provides real-time feedback on the action in the autonomous driving scenario.

[0091] In this embodiment of the invention, the strategy state sequence refers to a set of multi-dimensional rules that integrate vehicle position / speed, traffic rules, vehicle driving history trajectory, and path planning in an intelligent driving scenario.

[0092] As an embodiment of the present invention, obtaining the policy state sequence of the agent policy network executing the task includes: obtaining the initial state of policy execution, and generating the policy state sequence by interacting with the environment through the agent policy network for a preset time step length.

[0093] For example, obtaining the policy state sequence of an agent's policy network executing a task can be achieved using the following implementation steps:

[0094] Step 1: Generate reinforcement learning trajectories using the current policy via Soft Actor-Critic (SAC).

[0095] Step 2: Use agent policy network in, This represents the policy of choosing action a given state s at time step t, using an agent policy network. Interact with the environment to obtain a state sequence S generated by the current agent's actions. agent Specifically, let the strategy start from the initial state and perform T steps of interaction to obtain... The trajectory describes the behavioral path of the current agent policy network and is comparable to the expert demonstration trajectory in the task space.

[0096] In this embodiment of the invention, Soft Actor-Critic (SAC) refers to a deep reinforcement learning algorithm designed specifically for continuous actions. Soft Actor-Critic integrates the Maximum Entropy principle, the Actor-Critic framework, and dual-Q network technology.

[0097] The steps of converting the state sequence into policy features in this embodiment of the invention are the same as the steps of extracting the dynamic features of each expert state in the expert demonstration trajectory using a flow matching model, and will not be repeated here. The final policy features are as follows:

[0098]

[0099] S3. Based on the training stage of the agent's policy network, construct a reward function using expert feature sequences and policy features.

[0100] As an embodiment of the present invention, a reward function is constructed using expert feature sequences and policy features based on the training stage of the agent policy network, including:

[0101] Identify the current stage of the training process, which includes the initial stage, the middle stage, and the later stage.

[0102] In the initial stage, the first reward function is constructed using the cosine scheduling function and the mean square error function with the first weight.

[0103] In the intermediate stage, a second reward function is constructed using the mean squared error;

[0104] In the later stage, a third reward function is constructed using the sample contrast loss function and the mean squared error function with a second weight.

[0105] Based on the current training stage, the reward value for the initial stage is calculated using the first reward function, the reward value for the middle stage is calculated using the second reward function, and the reward value for the later stage is calculated using the third reward function.

[0106] For example, based on the training stage of the agent's policy network, a reward function can be constructed using expert feature sequences and policy features, which can be achieved through the following implementation steps:

[0107] Obtain the policy state feature f agent,t and expert state characteristics f expert,t Then, based on the current training step (step) and its corresponding stage, the corresponding reward function (r) is executed. t Construction process:

[0108] Phase One (Initial Phase):

[0109] Using cosine scheduling function The reward is formed by combining the mean square error perturbation of MSE: Where λ1 is the first weight.

[0110] Phase Two (Mid-Term Phase):

[0111] The fitting method primarily uses the mean squared error of MSE.

[0112] Phase Three (Later Stage):

[0113] The InfoNCE loss function is used as the primary supervisor, supplemented by the mean squared error of MSE:

[0114] Where λ2 is the second weight.

[0115] In this embodiment of the invention, the initial stage is the coarse guidance stage, the middle stage is the fine fitting stage, and the later stage is the semantic alignment stage.

[0116] In this embodiment of the invention, the InfoNCE loss function is used for semantic alignment in the later stage, specifically by constructing positive and negative sample pairs to achieve feature consistency constraints. Positive sample pairs consist of "features generated by the policy network at the current moment" and "expert features from the previous moment or the same state in the corresponding expert demonstration," aiming to utilize the temporal continuity information of the expert demonstration trajectory for semantic supervision. Negative samples select features from other time steps, other trajectories, or different states to enhance discriminative power. By maximizing the similarity of positive samples and minimizing the similarity of negative samples, the model is guided to learn the structural consistency between the policy and the expert in the temporal dimension, thereby extracting a more semantically deep state representation and improving generalization ability and behavior prediction accuracy.

[0117] S4. Calculate the reward result based on the reward function, and use the reward result to optimize the median function network of the agent's policy network.

[0118] As one embodiment of the present invention, optimizing the median function network of the agent's policy network using reward results includes:

[0119] Calculate the estimated value of each Q network in the dual-Q network structure based on the reward results;

[0120] Compare the value estimates of each Q-network and extract the smaller value estimate from the value estimates of each Q-network to obtain the target value estimate;

[0121] Optimize the median function network of the agent's policy network using the target value estimate.

[0122] In this embodiment of the invention, the dual-Q network structure is an important component of the agent policy network.

[0123] For example, calculating the reward result based on the reward function and using the reward result to optimize the median function network of the agent's policy network can be achieved through the following implementation steps:

[0124] Based on the reward r calculated in the previous step tThis updates the value function (Critic) in Soft Actor-Critic. The value function objective is as follows:

[0125]

[0126] Where γ is the first weight parameter and a is the second weight parameter, a double Q network structure is used to update Q1 and Q2, where Q1 is the first value estimate and Q2 is the second value estimate:

[0127]

[0128] To avoid overestimation, the target value is min(Q1,Q2).

[0129] S5. Optimize the agent policy network based on the optimized value function network, and sample the action sequence of the current policy from the optimized agent policy network.

[0130] In this embodiment of the invention, the action sequence refers to the sequence of underlying control instructions generated by the autonomous driving system during the continuous decision-making process. For example, it directly controls the lateral (steering) and longitudinal (speed) movements of the vehicle and ultimately generates an executable trajectory segment.

[0131] In this embodiment of the invention, optimizing the agent policy network based on the optimized value function network can be achieved using the following implementation steps:

[0132] Policy network output action distribution By maximizing reward-driven optimization, the objective function is...

[0133]

[0134] Where Q(s) t ,a t ) is the minimum value of the two-valued function min(Q1,Q2), used to approximate the Advantage function and encourage the output of high-reward actions.

[0135] In this embodiment of the invention, the advantage function is a core indicator that measures the degree of improvement in the average action value of a specific action a relative to the current state s.

[0136] Furthermore, sampling the action sequence of the current policy from the optimized agent policy network can be achieved through the following steps: after the agent policy network is updated, continue to interact with the environment and sample the action sequence of the current policy. New action sequences are sampled.

[0137] In this embodiment of the invention, the environment refers to the physical or logical rules that can be implemented by the vehicle in an autonomous driving scenario.

[0138] S6. Generate trajectory segments based on the action sequence, and use the trajectory segments as training samples to iteratively optimize the agent policy network. After the iterative optimization is completed, the target agent policy network is obtained.

[0139] In this embodiment of the invention, trajectory segments are generated based on action sequences, and these trajectory segments are used as training samples to iteratively optimize the agent policy network. After the iterative optimization is completed, the target agent policy network is obtained. This can be achieved using the following implementation steps:

[0140] According to the current strategy π θ Sampling action a in the environment t ~π θ (a t |s t );

[0141] Perform sampling action a t ~π θ (a t |s t ), observe the next state s t+1 The interaction results will be used to form new samples (s) t ,a t ,r t ,s t+1 Add it to the experience replay cache. This is used for training the subsequent agent policy network.

[0142] In this embodiment of the invention, the experience replay cache is a core component in deep reinforcement learning, and is essentially a circular queue that stores the historical interaction data of the agent.

[0143] This invention utilizes a flow matching model to extract dynamic features of each expert state in an expert demonstration trajectory, efficiently extracting feature representations of expert states and improving feature extraction efficiency. Furthermore, by acquiring the policy state sequence of the agent policy network executing the task and converting it into policy features using the flow matching model, comparability between policy states and expert states in a unified feature space is achieved. Using the same flow matching model to process the policy-generated state sequence and mapping it to the same feature space as the expert feature sequence ensures that expert features and policy features are in a comparable metric space, avoiding distribution drift problems. Additionally, based on the training stage of the agent policy network, a reward function is constructed using the expert feature sequence and policy features, introducing a dynamic reward scheduling mechanism to improve training stability and adaptability. The reward result is calculated based on the reward function, and the reward result is used to optimize the value function network in the agent policy network. Dynamic reward signals can guide value function learning. Furthermore, the value function guides policy network updates, and trajectory fragments are generated based on action sequences. These trajectory fragments are used as training samples to iteratively optimize the agent policy network. After iterative optimization, the target agent policy network is obtained, improving the efficiency of state modeling and the stability of the reward structure in imitation learning.

[0144] like Figure 2 The diagram shown is a functional block diagram of an imitation learning device based on flow matching and dynamic reward scheduling according to an embodiment of the present invention.

[0145] The imitation learning device 100 based on flow matching and dynamic reward scheduling described in this invention can be installed in an electronic device. Depending on the functions implemented, the imitation learning device 100 may include an expert feature sequence extraction module 101, a policy feature extraction module 102, and a target agent policy network acquisition module 103.

[0146] The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0147] In this embodiment, the functions of each module / unit are as follows:

[0148] The expert feature sequence extraction module 101 is used to acquire the expert demonstration trajectory, extract the dynamic features of each expert state in the expert demonstration trajectory using a flow matching model, and integrate the dynamic features of each expert state to obtain the expert feature sequence.

[0149] In this embodiment of the invention, the expert demonstration trajectory refers to the state-action sequence generated by an expert (such as a human or a high-performance intelligent agent) during the execution of a driving task. The expert demonstration trajectory records how the expert gradually executes the optimal action from the initial state to complete the task.

[0150] In this embodiment of the invention, the flow matching model is a generative modeling framework based on continuous normalized flow. It learns the probabilistic flow path between the data distribution and the simple prior distribution by directly learning the data distribution. Its core is to describe the evolutionary trajectory using differential equations.

[0151] As an embodiment of the present invention, the dynamic features of each expert state in the expert demonstration trajectory are extracted using a flow matching model, including:

[0152] Transform the expert demonstration trajectory into a continuous state sequence;

[0153] State transition pairs are constructed using continuous state sequences and time steps of expert demonstration trajectories. Each state transition pair includes adjacent time steps and the state sequences corresponding to those adjacent time steps.

[0154] Dynamic features are obtained by extracting feature vectors of state transition pairs using a flow matching model.

[0155] Furthermore, before extracting the feature vectors of state transition pairs using the flow matching model to obtain dynamic features, the flow matching model is further optimized.

[0156] Furthermore, the convection matching model is optimized, including:

[0157] A vector field function is constructed based on the state transition pair, and a loss function for the flow matching model is constructed based on the feedforward neural network and the vector field function.

[0158] The flow matching model's state flow direction is trained using its loss function, and the network parameters are updated through backpropagation. After the network parameters are updated, the flow matching model is optimized.

[0159] For example, obtaining the expert demonstration trajectory and extracting the dynamic features of each expert state in the expert demonstration trajectory using a flow matching model, and integrating the dynamic features of each expert state to obtain the expert feature sequence, can be achieved using the following implementation steps:

[0160] Step 101: Describe the expert demonstration trajectory S expert Input stream matching model.

[0161] The expert demonstration trajectory consists of a continuous sequence of states, representing the behavioral path of a human expert in the task, denoted as .

[0162]

[0163] in, This represents the expert state at the first time step, where T is the time step length.

[0164] Step 102: Extract dynamic features from the expert state using a flow matching model.

[0165] Flow matching originates from the Probability Flow Ordinary Differential Equation (ODE), which describes the evolution of a trajectory by constructing a "dynamic consistency" function between states. This method does not directly model the distribution of states, but rather models the "flow trend" of states changing over time.

[0166] Constructing state transition pairs from expert state trajectories

[0167]

[0168] Where t is the time step number.

[0169] The goal is to learn a vector field function. The goal is to make the predicted state increments (i.e., the changes in state over time) as close as possible to the actual state transitions.

[0170]

[0171] Therefore, the flow matching loss function is defined as follows:

[0172]

[0173] here, It is a feedforward neural network (multilayer perceptron MLP) used to approximate the "flow direction" at each state point.

[0174] The physical meaning of this loss function is: if the expert state changes from s expert,t Real evolution to s expert,t+1 Then we hope that the network will predict the direction of flow. Consistent with the actual direction as much as possible.

[0175] Step 103: Train the flow matching network to obtain the dynamic features of each state.

[0176] The optimization process updates the network parameters θ through backpropagation, ultimately resulting in:

[0177]

[0178] After training, for any expert state sexpert,t Each of these can be used to extract a feature vector consistent with its dynamic behavior:

[0179]

[0180] in, For the feature space, d s For the feature dimension, feature f expert,t It is a "dynamic behavioral feature" extracted from the original state, which can capture the trend information of the expert in that state (i.e. which direction the expert intends to go).

[0181] Ultimately, the entire expert trajectory is converted into a corresponding feature sequence:

[0182] The policy feature extraction module 102 is used to obtain the policy state sequence of the agent policy network executing the task, and use the flow matching model to convert the state sequence into policy features.

[0183] In this embodiment of the invention, the agent policy network is the core neural network that drives agent decision-making in reinforcement learning (RL). Its function is to map the environmental state to the optimal action (or action probability distribution). The environmental state includes, but is not limited to, the raw sensor data such as the vehicle's position, speed, heading angle, acceleration, and steering angle in the autonomous driving scenario. The agent policy network includes, but is not limited to, the Soft Actor-Critic (SAC) network.

[0184] In this embodiment of the invention, the policy in the agent policy network is the rule for selecting actions in the autonomous driving scenario, the action is the specific path planning of the vehicle in the autonomous driving scenario, and the reward is a scalar signal that provides real-time feedback on the action in the autonomous driving scenario.

[0185] In this embodiment of the invention, the strategy state sequence refers to a set of multi-dimensional rules that integrate vehicle position / speed, traffic rules, vehicle driving history trajectory, and path planning in an intelligent driving scenario.

[0186] As an embodiment of the present invention, obtaining the policy state sequence of the agent policy network executing the task includes: obtaining the initial state of policy execution, and generating the policy state sequence by interacting with the environment through the agent policy network for a preset time step length.

[0187] For example, obtaining the policy state sequence of an agent's policy network executing a task can be achieved using the following implementation steps:

[0188] Step 1: Generate reinforcement learning trajectories using the current policy via Soft Actor-Critic (SAC).

[0189] Step 2: Use agent policy network in, This represents the policy of choosing action a given state s at time step t, using an agent policy network. Interact with the environment to obtain a state sequence S generated by the current agent's actions. agent Specifically, let the strategy start from the initial state and perform T steps of interaction to obtain... The trajectory describes the behavioral path of the current agent policy network and is comparable to the expert demonstration trajectory in the task space.

[0190] In this embodiment of the invention, Soft Actor-Critic (SAC) refers to a deep reinforcement learning algorithm designed specifically for continuous actions. Soft Actor-Critic integrates the Maximum Entropy principle, the Actor-Critic framework, and dual-Q network technology.

[0191] The steps of converting the state sequence into policy features in this embodiment of the invention are the same as the steps of extracting the dynamic features of each expert state in the expert demonstration trajectory using a flow matching model, and will not be repeated here. The final policy features are as follows:

[0192]

[0193] The target agent policy network acquisition module 103 is used to construct a reward function using expert feature sequences and policy features according to the training stage of the agent policy network; calculate the reward result according to the reward function, and optimize the median function network of the agent policy network using the reward result; optimize the agent policy network according to the optimized median function network, and sample the action sequence of the current policy from the optimized agent policy network; generate trajectory segments according to the action sequences, and use the trajectory segments as training samples to iteratively optimize the agent policy network, and obtain the target agent policy network after the iterative optimization is completed.

[0194] As an embodiment of the present invention, a reward function is constructed using expert feature sequences and policy features based on the training stage of the agent policy network, including:

[0195] Identify the current stage of the training process, which includes the initial stage, the middle stage, and the later stage.

[0196] In the initial stage, the first reward function is constructed using the cosine scheduling function and the mean square error function with the first weight.

[0197] In the intermediate stage, a second reward function is constructed using the mean squared error;

[0198] In the later stage, a third reward function is constructed using the sample contrast loss function and the mean squared error function with a second weight.

[0199] Based on the current training stage, the reward value for the initial stage is calculated using the first reward function, the reward value for the middle stage is calculated using the second reward function, and the reward value for the later stage is calculated using the third reward function.

[0200] For example, based on the training stage of the agent's policy network, a reward function can be constructed using expert feature sequences and policy features, which can be achieved through the following implementation steps:

[0201] Obtain the policy state feature f agent,t and expert state characteristics f expert,t Then, based on the current training step (step) and its corresponding stage, the corresponding reward function (r) is executed. t Construction process:

[0202] Phase One (Initial Phase):

[0203] Using cosine scheduling function The reward is formed by combining the mean square error perturbation of MSE: Where λ1 is the first weight.

[0204] Phase Two (Mid-Term Phase):

[0205] The fitting method primarily uses the mean squared error of MSE.

[0206] Phase Three (Later Stage):

[0207] The InfoNCE loss function is used as the primary supervisor, supplemented by the mean squared error of MSE:

[0208] Where λ2 is the second weight.

[0209] In this embodiment of the invention, the initial stage is the coarse guidance stage, the middle stage is the fine fitting stage, and the later stage is the semantic alignment stage.

[0210] In this embodiment of the invention, the InfoNCE loss function is used for semantic alignment in the later stage, specifically by constructing positive and negative sample pairs to achieve feature consistency constraints. Positive sample pairs consist of "features generated by the policy network at the current moment" and "expert features from the previous moment or the same state in the corresponding expert demonstration," aiming to utilize the temporal continuity information of the expert demonstration trajectory for semantic supervision. Negative samples select features from other time steps, other trajectories, or different states to enhance discriminative power. By maximizing the similarity of positive samples and minimizing the similarity of negative samples, the model is guided to learn the structural consistency between the policy and the expert in the temporal dimension, thereby extracting a more semantically deep state representation and improving generalization ability and behavior prediction accuracy.

[0211] As one embodiment of the present invention, optimizing the median function network of the agent's policy network using reward results includes:

[0212] Calculate the estimated value of each Q network in the dual-Q network structure based on the reward results;

[0213] Compare the value estimates of each Q-network and extract the smaller value estimate from the value estimates of each Q-network to obtain the target value estimate;

[0214] Optimize the median function network of the agent's policy network using the target value estimate.

[0215] In this embodiment of the invention, the dual-Q network structure is an important component of the agent policy network.

[0216] For example, calculating the reward result based on the reward function and using the reward result to optimize the median function network of the agent's policy network can be achieved through the following implementation steps:

[0217] Based on the reward r calculated in the previous step t This updates the value function (Critic) in Soft Actor-Critic. The value function objective is as follows:

[0218]

[0219] Where γ is the first weight parameter and a is the second weight parameter, a double Q network structure is used to update Q1 and Q2, where Q1 is the first value estimate and Q2 is the second value estimate:

[0220]

[0221] To avoid overestimation, the target value is min(Q1,Q2).

[0222] In this embodiment of the invention, the action sequence refers to the sequence of underlying control instructions generated by the autonomous driving system during the continuous decision-making process. For example, it directly controls the lateral (steering) and longitudinal (speed) movements of the vehicle and ultimately generates an executable trajectory segment.

[0223] In this embodiment of the invention, optimizing the agent policy network based on the optimized value function network can be achieved using the following implementation steps:

[0224] Policy network output action distribution By maximizing reward-driven optimization, the objective function is...

[0225]

[0226] Where Q(s) t ,a t ) is the minimum value of the two-valued function min(Q1,Q2), used to approximate the Advantage function and encourage the output of high-reward actions.

[0227] In this embodiment of the invention, the advantage function is a core indicator that measures the degree of improvement in the average action value of a specific action a relative to the current state s.

[0228] Furthermore, sampling the action sequence of the current policy from the optimized agent policy network can be achieved through the following steps: after the agent policy network is updated, continue to interact with the environment and sample the action sequence of the current policy. New action sequences are sampled.

[0229] In this embodiment of the invention, the environment refers to the physical or logical rules that can be implemented by the vehicle in an autonomous driving scenario.

[0230] In this embodiment of the invention, trajectory segments are generated based on action sequences, and these trajectory segments are used as training samples to iteratively optimize the agent policy network. After the iterative optimization is completed, the target agent policy network is obtained. This can be achieved using the following implementation steps:

[0231] According to the current strategy π θ Sampling action a in the environment t ~π θ (a t |s t );

[0232] Perform sampling action a t ~π θ (a t |s t ), observe the next state s t+1 The interaction results will be used to form new samples (s) t ,a t ,rt ,s t+1 Add it to the experience replay cache. This is used for training the subsequent agent policy network.

[0233] In this embodiment of the invention, the experience replay cache is a core component in deep reinforcement learning, and is essentially a circular queue that stores the historical interaction data of the agent.

[0234] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements an imitation learning method based on flow matching and dynamic reward scheduling, according to an embodiment of the present invention.

[0235] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as an imitation learning method program based on stream matching and dynamic reward scheduling.

[0236] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a program for imitation learning based on flow matching and dynamic reward scheduling) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0237] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of an imitation learning method program based on stream matching and dynamic reward scheduling, but also to temporarily store data that has been output or will be output.

[0238] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0239] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0240] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0241] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0242] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0243] The program for an imitation learning method based on stream matching and dynamic reward scheduling, stored in the memory 11 of the electronic device, is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0244] The expert demonstration trajectory is obtained, and the dynamic features of each expert state in the expert demonstration trajectory are extracted using a flow matching model. The dynamic features of each expert state are then integrated to obtain the expert feature sequence.

[0245] Obtain the policy state sequence of the agent policy network executing the task, and use the flow matching model to convert the state sequence into policy features;

[0246] Based on the training stage of the agent policy network, a reward function is constructed using expert feature sequences and policy features;

[0247] The reward result is calculated based on the reward function, and the reward result is used to optimize the median function network of the agent's policy network;

[0248] The agent policy network is optimized based on the optimized value function network, and the action sequence of the current policy is sampled from the optimized agent policy network.

[0249] Trajectory segments are generated based on the action sequence, and these trajectory segments are used as training samples to iteratively optimize the agent's policy network. After the iterative optimization is completed, the target agent's policy network is obtained.

[0250] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0251] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0252] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0253] The expert demonstration trajectory is obtained, and the dynamic features of each expert state in the expert demonstration trajectory are extracted using a flow matching model. The dynamic features of each expert state are then integrated to obtain the expert feature sequence.

[0254] Obtain the policy state sequence of the agent policy network executing the task, and use the flow matching model to convert the state sequence into policy features;

[0255] Based on the training stage of the agent policy network, a reward function is constructed using expert feature sequences and policy features;

[0256] The reward result is calculated based on the reward function, and the reward result is used to optimize the median function network of the agent's policy network;

[0257] The agent policy network is optimized based on the optimized value function network, and the action sequence of the current policy is sampled from the optimized agent policy network.

[0258] Trajectory segments are generated based on the action sequence, and these trajectory segments are used as training samples to iteratively optimize the agent's policy network. After the iterative optimization is completed, the target agent's policy network is obtained.

[0259] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0260] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0261] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0262] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0263] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0264] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0265] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0266] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An imitation learning method based on flow matching and dynamic reward scheduling, characterized in that, The method includes: The expert demonstration trajectory is obtained, and the dynamic features of each expert state in the expert demonstration trajectory are extracted using a flow matching model. The dynamic features of each expert state are then integrated to obtain the expert feature sequence. The strategy state sequence of the intelligent agent policy network executing the task is obtained, and the state sequence is converted into policy features using the flow matching model. The environmental state includes the raw sensor data of the vehicle's position, speed, heading angle, acceleration and steering angle in the autonomous driving scenario. The intelligent agent policy network is a Soft Actor-Critic network. In the intelligent agent policy network, the policy is the rule for selecting actions in the autonomous driving scenario. The action is the specific path planning of the vehicle in the autonomous driving scenario. The reward is the scalar signal that gives the current action the immediate feedback of the action in the autonomous driving scenario. The strategy state sequence refers to the set of multi-dimensional rules that integrate vehicle position / speed, traffic rules, vehicle driving history trajectory and path planning in the intelligent driving scenario. Based on the training stage of the agent's policy network, a reward function is constructed using expert feature sequences and policy features. This construction of the reward function based on the training stage of the agent's policy network includes: Identify the current stage of the training process, which includes the initial stage, the middle stage, and the later stage. In the initial stage, the first reward function is constructed using the cosine scheduling function and the mean square error function with the first weight. In the intermediate stage, a second reward function is constructed using the mean squared error; In the later stage, a third reward function is constructed using the sample contrast loss function and the mean squared error function with a second weight. Based on the current training phase, the reward value for the initial phase is calculated using the first reward function, the reward value for the middle phase is calculated using the second reward function, and the reward value for the later phase is calculated using the third reward function. The reward result is calculated based on the reward function, and the reward result is used to optimize the median function network of the agent's policy network; The agent policy network is optimized based on the optimized value function network, and the action sequence of the current policy is sampled from the optimized agent policy network. The action sequence refers to the sequence of underlying control instructions generated by the autonomous driving system during the continuous decision-making process. Trajectory segments are generated based on the action sequence, and these trajectory segments are used as training samples to iteratively optimize the agent's policy network. After the iterative optimization is completed, the target agent's policy network is obtained.

2. The imitation learning method based on flow matching and dynamic reward scheduling as described in claim 1, characterized in that, The extraction of dynamic features of each expert state in the expert demonstration trajectory using a flow matching model includes: Transform the expert demonstration trajectory into a continuous state sequence; State transition pairs are constructed using continuous state sequences and time steps of expert demonstration trajectories. Each state transition pair includes adjacent time steps and the state sequences corresponding to those adjacent time steps. Dynamic features are obtained by extracting feature vectors of state transition pairs using a flow matching model.

3. The imitation learning method based on flow matching and dynamic reward scheduling as described in claim 2, characterized in that, Before extracting the feature vectors of state transition pairs using the flow matching model to obtain dynamic features, the flow matching model is optimized.

4. The imitation learning method based on flow matching and dynamic reward scheduling as described in claim 3, characterized in that, The optimization of the convection matching model includes: A vector field function is constructed based on the state transition pair, and a loss function for the flow matching model is constructed based on the feedforward neural network and the vector field function. The flow matching model's state flow direction is trained using its loss function, and the network parameters are updated through backpropagation. After the network parameters are updated, the flow matching model is optimized.

5. The imitation learning method based on flow matching and dynamic reward scheduling as described in claim 1, characterized in that, The process of obtaining the policy state sequence of the agent policy network executing the task includes: obtaining the initial state of policy execution, and generating the policy state sequence by interacting with the environment through the agent policy network for a preset time step length.

6. The imitation learning method based on flow matching and dynamic reward scheduling as described in claim 1, characterized in that, The method of optimizing the median function network of the agent's policy network using reward results includes: Calculate the estimated value of each Q network in the dual-Q network structure based on the reward results; Compare the value estimates of each Q-network and extract the value estimate with the smallest value among the value estimates of each Q-network to obtain the target value estimate; Optimize the median function network of the agent's policy network using the target value estimate.

7. An imitation learning device based on stream matching and dynamic reward scheduling, characterized in that, The apparatus can implement the imitation learning method based on stream matching and dynamic reward scheduling as described in any one of claims 1 to 6, and the apparatus includes: The expert feature sequence extraction module is used to obtain the expert demonstration trajectory and use the flow matching model to extract the dynamic features of each expert state in the expert demonstration trajectory. The dynamic features of each expert state are integrated to obtain the expert feature sequence. The strategy feature extraction module is used to obtain the strategy state sequence of the intelligent agent strategy network executing the task, and to convert the state sequence into strategy features using the flow matching model. The environmental state includes the raw sensor data of the vehicle's position, speed, heading angle, acceleration and steering angle in the autonomous driving scenario. Its intelligent agent strategy network is a Soft Actor-Critic network. In the intelligent agent strategy network, the strategy is the rule for selecting actions in the autonomous driving scenario. The action is the specific path planning of the vehicle in the autonomous driving scenario. The reward is the scalar signal that gives the current action the immediate feedback of the action in the autonomous driving scenario. Its strategy state sequence refers to the set of multi-dimensional rules that integrate vehicle position / speed, traffic rules, vehicle driving history trajectory and path planning in the intelligent driving scenario. The target agent policy network acquisition module is used to construct a reward function based on the training stage of the agent policy network using expert feature sequences and policy features. This construction includes: identifying the current training stage, which may include an initial stage, a middle stage, and a later stage; constructing a first reward function using a cosine scheduling function and a mean squared error function with a first weight in the initial stage; constructing a second reward function using the mean squared error function in the middle stage; and constructing a third reward function using a sample comparison loss function and a mean squared error function with a second weight in the later stage. The system calculates the reward value for the initial stage using the first reward function, the reward value for the middle stage using the second reward function, and the reward value for the later stage using the third reward function. Based on the reward function calculations, the reward results are used to optimize the median function network of the agent policy network. The optimized median function network is then used to further optimize the agent policy network, and action sequences of the current policy are sampled from the optimized agent policy network. These action sequences refer to the sequence of underlying control commands generated by the autonomous driving system during continuous decision-making. Trajectory segments are generated based on the action sequences, and these trajectory segments are used as training samples to iteratively optimize the agent policy network. After iterative optimization, the target agent policy network is obtained.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the imitation learning method based on stream matching and dynamic reward scheduling as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the imitation learning method based on stream matching and dynamic reward scheduling as described in any one of claims 1 to 6.

Images