Intelligent agent control strategy model training method and device, equipment and storage medium

Abstract

The invention discloses an intelligent agent control strategy model training method and device, equipment and a storage medium. The method comprises the following steps: constructing a training sample set; inputting each vehicle state sample in the training sample set into the to-be-trained strategy model to obtain corresponding second action reasoning data output by the to-be-trained strategy model; determining a target loss function type according to a training demand corresponding to the to-be-trained strategy model; determining a target loss function value according to first action reasoning data output by a reference strategy model for each vehicle state sample, second action reasoning data output by a to-be-trained strategy model for each vehicle state sample and a target loss function type, and performing parameter updating on the to-be-trained strategy model based on the target loss function value until a training stop condition is met, thereby obtaining a target strategy model. Therefore, by dynamically selecting the loss function type, the adaptability of the model training process to different training requirements is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle energy management, and in particular to a method and device for training a control strategy model of an intelligent agent, equipment and a storage medium. BACKGROUND

[0002] Reinforcement learning is increasingly widely used in the fields of intelligent control, autonomous driving and energy management. Reinforcement learning agents can learn excellent control strategies in high-dimensional state spaces and continuous action spaces.

[0003] In the related art, in the field of vehicle energy management, a knowledge transfer method is usually applied to reinforcement learning to utilize existing experience or knowledge learned in other tasks to improve the training efficiency and generalization ability of the control strategy model of the intelligent agent. However, the transfer mechanism in the related art is difficult to flexibly adapt to the model training needs of different tasks, resulting in the need to improve the adaptability of the training process of the control strategy model to different training task requirements. SUMMARY

[0004] The present application aims to at least partially solve one of the technical problems in the related art. To this end, the present application proposes a method and device for training a control strategy model of an intelligent agent, which effectively improves the adaptability of the training process of the control strategy model of the intelligent agent to different training task requirements by dynamically selecting a loss function type.

[0005] To achieve the above-mentioned purpose, the first aspect of the present application proposes a method for training a control strategy model of an intelligent agent, the intelligent agent being applied to a vehicle energy management system, the method comprising: constructing a training sample set; wherein the training sample set comprises a plurality of vehicle state samples; the plurality of vehicle state samples comprise a plurality of vehicle state inference data obtained by a reference strategy model performing continuous action inference on first preset state data; each vehicle state sample corresponds to first action inference data output by the reference strategy model; the reference strategy model is a trained model; inputting each vehicle state sample into a to-be-trained strategy model to obtain corresponding second action inference data output by the to-be-trained strategy model; determining a target loss function type from a plurality of candidate loss function types according to the training requirements of the to-be-trained strategy model; determining a target loss function value according to the first action inference data output by the reference strategy model for each vehicle state sample, the second action inference data output by the to-be-trained strategy model for each vehicle state sample, and the target loss function type, to update the parameters of the to-be-trained strategy model based on the target loss function value until a training stop condition is met, and obtain a target strategy model.

[0006] According to an embodiment of the present application, the manner of obtaining the plurality of vehicle state inference data comprises: inputting first preset state data into the reference strategy model to perform action inference to obtain current action inference data corresponding to the first preset state data, and adding 1 to the inference number; obtaining vehicle state inference data corresponding to the current action inference data; taking the vehicle state inference data corresponding to the current action inference data as the first preset state data, and returning to the step of inputting the first preset state data into the reference strategy model to perform action inference until the inference number reaches a preset number threshold.

[0007] According to an embodiment of the present application, the action inference data output by each of the reference strategy model and the to-be-trained strategy model for each vehicle state sample satisfies a preset probability distribution.

[0008] According to an embodiment of the present application, the plurality of candidate loss function types comprise a relative entropy loss function type and a mean square error loss function type.

[0009] According to an embodiment of the present application, the parameter size of the target strategy model is smaller than the parameter size of the reference strategy model.

[0010] According to an embodiment of the present application, after the target loss function value is determined, the control strategy model training method further comprises: drawing a loss value curve according to the target loss function value.

[0011] According to an embodiment of the present application, after the target strategy model is obtained, the control strategy model training method further comprises: inputting second preset state data into the reference strategy model and the target strategy model respectively to obtain a first inference sequence output by the reference strategy model and a second inference sequence output by the target strategy model; determining first evaluation index data corresponding to the reference strategy model according to the first inference sequence, and determining second evaluation index data corresponding to the target strategy model according to the second inference sequence; wherein the first evaluation index data comprises at least one of fuel consumption data, reward data, battery power deviation data, and engine start number; the second evaluation index data comprises an index type in the first evaluation index data; and based on the first evaluation index data and the second evaluation index data, a comparison is performed to determine an evaluation result corresponding to the target strategy model.

[0012] To achieve the above object, the second aspect of the present application provides a device for training a control strategy model of an agent, characterized in that the agent is applied to a vehicle energy management system, and the device comprises: a sample set construction module for constructing a training sample set; wherein the training sample set comprises a plurality of vehicle state samples; each vehicle state sample corresponds to first action inference data output by a reference strategy model; the reference strategy model is a trained model; an action inference data acquisition module for inputting each vehicle state sample into a to-be-trained strategy model to obtain corresponding second action inference data output by the to-be-trained strategy model; a loss function type determination module for determining a target loss function type from a plurality of candidate loss function types according to training requirements of the to-be-trained strategy model; and a target strategy model acquisition module for determining a target loss function value according to the first action inference data output by the reference strategy model for each vehicle state sample, the second action inference data output by the to-be-trained strategy model for each vehicle state sample, and the target loss function type, and performing parameter update on the to-be-trained strategy model based on the target loss function value until a training stop condition is met to obtain a target strategy model.

[0013] To achieve the above object, the third aspect of the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the control strategy model training method according to any one of the preceding embodiments when executing the computer program.

[0014] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the control strategy model training method according to any one of the preceding embodiments.

[0015] According to the embodiments of the present application, the to-be-trained model can directly learn an action pattern close to the optimal one without starting from zero to explore the optimal strategy by means of the sample data provided by the trained reference strategy model, thereby greatly reducing the number of training iterations, avoiding falling into a local optimal solution, and significantly improving the efficiency of model training by means of knowledge transfer technology. By dynamically selecting the loss function type, the training target can be adjusted according to different training scenarios or requirements, etc. In this way, the adaptability of the training process of the control strategy model of the agent in the field of vehicle energy management to different training task requirements is effectively improved, and the target strategy model obtained finally can be flexibly adapted to diversified vehicle energy management requirements.

[0016] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a control strategy model training method according to an embodiment of the present specification.

[0018] Figure 2 A flowchart of obtaining a plurality of vehicle state inference data according to an embodiment of the present specification.

[0019] Figure 3a A flowchart of another control strategy model training method according to an embodiment of the present specification.

[0020] Figure 3b A flowchart of a distillation training method of a hybrid electric vehicle energy management strategy model according to an embodiment of the present specification.

[0021] Figure 4 A block diagram of a control strategy model training device according to an embodiment of the present specification.

[0022] Figure 5 A block diagram of a computer device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0023] Embodiments of the present application are described in detail below with reference to the attached drawings, which are shown by way of example, and wherein the same or similar components have the same or similar designations throughout the several figures. The embodiments described below are exemplary and are intended to be illustrative of the present application and are not to be construed as limiting thereof.

[0024] At present, reinforcement learning (RL) is increasingly widely used in the fields of intelligent control, autonomous driving and energy management. Reinforcement learning agents can learn excellent control strategies in high-dimensional state space and continuous action space.

[0025] In the related art, a knowledge transfer method is applied to reinforcement learning to utilize existing experience or knowledge learned in other tasks to improve the training efficiency and generalization ability of reinforcement learning. However, the commonly used transfer mechanism (especially the loss function design) is still difficult to flexibly adapt to the training needs of different tasks. Specifically, most transfer methods rely on a fixed loss function form (such as mean square error or standard policy gradient loss), making it difficult to achieve dynamic adaptation and efficient transfer of the loss function. Therefore, although knowledge transfer helps to break through the dependence of reinforcement learning on a large amount of interaction data, the transfer paradigm based on a fixed loss structure in the related art still cannot meet the requirements of model generalization and adaptation ability in diversified practical application scenarios.

[0026] Since reinforcement learning models (especially policy network models based on deep neural networks) usually have large parameter quantities and complex structures, the model inference speed is usually slow, and the deployment resource demand is high, which limits the real-time performance and vehicle embedded deployment capability of reinforcement learning models in vehicle energy management. Therefore, the knowledge distillation (KD) method is often used in the related art to improve the training efficiency of reinforcement learning and to realize control policy compression, which transfers the knowledge of a strong but complex teacher network to a simplified student network, so as to obtain a model with acceptable performance and lighter structure, and completes the knowledge transfer of the excellent control policy learned by the large teacher model to the lightweight student model.

[0027] For the training of the control strategy model of the agent in the vehicle energy management system, different training stages or different subtasks may require the distillation process to dynamically adjust the focus of knowledge transfer, for example, sometimes the action needs to be strictly imitated, sometimes the shape of the value function needs to be aligned, and sometimes the uncertainty of the policy in a specific state needs to be focused on. However, the fixed loss function design relied on by the knowledge distillation in the related art lacks the ability to adapt to such differentiated needs, making it difficult to flexibly adapt to the diversified model training needs of the agent in the vehicle energy management system, and not considering the difference in policy distribution between the teacher model and the student model, which cannot fully preserve the diversity of the policy.

[0028] The fixed loss function design relied on by knowledge distillation is difficult to flexibly adapt to the diversified model training needs of the agent in the vehicle energy management system. For example, when the training requirement is to let the student model not only imitate the final action output of the teacher model, but also understand its internal decision-making process, the commonly used mean square error (MSE) loss function cannot guarantee that the student model learns the policy generalization and inference process of the teacher model in similar state space.

[0029] To improve the adaptability of knowledge transfer in the field of vehicle energy management to different training task requirements, it is necessary to propose a method, apparatus, device, and storage medium for training control strategy models of intelligent agents. The control strategy model training method proposed in this specification first utilizes a pre-trained reference strategy model to construct a training sample set for training the strategy model to be trained. Specifically, first preset state data is input into the reference strategy model, causing the reference strategy model to perform continuous action reasoning based on the preset state data, generating a series of vehicle state reasoning data. Based on this series of vehicle state reasoning data, several vehicle state samples are formed, thus constituting the training sample set. Each vehicle state sample corresponds to the first action reasoning data output by the reference strategy model for the corresponding vehicle state.

[0030] Secondly, each vehicle state sample in the training sample set is input into the policy model to be trained, so that the policy model to be trained outputs the corresponding second action inference data for each vehicle state sample.

[0031] Secondly, based on the training requirements of the policy model to be trained, a suitable target loss function type is selected from multiple candidate loss function types. Different training requirements may require different types of loss functions to measure the difference between the model output and the expected output, thereby better guiding model training.

[0032] Then, based on the action inference data output by the reference policy model and the policy model to be trained for the vehicle state sample, and the selected target loss function type, the target loss function value corresponding to the vehicle state sample is calculated.

[0033] Finally, the parameters of the policy model to be trained are updated based on the target loss function value. The above process is repeated until the training stopping condition is met, and the target policy model that has been trained is finally obtained.

[0034] Therefore, by leveraging sample data provided by a pre-trained reference policy model, the model to be trained does not need to explore the optimal policy from scratch. It can directly learn near-optimal action patterns, significantly reducing the number of training iterations, avoiding getting trapped in local optima, and significantly improving the efficiency of model training through knowledge transfer techniques. By dynamically selecting the loss function type, the training objective can be adjusted according to different training scenarios or needs (such as urban roads, highway driving, battery protection, etc.). For example, if the smoothness of action decisions is emphasized, a loss function with a regularization term can be used; if it is necessary to suppress abnormal outputs, a robust loss function (such as the Huber loss function) can be used, and so on. In this way, the adaptability of the training process of the control policy model of the agent in the field of vehicle energy management to the needs of different training tasks is effectively improved, so that the final target policy model can flexibly adapt to diverse vehicle energy management needs.

[0035] This specification provides a method for training a control strategy model for an intelligent agent, which is applied to a vehicle energy management system. (Refer to...) Figure 1 As shown, the training method for this control strategy model may include the following steps.

[0036] S110. Construct a training sample set; wherein, the training sample set includes several vehicle state samples; the several vehicle state samples include multiple vehicle state reasoning data obtained by the reference policy model performing continuous action reasoning on the first preset state data; each vehicle state sample corresponds to the first action reasoning data output by the reference policy model; the reference policy model is the model that has been trained.

[0037] S120. Input each vehicle state sample into the policy model to be trained to obtain the corresponding second action inference data output by the policy model to be trained.

[0038] S130. Based on the training requirements corresponding to the policy model to be trained, determine the target loss function type from multiple candidate loss function types.

[0039] S140. Based on the first action inference data output by the reference policy model for each vehicle state sample, the second action inference data output by the policy model to be trained for each vehicle state sample, and the target loss function type, determine the target loss function value, and update the parameters of the policy model to be trained based on the target loss function value until the training stopping condition is met, thereby obtaining the target policy model.

[0040] The reference strategy model is a pre-trained control strategy model for vehicle energy management. It can output optimal or near-optimal control actions (i.e., action inference data, such as engine start / stop, motor torque distribution, and battery charge / discharge) based on the current vehicle state (e.g., vehicle speed, battery charge, engine speed, power demand, road gradient, etc.). It acts as an expert or mentor during the learning process of the strategy model to be trained.

[0041] Vehicle state inference data refers to the subsequent vehicle state that is inferred and evolved after the agent performs the corresponding control action based on the action inference data currently output by the reference policy model.

[0042] Action inference data is the output of control decisions made by the strategy model for a certain vehicle state, such as engine power distribution, battery charge and discharge, engine torque, etc.

[0043] The policy model to be trained is also a control policy model used for vehicle energy management, and it is also used to infer control actions based on the input vehicle state. The training requirements for the policy model to be trained are also the training objectives, such as ensuring the smoothness of the action decisions of the trained model and ensuring the accuracy of the action decisions of the trained model.

[0044] Specifically, the first preset state data is input into a well-trained and high-performance reference policy model (e.g., a high-precision model trained based on reinforcement learning or expert rules). The reference policy model then performs continuous action reasoning on the first preset state data, resulting in a continuous action sequence. From this continuous action sequence, a vehicle state sequence can be derived, yielding multiple vehicle state reasoning data. Each vehicle state reasoning data serves as a vehicle state sample, resulting in several vehicle state samples that constitute a training sample set. Each vehicle state sample corresponds to the first action reasoning data obtained by the reference policy model through corresponding action reasoning. Each vehicle state sample is then input into the policy model to be trained, causing the policy model to output corresponding second action reasoning data.

[0045] Then, based on the specific training requirements of the model to be trained (such as focusing more on action accuracy, robustness, response speed, or computational efficiency), the most suitable target loss function type is selected from multiple candidate loss function types (e.g., MSE loss function, cross-entropy loss function, Huber loss function, weighted loss function, etc.).

[0046] For these vehicle state samples, based on the corresponding first action inference data output by the reference policy model, the corresponding second action inference data output by the policy model to be trained, and the target loss function type, the corresponding loss value is calculated, which is the target loss function value, to measure the difference between the output of the policy model to be trained and the output of the reference policy model.

[0047] Based on the target loss function value, the parameters of the policy model to be trained are updated using the backpropagation algorithm until the training stopping condition is met (e.g., loss convergence, reaching the maximum number of iterations, performance no longer improving, etc.), ultimately yielding the trained target policy model. Then, knowledge transfer is performed based on the reference policy model to obtain a template policy model.

[0048] For example, multiple candidate loss function types may include MSE loss function, L1 loss function, weighted loss function, MAE (Mean Absolute Error) loss function, KL divergence (Kullback-Leibler Divergence) loss function, etc.

[0049] Assuming the training requirement for the policy model to be trained is to ensure that the action output of the model to be trained is as consistent as possible with the output of the reference policy model, with no significant deviation, the MSE loss function can be chosen as the target loss function type. The target loss function value can be calculated according to the following example formula:

[0050] Where n represents the number of vehicle state samples; It is the action output by the reference strategy model for the i-th vehicle state sample; It is the action output by the policy model to be trained for the i-th vehicle state sample.

[0051] If the training requirement for the policy model to be trained is to avoid extreme actions (such as a battery experiencing a sudden large current discharge), then the L1 loss function can be chosen as the target loss function type. The target loss function value can be calculated using the following example formula:

[0052] Assuming the training requirement for the policy model to be trained is to improve action accuracy under key conditions (such as high-speed driving, rapid acceleration, etc.), a weighted loss function can be chosen as the target loss function type. The target loss function value can be calculated according to the following example formula:

[0053] in, It is the weight of the i-th vehicle state sample.

[0054] If the training requirement for the policy model to be trained is that its energy allocation strategy should be as efficient as the reference policy model, while strictly limiting the extreme deviations in battery charging and discharging power, then the MSE loss function and the truncated MSE loss function can be chosen as the target loss function types. The target loss function value is the weighted sum of the MSE loss value and the truncated MSE loss value. The truncated MSE loss value can be calculated using the following example formula:

[0055] in, is the safety threshold; k is a penalty coefficient greater than 1.

[0056] The target loss function value is L=αL MSE +(1-α)L 截断MSE Where α is the weighting coefficient, which can be set according to the actual application scenario.

[0057] Assuming the training requirement for the strategy model to be trained is that the single-step energy control action (such as motor output torque) must be consistent with the output of the reference strategy model, and that the cumulative error of energy consumption during long-term driving must also be controlled, the MAE loss function and the cumulative error loss function (which penalizes the cumulative deviation of energy consumption within a certain number of action steps) can be selected as the target loss function type. The target loss function value is the weighted sum of the MAE loss value and the cumulative error loss value. The cumulative error loss value can be calculated according to the following example formula:

[0058] Where E represents energy consumption and m represents the number of steps.

[0059] The target loss function value is L=βL L1 +(1-β)L2. Where β is the weighting coefficient, which can be set according to the actual application scenario.

[0060] It should be noted that the data dimensions and formats of the vehicle state data (including preset state data, vehicle state inference data, vehicle state samples, etc.) in this manual can be determined according to actual application requirements. The candidate loss function type and the method of determining the target loss function type according to the training requirements of the policy model to be trained can also be determined according to the actual application scenario. This manual does not impose specific limitations on them.

[0061] In some cases, several vehicle state samples include first preset state data and multiple vehicle state inference data obtained by the reference strategy model through continuous action inference based on the first preset state data.

[0062] In the above implementation, by utilizing sample data provided by a pre-trained reference policy model, the model to be trained does not need to explore the optimal policy from scratch. It can directly learn near-optimal action patterns, significantly reducing the number of training iterations and avoiding getting trapped in local optima. Knowledge transfer techniques are used to significantly improve the efficiency of model training. By dynamically selecting the loss function type, the training objective can be adjusted according to different training scenarios or needs. For example, if the smoothness of action decisions is emphasized, a loss function with a regularization term can be used; if it is necessary to suppress abnormal outputs, a robust loss function (such as the Huber loss function) can be used, and so on. This effectively improves the adaptability of knowledge transfer in the field of vehicle energy management to different training task requirements, enabling the final target policy model to flexibly adapt to diverse vehicle energy management needs.

[0063] Meanwhile, the reference strategy model serves as a benchmark, ensuring that the action output of the strategy model to be trained remains within a reasonable range, avoiding extreme actions that violate physical laws or safety constraints during training. The final target strategy model can learn the advantageous characteristics of the reference strategy model and may, through further optimization, surpass it, achieving more efficient vehicle energy distribution and consumption control.

[0064] In some implementations, reference Figure 2 As shown, the acquisition of multiple vehicle state inference data can include the following steps.

[0065] S210. Input the first preset state data into the reference strategy model to perform action reasoning, obtain the current action reasoning data corresponding to the first preset state data, and add 1 to the reasoning count.

[0066] S220. Obtain the vehicle state reasoning data corresponding to the current action reasoning data.

[0067] S230. Take the vehicle state reasoning data corresponding to the current action reasoning data as the first preset state data, and return to the step of inputting the first preset state data into the reference strategy model for action reasoning, until the number of reasoning times reaches the preset number threshold.

[0068] Among them, the vehicle state inference data corresponding to the current action inference data refers to the state that the vehicle may enter after the agent performs a control action based on the current action inference data.

[0069] Specifically, the first preset state data is input into the reference strategy model, and the reference strategy model outputs the optimal control action for the vehicle state corresponding to the state data, which is recorded as the current action inference data, and the inference count is incremented by 1. In the initial state, the inference count is 0.

[0070] Based on the current action reasoning data output by the reference policy model, the vehicle's physical model (such as energy consumption model, power transmission model, etc.) can be combined to calculate the new state that the vehicle is expected to enter when the agent performs the action, thus obtaining the vehicle state reasoning data corresponding to the current action reasoning data.

[0071] The vehicle state reasoning data is used as the first preset state data for the new round. The process returns to the step of inputting the first preset state data into the reference policy model for action reasoning, repeating the process of inputting into the reference policy model → obtaining new actions → calculating new states. The reasoning count is incremented by 1 after each loop until the count reaches a preset threshold, at which point the loop stops.

[0072] For example, the input to the reference strategy model is 5-dimensional vehicle state data, including engine speed, vehicle speed, battery SOC (State of Charge), clutch open / closed state parameters, and power demand. The output of the reference strategy model is 1-dimensional engine control action data, which is normalized to [-1, 1] using a Tanh (hyperbolic tangent) activation function before being mapped to engine torque. The input and output structures of the strategy model to be trained are identical to those of the reference strategy model.

[0073] The data acquisition process based on the reference policy model (i.e., the process of acquiring multiple vehicle state inference data) mainly includes: Step 1. Load the trained reference policy model in the environment env.

[0074] Step 2. Reset the initial battery SOC value in the environmental state (uniformly sampled within the range of 45-55%) to simulate a complete WLTC (World Light Vehicle Test Cycle) cycle. Specifically, under the initial cycle corresponding to each initial battery SOC value, the reference strategy model executes 1800 action inference steps (i.e., the preset threshold of 1800 steps), and correspondingly evolves to obtain 1799 subsequent vehicle states, thus forming a complete WLTC cycle.

[0075] Step 3. Input the current vehicle state (i.e., the first preset state data) containing the current battery SOC into the reference policy model to obtain the action output by the reference policy model. Record the triplet of the current state, action (i.e., the current action inference data) and the evolved next state (i.e., the vehicle state inference data), and increment the inference count by 1.

[0076] Step 4. If the number of inferences is less than 1800, take the next state obtained in step 3 as the current vehicle state and return to step 3 to perform the next action inference; if the number of inferences reaches 1800, stop the loop.

[0077] Following the steps described above, the reference policy model performs 1000 rounds × 1800 steps of action reasoning (i.e., 1800 steps of reasoning are executed in each round, and the initial battery SOC value in the environmental state needs to be reset in each round). This allows for the collection of 1000 rounds × 1800 steps of continuous action reasoning data and corresponding vehicle state reasoning data. The collected data is then used to construct a training dataset, `teacher_data`, for training the subsequent policy model.

[0078] It should be noted that the preset threshold for the number of attempts can be set according to the actual application scenario, etc., and is not specifically limited in this manual. In this manual, the reference strategy model can also be called the teacher strategy model, and the strategy model to be trained (and the target strategy model) can also be called the student strategy model.

[0079] In some implementations, the action inference data output by the reference policy model and the policy model to be trained for each vehicle state sample both satisfy a preset probability distribution.

[0080] Specifically, for a given vehicle state sample, both the reference policy model and the policy model to be trained do not directly output a specific control action value. Instead, they output a probability distribution as action inference data, such as the mean (μ) and variance (μ) of a Gaussian distribution. The final action performed by the agent is sampled from this probability distribution.

[0081] In some feasible implementations, the preset probability distribution can be any of the following: Gaussian distribution, Beta distribution, Laplace distribution, etc.

[0082] In some implementations, multiple candidate loss function types include relative entropy loss function type and mean squared error loss function type.

[0083] Among them, the relative entropy loss function type, namely the KL divergence loss function type, can be used to model the difference in probability distribution of continuous action space between the reference policy model and the policy model to be trained, so as to achieve accurate transfer of policy behavior at the probability level.

[0084] For example, the action inference data output by both the reference policy model and the policy model to be trained for each vehicle state sample follows a Gaussian distribution. A loss function encapsulation mechanism is designed, providing a loss function switching interface to support switching between the mean squared error (MSE) loss function and the KL divergence loss function. The KL divergence loss function is primarily used for scenarios where the training requirement is for the policy model to be trained to learn the decision logic of the reference policy model, while the MSE loss function is primarily used for scenarios where the training requirement is for the policy model to be trained to learn the decision results of the reference policy model.

[0085] For a given vehicle state sample, the KL divergence loss function can be represented as follows:

[0086] in, It is the average action output by the reference strategy model; It is the average action output of the policy model to be trained; It is the standard deviation of the output of the reference strategy model; It is the standard deviation of the output of the policy model to be trained.

[0087] For a given vehicle state sample, the MSE loss function can be represented as follows:

[0088] Users can flexibly switch the type of loss function to use by using the parameters mode='kl' or mode='mse' to meet different training needs. At the same time, the encapsulated function module can uniformly schedule loss calculation and gradient update to improve module reusability.

[0089] Therefore, a switchable dual-mode distillation mechanism is achieved by supporting knowledge transfer methods based on KL divergence loss and MSE loss functions through a unified interface. This allows for flexible switching within a unified training framework, and the interface is concisely encapsulated and applicable to various policy models. Furthermore, by designing action distribution transfer based on the KL divergence loss function and using an explicit probability distribution alignment loss function for knowledge transfer of action outputs, policy fidelity can be effectively improved.

[0090] In some scenarios, the relative entropy loss function type can be used as the default target loss function type.

[0091] Understandably, in some cases, both the relative entropy loss function type and the mean squared error loss function type can be selected as the target loss function type, but the details will not be elaborated here.

[0092] In some implementations, the parameter size of the target policy model is smaller than that of the reference policy model.

[0093] Among them, parameter size refers to the total number of learnable parameters in the model (such as the weights and biases of a neural network), which directly affects the computational complexity, storage requirements and running efficiency of the model.

[0094] Specifically, the smaller parameter size of the target policy model compared to the reference policy model means that the target policy model is obtained through knowledge distillation based on the reference policy model. The reference policy model, serving as the training benchmark, is a complex expert model with a large parameter size (such as deep neural networks or complex reinforcement learning models). Its advantages include high decision-making accuracy and the ability to handle complex scenarios, but it also incurs higher computational costs.

[0095] The target policy model is a lightweight model with fewer parameters and a more concise structure (such as shallow networks, decision trees, lightweight neural networks, etc.) formed by compressing the number and width of network layers and learning the decision logic of reference policy models, thus significantly reducing complexity while ensuring core performance.

[0096] For example, the reference policy model TeacherActor is a three-layer neural network containing two layers of 128 hidden nodes, activated by the ReLU (Rectified Linear Unit) function, and the output is a one-dimensional engine torque control action. The action space is normalized to [-1, 1] by the Tanh activation function and then mapped to the engine torque.

[0097] The target policy model, StudentActor, is a lightweight structure that uses two layers of 64 hidden nodes. Its input and output structure is consistent with that of the reference policy model, thus achieving policy compression.

[0098] It should be noted that the specific model structures of the reference policy model and the target policy model (or the policy model to be trained) can be selected according to actual application needs, and this specification does not impose specific limitations.

[0099] In some implementations, after determining the target loss function value, the control policy model training method may further include: plotting a loss value curve based on the target loss function value.

[0100] Specifically, the target loss function values ​​calculated in each round (or batch of samples) during the training process are connected sequentially according to the training steps (such as the number of iterations) to form a curve that reflects the trend (or convergence trend) of the loss value, i.e., the loss value curve.

[0101] The loss curve visually represents how the target loss function value changes over training iterations. Observing the trend of the curve helps users clearly understand the model's performance improvement throughout the training process and determine whether the model is optimizing in the expected direction.

[0102] The loss curve can help users determine whether the model has converged. If the curve stabilizes after a certain number of iterations, it means that the model's performance is no longer significantly improving and may be close to convergence. Conversely, if the curve continues to fluctuate or the downward trend is not obvious, further adjustments to the training parameters or model structure may be needed.

[0103] During training, various anomalies may occur, such as overfitting, underfitting, vanishing gradients, or exploding gradients. Loss curves can serve as an intuitive tool to help users identify these problems promptly. Therefore, using loss curves as a visual diagnostic tool during model training transforms abstract loss values ​​into intuitive trend curves, helping users monitor training status in real time, identify problems promptly, and optimize training strategies, ultimately improving the training efficiency and reliability of the target policy model.

[0104] In some implementations, referenceFigure 3a As shown, after obtaining the target policy model, the control policy model training method may also include: S310. Input the second preset state data into the reference strategy model and the target strategy model respectively to obtain the first inference sequence output by the reference strategy model and the second inference sequence output by the target strategy model.

[0105] S320. Determine the first evaluation index data corresponding to the reference strategy model based on the first inference sequence, and determine the second evaluation index data corresponding to the target strategy model based on the second inference sequence; wherein, the first evaluation index data includes at least one of fuel consumption data, reward data, battery charge deviation data, and engine start times; the second evaluation index data includes the index type in the first evaluation index data.

[0106] S330. Based on the comparison of the first evaluation index data and the second evaluation index data, determine the evaluation result corresponding to the target strategy model.

[0107] The first inference sequence includes continuous action inference data obtained by the reference strategy model through continuous action inference on the second preset state data, and the second inference sequence includes continuous action inference data obtained by the target strategy model through continuous action inference on the second preset state data.

[0108] Specifically, the second preset state data is input into the reference policy model and the target policy model respectively to obtain the temporal decision sequence containing continuous actions output by the reference policy model, namely the first inference sequence, and the temporal decision sequence containing continuous actions output by the target policy model, namely the second inference sequence.

[0109] For the first inference sequence output by the reference strategy model, the corresponding first evaluation index data is calculated and determined. The first evaluation index data covers at least one aspect of the following: fuel consumption data (reflecting the vehicle's fuel consumption under a specific inference sequence), reward data (an index that quantifies the rationality and effectiveness of a specific inference sequence according to a preset reward mechanism), battery charge deviation data (measuring the degree of deviation between the battery charge and the expected charge after the execution of a specific inference sequence), and engine start count (in hybrid vehicles, the number of engine starts affects the vehicle's energy consumption and driving comfort).

[0110] For the second inference sequence output by the target strategy model, determine its corresponding second evaluation index data according to the same index type as the first evaluation index data.

[0111] The data for the first evaluation indicator and the data for the second evaluation indicator are compared item by item. This can be done by calculating statistical indicators such as the difference value and relative error between the two, or by using visualization methods such as bar charts. Based on the comparison results, the evaluation result corresponding to the target strategy model is determined. The evaluation result can be a comprehensive score, a detailed evaluation of each indicator, or a comprehensive evaluation result report, etc., used to determine the performance difference between the target strategy model and the reference strategy model.

[0112] It is understandable that the second preset state data may be the same as or different from the first preset state data.

[0113] For example, in order to ensure the effectiveness of knowledge transfer (or knowledge distillation), and to assess the performance of the target policy model and the reference policy model, a multi-index evaluation mechanism is designed to systematically evaluate the knowledge distillation effect from multiple perspectives, such as policy distribution modeling, structure compression, training efficiency, and deployment evaluation.

[0114] Specifically, the evaluation metrics include: average fuel consumption in a single simulation (i.e., total fuel consumption difference, in grams, used to measure the energy-saving effect of the strategy), battery SOC deviation (the deviation between the battery charge at the end of a single simulation and a preset charge threshold (e.g., 50%), in percentage, used to measure battery charge stability), engine start-stop cycles (the number of times the engine engages from zero torque, used to evaluate the start-stop smoothness of the strategy), and total reward (environmental reward, representing the overall performance of the strategy). In some feasible implementations, the battery SOC deviation metric can also be the average battery SOC deviation metric, obtained by recording the deviation between the battery charge at the end of a single simulation and the preset charge threshold, and calculating the average deviation between the battery charge and the preset charge threshold for a specified number of simulations (e.g., 5 or 3 cycles).

[0115] A visualization module is also designed to plot a comparison chart of the battery SOC in the next state corresponding to each step of the inference data of the target policy model and the reference policy model in a single round of simulation, as well as a bar chart comparing the target policy model and the reference policy model for the aforementioned evaluation metrics. This visualization module can also be used to output loss value curves during model training.

[0116] Meanwhile, an evaluation interface function, evaluate_policy(), was designed that can be reused for performance evaluation between different policy models.

[0117] For example, in the field of intelligent agent compression distillation in hybrid vehicle EMS (Energy Management System), the related technologies that use behavior cloning (BC) as a distillation method have the following shortcomings: (1) Most reinforcement learning distillation methods rely on the MSE loss function and do not consider the differences in policy distribution between the teacher policy model (i.e., the reference policy model) and the student policy model (i.e., the target policy model or the policy model to be trained), and cannot fully preserve the diversity of policies; (2) There is a lack of explicit alignment mechanism for action probability distribution, which has limited effect, especially in continuous action space; (3) There is a lack of a sound distillation effect evaluation mechanism, and the deployment effect of the student policy model lacks quantitative analysis; (4) The design of distillation interface and loss function lacks flexibility, which limits the scope of application of the method.

[0118] Therefore, a distillation training method for hybrid electric vehicle energy management strategy models is proposed, which combines distributed modeling capabilities, lightweight structural design, and multi-index evaluation mechanisms. By introducing the KL divergence loss function logic, a knowledge distillation method based on KL divergence reinforcement learning continuous action space strategy model is designed. An evaluation and comparison mechanism that embeds the EMS teacher strategy model and student strategy model is integrated to achieve the goals of lightweight deployment of EMS agents and minimization of performance loss.

[0119] refer to Figure 3b As shown, the main process of this distillation training method includes parameter setting, network initialization, collecting sample data based on the teacher strategy model, training the student strategy model, evaluating the performance of the teacher strategy model and the student strategy model, printing the evaluation results and drawing a comparison chart.

[0120] The initialization of the network refers to defining the network structures of the teacher's policy model and the student's policy model. The teacher's policy model receives 5-dimensional vehicle states as input, including engine speed, vehicle speed, battery SOC, clutch engagement / disengagement parameters, and power demand. The teacher's policy model's network structure consists of two fully connected layers, each with 128 dimensions, activated by the ReLU function, and outputs one-dimensional normalized action data. The student's policy model has the same input / output structure as the teacher's policy model, but its network width is pruned to two 64-dimensional fully connected layers, employing a lightweight structure.

[0121] Sample data is collected based on the teacher's policy model, including collecting triplet data of state-action-next_state under the teacher's policy model. Each round of sampling involves 1800 steps, and the process is repeated for 1000 sampling rounds (episodes) to cover a broad state space. The triplet data is designed in the form of {s_t (state), a_t (action), s_(t+1) (next_state)}, and is used to train the student's policy model. For a detailed description of the sample data collection process, please refer to the description of the data collection process based on the reference policy model in this manual; it will not be repeated here.

[0122] During the training of the student policy model, a loss function encapsulation mechanism is adopted, which encapsulates the KL divergence loss function and the MSE loss function through a unified structure to support switching between the two loss function modes. At the same time, the function encapsulation module uniformly schedules loss calculation and gradient update, improving module reusability.

[0123] The main steps of the distillation training process include: (1) defining a unified distill_train() distillation training function; (2) selecting the loss function type, mode="kl" indicating that the KL divergence loss function is used to compare the difference between the action distributions output by the teacher policy model and the student policy model; (3) using a batch size of 64 during distillation training, training for 1000 epochs, and fixing the learning rate at 1e-4; (4) outputting the loss value curve during the distillation training process for visualization of convergence.

[0124] When evaluating the performance of the teacher's and student's policy models, the `evaluate_policy()` function is called to assess their performance, including the following performance metrics: fuel consumption per round, total reward, mean SOC deviation over 5 rounds (mean absolute deviation from the preset SOC threshold of 50%), and number of engine starts. Each round of simulation generates a SOC trajectory comparison chart, and the evaluation results for each performance metric are displayed as a bar chart. The loss curve and performance metric comparison chart during the distillation training process are ultimately saved as `DDPG_distill_compare_kl.png`.

[0125] The entire process described above can be implemented using PyTorch, with the final teacher policy model size being approximately 70KB and the student policy model size being approximately 20KB. The inference speed of the student policy model is significantly improved, meeting the requirements for embedded deployment. Furthermore, this distillation training method is highly scalable, supporting the switching of algorithms (such as DDPG (Deep Deterministic Policy Gradient), PPO (Proximal Policy Optimization), TD3 (TwinDelayed Deep Deterministic Policy Gradient), etc.) and reward function design.

[0126] It should be noted that the teacher's policy model can be derived from mainstream reinforcement learning algorithms such as PPO, TD3, and SAC (SoftActor-Critic). The distillation objective can be switched to a state-action value function (Q-function) or a state-value function (V-function). The student's policy model can be further optimized using model compression techniques such as pruning, quantization, and low-rank decomposition instead of structural simplification.

[0127] Through actual simulation experiments on the above-mentioned distillation training method, the performance evaluation results of the student strategy model were obtained. The performance evaluation results show that the student strategy model's stability performance for SOC is basically close to that of the teacher strategy model, and the average fuel consumption performance of the student strategy model is better than that of the teacher strategy model during inference. The reward of the student strategy model during inference is lower because the average stability deviation of SOC is slightly larger, which is within the normal acceptable range. The average number of engine starts is the same for both the student strategy model and the teacher strategy model.

[0128] Therefore, the above-mentioned distillation training method achieves flexible switching of different loss functions under a unified training framework through a switchable dual-mode distillation mechanism; it achieves accurate transfer of policy behavior at the probabilistic level through action distribution transfer design based on KL divergence, improving policy fidelity; it reduces parameter size and memory usage through lightweight student policy model design, showing significant advantages in embedded deployment scenarios; and it evaluates the teacher policy model and student policy model separately through a complete evaluation and visualization process, quantifying performance indicators such as fuel consumption, SOC stability, policy reward value, and engine start-up frequency, and compares distillation effects through a multi-indicator comparison and verification mechanism and visualization methods to ensure controllable performance after distillation. The above-mentioned distillation training method has the advantages of high distillation efficiency and low performance loss, effectively solving problems such as excessively large parameter size of the teacher policy model, difficulty in deployment when vehicle-side resources are limited, and slow inference. It comprehensively improves the vehicle-side usability and lightweight nature of the hybrid electric vehicle EMS intelligent control policy model, and has broad engineering promotion prospects and practical application value.

[0129] Furthermore, a control action decision-making method for an intelligent agent is provided, applied to an intelligent agent in a vehicle energy management system. The control action decision-making method includes: acquiring current vehicle state data; using a target policy model trained according to the control policy model training method in any of the preceding embodiments to perform action reasoning on the current vehicle state data, obtaining current action reasoning data corresponding to the current vehicle state data; and determining a target control action based on the current action reasoning data.

[0130] This specification also provides a control strategy model training device for an intelligent agent, which is applied to a vehicle energy management system. (See reference...) Figure 4 As shown, the control strategy model training device 400 may include: a sample set construction module 410, an action reasoning data acquisition module 420, a loss function type determination module 430, and a target strategy model acquisition module 440.

[0131] The sample set construction module 410 is used to construct a training sample set; wherein, the training sample set includes several vehicle state samples; the several vehicle state samples include multiple vehicle state reasoning data obtained by the reference policy model performing continuous action reasoning on the first preset state data; each vehicle state sample corresponds to the first action reasoning data output by the reference policy model; the reference policy model is a model that has been trained.

[0132] The action reasoning data acquisition module 420 is used to input each vehicle state sample into the policy model to be trained, and obtain the corresponding second action reasoning data output by the policy model to be trained.

[0133] The loss function type determination module 430 is used to determine the target loss function type from multiple candidate loss function types based on the training requirements corresponding to the policy model to be trained.

[0134] The target policy model acquisition module 440 is used to determine the target loss function value based on the first action inference data output by the reference policy model for each vehicle state sample, the second action inference data output by the policy model to be trained for each vehicle state sample, and the target loss function type. The target loss function value is then used to update the parameters of the policy model to be trained until the training stopping condition is met, thus obtaining the target policy model.

[0135] Specific limitations regarding the control strategy model training device can be found in the limitations of the control strategy model training method described above, and will not be repeated here. Each module in the aforementioned control strategy model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0136] This specification also provides a computer device, see embodiments thereof. Figure 5 As shown, the computer device 500 may include a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements the control strategy model training method in any of the foregoing embodiments.

[0137] This specification also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the control strategy model training method of any of the foregoing embodiments.

[0138] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0139] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0140] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0141] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0142] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0143] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for training a control strategy model for an intelligent agent, characterized in that, The intelligent agent is applied to a vehicle energy management system, and the method includes: Construct a training sample set; wherein the training sample set includes several vehicle state samples; the several vehicle state samples include multiple vehicle state inference data obtained by the reference policy model performing continuous action inference on the first preset state data; each vehicle state sample corresponds to the first action inference data output by the reference policy model; the reference policy model is a model that has been trained. Each vehicle state sample is input into the policy model to be trained to obtain the corresponding second action inference data output by the policy model to be trained. Based on the training requirements corresponding to the policy model to be trained, the target loss function type is determined from multiple candidate loss function types; Based on the first action inference data output by the reference policy model for each vehicle state sample, the second action inference data output by the policy model to be trained for each vehicle state sample, and the target loss function type, a target loss function value is determined. The parameters of the policy model to be trained are then updated based on the target loss function value until the training stopping condition is met, thereby obtaining the target policy model.

2. The method according to claim 1, characterized in that, The methods for acquiring the multiple vehicle state inference data include: The first preset state data is input into the reference strategy model to perform action reasoning, and the current action reasoning data corresponding to the first preset state data is obtained. The reasoning count is then incremented by 1. Obtain the vehicle state inference data corresponding to the current action inference data; The vehicle state reasoning data corresponding to the current action reasoning data is used as the first preset state data, and the process returns to the step of inputting the first preset state data into the reference strategy model for action reasoning, until the number of reasoning attempts reaches the preset number threshold.

3. The method according to claim 1, characterized in that, The action reasoning data output by the reference strategy model and the strategy model to be trained for each vehicle state sample both satisfy a preset probability distribution.

4. The method according to claim 3, characterized in that, The multiple candidate loss function types include relative entropy loss function type and mean squared error loss function type.

5. The method according to claim 1, characterized in that, The parameter size of the target strategy model is smaller than that of the reference strategy model.

6. The method according to claim 1, characterized in that, After determining the target loss function value, the method further includes: Plot the loss curve based on the target loss function value.

7. The method according to any one of claims 1 to 6, characterized in that, After obtaining the target strategy model, the method further includes: The second preset state data is input into the reference strategy model and the target strategy model respectively to obtain the first inference sequence output by the reference strategy model and the second inference sequence output by the target strategy model; The first evaluation index data corresponding to the reference strategy model is determined based on the first inference sequence, and the second evaluation index data corresponding to the target strategy model is determined based on the second inference sequence; wherein, the first evaluation index data includes at least one of fuel consumption data, reward data, battery charge deviation data, and engine start count; the second evaluation index data includes the index type in the first evaluation index data; The evaluation result corresponding to the target strategy model is determined by comparing the first evaluation index data and the second evaluation index data.

8. A training device for a control strategy model of an intelligent agent, characterized in that, The intelligent agent is applied to a vehicle energy management system, and the device includes: A sample set construction module is used to construct a training sample set; wherein, the training sample set includes several vehicle state samples; the several vehicle state samples include multiple vehicle state inference data obtained by the reference policy model performing continuous action inference on the first preset state data; each vehicle state sample corresponds to the first action inference data output by the reference policy model; the reference policy model is a model that has been trained. The action reasoning data acquisition module is used to input each vehicle state sample into the policy model to be trained, and obtain the corresponding second action reasoning data output by the policy model to be trained. The loss function type determination module is used to determine the target loss function type from multiple candidate loss function types based on the training requirements corresponding to the policy model to be trained; The target policy model acquisition module is used to determine the target loss function value based on the first action inference data output by the reference policy model for each vehicle state sample, the second action inference data output by the policy model to be trained for each vehicle state sample, and the target loss function type, so as to update the parameters of the policy model to be trained based on the target loss function value until the training stopping condition is met, thereby obtaining the target policy model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images