Hybrid off-road vehicle energy management method based on model reinforcement learning

Abstract

The invention discloses a hybrid off-road vehicle energy management method based on model reinforcement learning, and relates to the field of vehicle energy management control. The method comprises the following steps: acquiring historical information data of the hybrid off-road vehicle when the hybrid off-road vehicle is off-line; constructing a high-order Markov chain model at the current moment and training the intelligent agent by adopting a reinforcement learning method to obtain an energy management control strategy at the current moment; performing online updating on the high-order Markov chain model at the current moment by adopting a recursion method; based on the induction matrix norm, determining a matrix norm difference between the high-order Markov chain model at the current moment and the transition probability matrix corresponding to the updated high-order Markov chain model; and judging whether the norm difference of the matrix is greater than a set threshold value or not so as to update the model or control the driving condition of the hybrid off-road vehicle by adopting an energy management control strategy at the current moment. The invention aims to realize power distribution of the hybrid off-road vehicle.

Description

Technical Field

[0001] The present application relates to the field of vehicle energy management control, and in particular to a hybrid off-road vehicle energy management method based on model-based reinforcement learning. Background Art

[0002] Currently, energy management issues for hybrid vehicles, especially energy management methods based on reinforcement learning, are emerging one after another, but certain problems still exist in the specific implementation process. The performance of energy management strategies based on reinforcement learning is highly dependent on their training conditions. When the actual application conditions differ significantly from the training conditions, the performance of energy management strategies based on reinforcement learning will be severely reduced. Therefore, reinforcement learning has a good application effect in scenarios with small changes and high repeatability in the working environment, such as buses, closed environment vehicles, and rail vehicles. However, due to the complex and changeable conditions of off-road vehicles, traditional reinforcement learning methods cannot guarantee the similarity between training data and actual application conditions. Therefore, when applied, they often lack the ability to adapt to diverse conditions and find it difficult to maintain good control effects.

[0003] The energy management problem of allocating power to multiple power sources in a hybrid vehicle is a complex, multi-input, multi-objective, and nonlinear problem. Existing research has categorized the energy management approach of hybrid vehicles into two main categories: rule-based and optimization-based strategies. Rule-based strategies often control the system by setting thresholds based on engineering experience. While this control scheme is simple and easy to implement, it struggles to achieve optimal control results. In contrast, optimization-based strategies establish an objective function and employ a variety of global or real-time optimization algorithms to solve the problem, resulting in more optimal control solutions. Global optimization strategies, based on prior knowledge of the complete driving cycle, can yield a globally optimal control decision. Commonly used algorithms include Pontryagin's minimum principle (PMP), convex optimization (CO), and dynamic programming (DP). However, in real-world applications, driving cycles are often difficult to predict, so global optimization algorithms are often used as benchmarks to evaluate other strategies. Real-time optimization strategies overcome this difficulty, enabling immediate decisions based on past and instantaneous information to achieve a more optimal control strategy. Common optimization strategies include those based on model predictive control (MPC) or equivalent consumption minimization strategy (ECMS). Real-time optimization algorithms offer fast response times and are better able to make decisions under unknown operating conditions, but they cannot guarantee that their control solutions will achieve global optimality.

[0004] Beyond these two broad approaches, intelligent algorithms are also being applied to energy management due to the rapid development of machine learning and artificial intelligence. Commonly used algorithms for intelligent energy management strategies include various neural networks (NNs) and reinforcement learning (RL). Among these intelligent algorithms, RL is increasingly being applied to energy management problems due to its near-global optimality and excellent real-time performance.

[0005] Most existing hybrid vehicle energy management methods are unable to adapt to the rapid and uncertain changes in off-road operating conditions. While widely used rule-based energy management strategies are simple and easy to implement, they lack adaptability and struggle to achieve effective optimization control in rapidly changing off-road environments. Optimization-based strategies, faced with diverse off-road operating conditions, often require long computational times to explore a larger solution space and, in some cases, may even fail to converge. Energy management strategies using machine learning methods inevitably face discrepancies between training conditions and actual operating conditions. Consequently, conventional machine learning-based approaches struggle to adapt to the rapidly changing and diverse off-road conditions.

[0006] Existing hybrid vehicle energy management methods are difficult to adapt to off-road environments. Therefore, how to distribute the power of hybrid vehicles in off-road environments is crucial. Summary of the Invention

[0007] The purpose of this application is to provide a hybrid off-road vehicle energy management method based on model-based reinforcement learning, which can realize power distribution of hybrid off-road vehicles.

[0008] To achieve the above objectives, this application provides the following solutions:

[0009] This application provides a hybrid off-road vehicle energy management method based on model-based reinforcement learning, including:

[0010] Acquiring historical information data of the hybrid off-road vehicle when it is offline; the information data includes: vehicle power demand;

[0011] Constructing a high-order Markov chain model at the current moment; the high-order Markov chain model is used to determine and predict the vehicle's required power and obtain a state quantity of the hybrid off-road vehicle; the high-order Markov chain model is determined based on historical information data;

[0012] Based on the current high-order Markov chain model, the intelligent agent is trained using reinforcement learning to obtain the current energy management control strategy; the intelligent agent is the RL controller in the hybrid off-road vehicle;

[0013] The recursive method is used to update the current high-order Markov chain model online to obtain the updated high-order Markov chain model;

[0014] Determine the matrix norm difference between the transition probability matrix included in the current high-order Markov chain model and the updated high-order Markov chain model based on the induced matrix norm;

[0015] Determining whether the matrix norm difference is greater than a set threshold;

[0016] If so, the updated high-order Markov chain model is used as the high-order Markov chain model for the next moment, the next moment is updated to the current moment, and the answer is "Based on the high-order Markov chain model at the current moment, the agent is trained using the reinforcement learning method to obtain the energy management control strategy at the current moment."

[0017] If not, the energy management control strategy at the current moment is used to control the driving condition of the hybrid off-road vehicle, and the method of "using the recursive method to update the high-order Markov chain model at the current moment online to obtain the updated high-order Markov chain model" is returned.

[0018] According to the specific embodiments provided in this application, this application has the following technical effects:

[0019] The present application provides a hybrid off-road vehicle energy management method based on model-based reinforcement learning. The method comprises: obtaining offline historical information data of the hybrid off-road vehicle; constructing a high-order Markov chain model at the current moment and training an agent using a reinforcement learning method to obtain an energy management control strategy at the current moment; online updating the high-order Markov chain model at the current moment using a recursive method; determining the matrix norm difference between the transition probability matrices corresponding to the high-order Markov chain model at the current moment and the updated high-order Markov chain model based on the induced matrix norm; and determining whether the matrix norm difference is greater than a set threshold to update the model or use the energy management control strategy at the current moment to control the driving condition of the hybrid off-road vehicle. The present application uses a high-order Markov chain model to predict vehicle power demand, thereby improving adaptability. By using reinforcement learning and recursive methods, the high-order Markov chain model and energy management control strategy can be updated online to ensure that the model used during the driving of the hybrid off-road vehicle matches the current state of the vehicle, thereby achieving power allocation for the hybrid off-road vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 Flowchart of the energy management method for hybrid off-road vehicles based on model-based reinforcement learning;

[0022] Figure 2A schematic diagram of the technical concept of a hybrid off-road vehicle energy management method based on model-based reinforcement learning;

[0023] Figure 3 Schematic diagram of training and iterative updating of the intelligent agent. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] In response to the problems existing in existing solutions, this application proposes a model-based adaptive reinforcement learning method that can update the model online and achieve good energy management effects under complex working conditions.

[0026] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] In an exemplary embodiment, Figure 1 As shown, a hybrid off-road vehicle energy management method based on model-based reinforcement learning is provided, comprising:

[0028] Step 100: Obtain historical information data of the hybrid off-road vehicle when it is offline. The information data includes: vehicle power demand.

[0029] Step 200: Construct a high-order Markov chain model for the current moment. This high-order Markov chain model includes a matrix of transition probabilities. This high-order Markov chain model is used to predict the vehicle's required power and obtain the state of the hybrid off-road vehicle. This high-order Markov chain model is determined based on historical information data.

[0030] Step 300: Based on the current high-order Markov chain model, the agent is trained using reinforcement learning to obtain the current energy management control strategy. The agent is the RL controller (Reinforcement Learning Controller) in the hybrid off-road vehicle.

[0031] Step 400: Using a recursive method, the current high-order Markov chain model is updated online to obtain an updated high-order Markov chain model.

[0032] Step 500: Determine the matrix norm difference between the transition probability matrix included in the current high-order Markov chain model and the updated high-order Markov chain model based on the induced matrix norm.

[0033] Step 600: Determine whether the matrix norm difference is greater than a set threshold.

[0034] Step 700 : If yes, the updated high-order Markov chain model is used as the high-order Markov chain model for the next moment, the next moment is updated to the current moment, and the process returns to step 300 .

[0035] Step 800 : If not, the current energy management control strategy is used to control the driving condition of the hybrid off-road vehicle, and the process returns to step 400 .

[0036] In one embodiment, the transition probability is determined by counting the number of state transitions in the historical information data change trajectory based on a linear programming method; the transition probability expression is:

[0037]

[0038] The expression corresponding to the transition probability matrix is:

[0039]

[0040] Among them, P(·) is the function corresponding to the transition probability; P dem,t is the vehicle power demand at time t; l t is the vehicle state at time t; P dem,t-1 is the vehicle power demand at time t-1; l t-1 is the vehicle state at time t-1; i is the weight of the i-th order; h1 is the order of the high-order Markov chain model; is the transition probability from the kth vehicle state quantity to the jth vehicle state quantity at the i-th order; k, j∈{1,2,...,m}; m is the total number of vehicle state quantities; P dem,t-h is the required power of the vehicle at time th; l t-h is the vehicle state at time th; is the transition probability of each state of the vehicle at time th; X t is the distribution of vehicle demand power transfer probability at time t; Q i is the matrix of the i-th order transition probability corresponding to the current high-order Markov chain model; X t-i is the distribution of vehicle demand power transfer probability at time ti.

[0041] The weight λ of the i-th order i The method for determining , specifically including:

[0042]

[0043] Among them, λ is the weight parameter; Q i is the matrix of the i-th order transition probability corresponding to the high-order Markov chain model at the current moment; is the estimate of the current transition probability matrix; ||·|| ∞ is the infinity norm of the matrix.

[0044] In one embodiment, based on the current high-order Markov chain model, the agent is trained using a reinforcement learning method to obtain the current energy management control strategy, which specifically includes:

[0045] Based on the current high-order Markov chain model, the reinforcement learning method is used to train and iteratively update the agent based on the quaternion to realize the interaction between the current high-order Markov chain model and the agent, and obtain the energy management control strategy at the current moment; the quaternion is (S t ,A t ,R t ,S t+1 ).

[0046] Among them, training and iterative updating of the intelligent agent include:

[0047] At each time step, we perform:

[0048] Determine the state S based on the current high-order Markov chain model t ; Status S t It is determined based on the state quantity.

[0049] The agent is based on the state S t Choose Action A t ; Based on Action A t Determine the result of the execution. At this time, the current high-order Markov chain model is converted to the new state S t+1 , and generates a reward R and passes it to the agent.

[0050] The energy management control strategy at the current moment is determined by the estimated value of the action value function in each state; the expression of the energy management control strategy at the current moment is:

[0051]

[0052] Among them, a is the action; s is the state; π * (a|s) is the optimal strategy for performing action a under state s; q *(s,a) is the state-action q-value function obtained by performing action a under state s; A is the action set of the intelligent agent.

[0053] In one embodiment, a recursive method is used to perform online updating on the current high-order Markov chain model to obtain an updated high-order Markov chain model, specifically including:

[0054] The recursive method is used to determine the recursive result of the average frequency; the calculation formula of the recursive result is:

[0055]

[0056] The updated transition probability is obtained according to the recursive result.

[0057] The updated transition probability matrix is ​​determined based on the updated transition probability to obtain an updated high-order Markov chain model; the expression of the updated transition probability is:

[0058]

[0059] Where L is the length of the time series; A constant representing whether a transition occurs from state k to state j at time t. A value of 1 indicates that a transition occurs, and a value of 0 indicates that no transition occurs. A constant representing whether a transition occurs from state k at time t. A value of 1 indicates that a transition occurs, and a value of 0 indicates that no transition occurs. is the total number of transitions from state k to state j; is the total number of transitions from state k to state j at the previous moment; is a constant representing whether a transition occurs from state k to state j under the time series length L; is the attenuation factor; is the total number of transitions starting from state k; is the total number of transitions starting from state k at the previous moment; is the updated transition probability from the kth vehicle state quantity to the jth vehicle state quantity at the i-th order.

[0060] As an optional implementation, the expression of the induced matrix norm is:

[0061]

[0062] Where IMN(Q||Q′) is the induced matrix norm of Q and Q′; Q is the state transition matrix at the current moment; Q′ is the new state transition matrix; ||QQ′||2 is the matrix norm difference; ε i (QQ′) is the eigenvalue between Q and Q′; T is the transpose.

[0063] To determine whether the matrix norm difference is greater than the set threshold, the calculation formula used is:

[0064]

[0065] Among them, h1 is the order of the high-order Markov chain model; λ i is the weight of the i-th order; Q i is the matrix of the i-th order transition probability corresponding to the high-order Markov chain model at the current moment; Q i ′ is the matrix of the i-th order transition probability corresponding to the updated high-order Markov chain model; μ is the threshold; IMN(Q i ||Q i ′) is about Q i With Q i ′ is the induced matrix norm.

[0066] In practical applications, the method mentioned in this application is mainly divided into the following three parts.

[0067] Part I: Building a high-order Markov chain model for initial vehicle control.

[0068] Part 2: Initialize the model-based reinforcement learning control strategy according to the training conditions and the initialized Markov chain model.

[0069] Part 3: Use the recursive method online to continuously calculate the new high-order Markov chain model and compare it with the model used in the current reinforcement learning energy management strategy; when the Markov difference between the new Markov chain model and the current strategy is greater than the threshold, the reinforcement learning energy management control strategy is updated.

[0070] The overall process diagram is shown in Figure 2 . When offline, a high-order Markov chain model is constructed and the initial control strategy is calculated based on the training conditions; when online, the dynamic transition probability matrix is ​​continuously calculated using the online recursive form; the matrix norm difference between the transition probability matrix of the current model and the dynamic transition probability matrix is ​​calculated; it is determined whether the norm difference is greater than the set threshold. If so, the high-order Markov chain model is updated and the control strategy is recalculated using the reinforcement learning algorithm; if not, the previous control strategy is used; and then the system returns to "When online, the dynamic transition probability matrix is ​​continuously calculated using the online recursive form".

[0071] The corresponding operation steps are as follows.

[0072] Step 1: Initialize the current system, use past data to calculate the vehicle's Markov chain model, and initialize the reinforcement learning control law.

[0073] The vehicle's power demand is selected as the state variable to be predicted. Based on the vehicle's power demand at the previous h moments, a high-order Markov chain model is constructed to predict the power demand at the current moment. The h-order Markov chain model can be described using transition probabilities, which can be written as:

[0074]

[0075] Among them, l1,...l t ∈{1,2,...m},P dem,t is the vehicle power demand at time t; the number of all possible power demands is m, and h represents the order of the high-order Markov chain model.

[0076] The transition probability can be estimated by counting the number of state transitions in a known demand power change trajectory. In traditional high-order Markov chain models, as the order h of the Markov chain model and the number m of possible states increase, the computational burden grows exponentially and quickly becomes too large to be effectively estimated. Therefore, a new model is introduced to approximate the traditional high-order Markov chain model with fewer parameters, in which the impact of each previous state on the current state is considered separately, and the optimal parameters are obtained based on linear programming. Therefore, the transition probability can be calculated as follows:

[0077]

[0078] in, is the transition probability from the kth vehicle state quantity to the jth vehicle state quantity at the i-th order; k,j∈{1,2,...,m}; that is, Indicates that from state P dem,t-i =k to P dem,t = transition probability of j, k, j∈{1, 2, ..., m}, λ i Represents the weight of the i-th order and limits the weight to a number between 0 and 1:

[0079]

[0080] Written in matrix form, we get:

[0081]

[0082] represents the transition probability matrix (TPM) of the i-th order. dem，t-i P is the vehicle power requirement at time ti; dem，t is the vehicle power demand at time t.

[0083] By counting the number of transitions between states in the training data and using the frequency estimation probability, the elements in the transition probability matrix can be obtained. The maximum likelihood estimate of is:

[0084]

[0085] in, Represents the state P dem,t-i =k to P dem,t = the number of transitions of j, and Represents the start state P dem,t-i =k number of transitions.

[0086] λ i It can be estimated by solving the following optimal problem:

[0087]

[0088] in, Represents the proportion of each state in the known demand power trajectory. By solving this optimal problem, we can obtain the weight of the power at each moment on the power at the current moment, that is, the weight of the i-th order.

[0089] Step 2: Based on the solved Markov chain model, a model-based reinforcement learning agent can be trained to intelligently control the vehicle's driving conditions.

[0090] like Figure 3 As shown in the figure, the basic process of model-based reinforcement learning is the interaction between the agent and the model. At each time step, the agent receives the state S from the model. t , and choose action A accordingly t ; As execution A t As a result, the model transitions to a new state S t+1 , and generates a reward R and passes it to the agent; at the same time, the agent uses the four-tuple (S t ,A t ,R t ,S t+1 ) updates its own strategy through iteration. Each element in the quadruple appears alternately until the end state, which constitutes a complete learning process.

[0091] During the learning process, the actual state S t Due to sampling time constraints, acquiring sufficient data from sensors in actual vehicles for reinforcement learning algorithms to converge can take a significant amount of time. However, in model-based reinforcement learning, high-order Markov chain models are used to simulate the transitions in actual vehicle states, allowing for rapid acquisition of the desired state.

[0092] The previously proposed h-order Markov chain model is used, which takes into account the previous h states S t-h+1 ,...,S t and the current step action A t , which transitions to state S t+1 The probability can be expressed as:

[0093] p(s t+1 |s1,a1,...,s t ,a t )=p(s t+1 |s t-h+1 ,...,s t ,a t ).

[0094] p(s t+1 |s t-h+1 ,...,s t ,a t ) represents the action a t From state s t-h+1 ,...,s t To state s t+1 The transition probability of state s t-h+1 ,...,s t Defined as Then the transition probability can be written as The reward function is a function of the battery state of charge and the change in engine speed:

[0095]

[0096] Where β1,β2>0 are weight coefficients. r is the reward of the system in a single step; R is the set of rewards in a single step; is the fuel consumption of the system; ΔSoC is the change in battery state of charge; Δn e is the change in engine speed.

[0097] R t Represents the reward at a single time step, that is, at time t, and in order to measure the long-term reward, G t Represents the long-term reward, which is the cumulative result of all rewards after the time step t, and uses γ to represent the decay of its influence over time. The recursive form of the reward function is:

[0098]

[0099] The policy π is a mapping from states to the probability of choosing each possible action. γ represents the forgetting parameter, R t+k Represents the single-step reward at time t+k.

[0100] The action value function qπ (s,a) is defined as the expected reward E starting from state s, selecting action a at the current time, and following the policy π to select actions thereafter π [G t |s t =s,A t =a]:

[0101] q π (s,a)=Ε π [G t |S t =s,A t =a].

[0102] By introducing the h1-order Markov chain model into the action-value function, the recursive form of the action-value function can be derived:

[0103]

[0104] In addition, the optimal action value function q * (s,a) is defined as the maximum action value function under all strategies:

[0105]

[0106] Its recursive form is:

[0107]

[0108] Assuming that q(s,a) is known, the optimal policy π* can be obtained by taking the action a corresponding to the maximum value of q(s,a) in each state:

[0109]

[0110] Since the true optimal action-value function is unknown, it is necessary to use the estimated value q of the optimal action-value function * (s t ,a t ) is used to replace the optimal action value function. In the Q learning algorithm, the optimal action value function Q(S t ,A t ) and the optimal action value function R after the current step update t +γQ(S t+1 ,A t+1 ) to update Q(S t ,A t ), expressed as:

[0111] Q(S t ,A t )←Q(S t ,A t )+α[Rt +γQ(S t+1 ,A t+1 )-Q(S t ,A t )].

[0112] Among them, α is the learning rate parameter. t ,A t This update is performed after each transition from one state to another state-action pair. In the Q-learning algorithm, the action at the next moment is chosen to be That is, in each state, the action with the largest action value is selected.

[0113] Pseudo code of the Q-learning algorithm based on the high-order Markov chain model:

[0114] Algorithm parameters: α∈(0,1],γ∈(0,1), order h1, number of iterations N, strategy π.

[0115] Arbitrarily initialize Q(s,a), including all s∈S,a∈A.

[0116] Arbitrarily initialize S1,S2,...,S h .

[0117] For t=h,h+1,...,N, loop:

[0118] According to the strategy π and S obtained from the Q table t , we get A t .

[0119] Take A t , calculate R t ,according to Observation S t+1 .

[0120]

[0121] Finish.

[0122] Return to Q.

[0123] Step 3: Use the recursive method to update the high-order Markov chain model online, and compare it with the high-order Markov chain model used at this time. If the difference between the two is large, update the high-order Markov chain model and reinforcement learning method.

[0124] In order to achieve real-time update of the transition probability matrix, the ratio of the number of transitions between states can be replaced by the real-time updated transition frequency, which is written as:

[0125]

[0126] Where i=1,...,h1 is the order of the high-order Markov chain model. The action at this step makes the system go from state P dem,t-i =k transitions to state P dem,t =j, otherwise its value is 0, The action at this step is to go from state P dem,t-i =Starting from k. and They are and The average frequency over the total time duration.

[0127] On this basis, the recursive formula for calculating the average frequency can be derived:

[0128]

[0129] The attenuation factor is For ease of calculation, the attenuation factor can be replaced by a constant It does not affect the convergence of the recursion. This parameter is called the forgetting factor and can control the update rate of the model.

[0130] Afterwards, the online recursive form for calculating the transition probability can be derived:

[0131]

[0132] In order to quantify the similarity between two transition probability matrices and determine when to update the Markov chain model (i.e., the transition probability matrix and weights representing the model) and the reinforcement learning control strategy in real time, the induced matrix norm (IMN) is introduced, which is expressed as:

[0133]

[0134] Among them, ε i (QQ′) represents the eigenvalues ​​of the matrix QQ′. Obviously, the larger the value of IMN, the greater the difference between Q and Q′. Therefore, the scalar IMN can be used to calculate the difference between the transition probability matrices.

[0135] For high-order Markov chain models, IMN is used as follows:

[0136]

[0137] Where μ is the threshold. Once IMN exceeds the threshold μ, the old transition probability matrix Q i Replaced with the updated transition probability matrix Q i ′ and recalculate the weight λ iTo update the high-order Markov chain model. In addition, it is necessary to re-calculate the control strategy based on the new model using the reinforcement learning method, which is exactly the same as the aforementioned learning method.

[0138] This application uses a high-order Markov chain model to predict the required power in off-road environments, which effectively simplifies the prediction of required power and improves its adaptability, ensuring that the model used during vehicle driving matches the current state of the vehicle; this application also uses reinforcement learning methods to achieve online updates of vehicle control strategies.

[0139] This application uses machine learning methods, such as reinforcement learning, to control the vehicle, enabling more detailed optimization control than rule-based control methods. Compared to optimization-based control strategies, the use of predictions from high-order Markov chain models enables faster control speeds and a larger solution space search. Furthermore, to address the difficulty of traditional methods in adapting to diverse off-road conditions, a method has been designed to update the high-order Markov chain model in real time based on the vehicle's driving state. When the high-order Markov chain model changes beyond a threshold, the model and the reinforcement learning control strategy based on it are updated to improve adaptability in complex operating environments.

[0140] In addition, the solution of this application can also use adaptive model predictive control (AMPC) to achieve model updating and control, but its optimal control effect may not be as good as the control effect obtained by the reinforcement learning method; in addition, a control method involving neural networks can also be used to achieve the update and control of the model mentioned in this application, but the computing resources required are large and there is a certain degree of difficulty in implementation on the vehicle.

[0141] Based on the same inventive concept, embodiments of the present application also provide a hybrid off-road vehicle energy management device based on model-based reinforcement learning, for implementing the aforementioned hybrid off-road vehicle energy management method based on model-based reinforcement learning. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the hybrid off-road vehicle energy management device based on model-based reinforcement learning provided below can be found in the aforementioned limitations of the hybrid off-road vehicle energy management method based on model-based reinforcement learning, and will not be further elaborated here.

[0142] In an exemplary embodiment, a hybrid off-road vehicle energy management device based on model-based reinforcement learning is provided, comprising:

[0143] The information data acquisition module is used to acquire historical information data of the hybrid off-road vehicle when it is offline; the information data includes: vehicle required power.

[0144] The model building module is used to construct a high-order Markov chain model at the current moment. The high-order Markov chain model is used to determine the predicted vehicle power demand and obtain the state quantity of the hybrid off-road vehicle.

[0145] The control strategy determination module is used to train the intelligent agent based on the current high-order Markov chain model using reinforcement learning to obtain the current energy management control strategy; the intelligent agent is the RL controller in the hybrid off-road vehicle.

[0146] The model updating module is used to update the current high-order Markov chain model online using a recursive method to obtain an updated high-order Markov chain model.

[0147] The matrix norm difference calculation module is used to determine the matrix norm difference between the current high-order Markov chain model and the transition probability matrix included in the updated high-order Markov chain model based on the induced matrix norm.

[0148] The judgment module is used to judge whether the matrix norm difference is greater than the set threshold. When the judgment result is yes, the updated high-order Markov chain model is used as the high-order Markov chain model of the next moment, the next moment is updated to the current moment, and the module returns to the "control strategy determination module". When the judgment result is no, the energy management control strategy of the current moment is used to control the driving condition of the hybrid off-road vehicle, and the module returns to the "model update module".

[0149] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a hybrid off-road vehicle energy management method based on model-based reinforcement learning is implemented.

[0150] Those skilled in the art will understand that the structure shown above is only a partial structure related to the solution of the present application and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than those shown above, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the above-mentioned method embodiments are implemented.

[0151] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0152] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0153] In this application, all actions to obtain signals, information, or data are performed in compliance with the relevant data protection laws and policies of the country in which they are located and with the authorization of the corresponding device owner. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with relevant regulations.

[0154] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0155] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0156] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A hybrid off-road vehicle energy management method based on model-based reinforcement learning, characterized in that: include: Obtaining historical information data of the hybrid off-road vehicle when it is offline; The information data includes: vehicle required power; Constructing a high-order Markov chain model at the current moment; the high-order Markov chain model is used to determine and predict the vehicle's required power and obtain a state quantity of the hybrid off-road vehicle; the high-order Markov chain model is determined based on historical information data; Based on the current high-order Markov chain model, the intelligent agent is trained using reinforcement learning to obtain the current energy management control strategy; the intelligent agent is the RL controller in the hybrid off-road vehicle; The recursive method is used to update the current high-order Markov chain model online to obtain the updated high-order Markov chain model; Determine the matrix norm difference between the transition probability matrix included in the current high-order Markov chain model and the updated high-order Markov chain model based on the induced matrix norm; Determining whether the matrix norm difference is greater than a set threshold; If so, the updated high-order Markov chain model is used as the high-order Markov chain model for the next moment, the next moment is updated to the current moment, and the answer is "Based on the high-order Markov chain model at the current moment, the agent is trained using reinforcement learning to obtain the energy management control strategy at the current moment." If not, the current energy management control strategy is used to control the driving condition of the hybrid off-road vehicle, and the method "use the recursive method to update the current high-order Markov chain model online to obtain the updated high-order Markov chain model" is returned.

2. The hybrid off-road vehicle energy management method based on model-based reinforcement learning according to claim 1, characterized in that: The transition probability is determined based on a linear programming method by counting the number of state transitions in the historical information data change trajectory; the expression of the transition probability is: The expression corresponding to the transition probability matrix is: Among them, P(·) is the function corresponding to the transition probability; P dem,t is the vehicle power demand at time t; l t is the vehicle state at time t; P dem,t-1 is the vehicle power demand at time t-1; l t-1 is the vehicle state at time t-1; i is the weight of the i-th order; h1 is the order of the high-order Markov chain model; is the transition probability from the kth vehicle state quantity to the jth vehicle state quantity at the i-th order; k, j∈{1,2,...,m}; m is the total number of vehicle state quantities; P dem,t-h is the required power of the vehicle at time th; l t-h is the vehicle state at time th; is the transition probability of each state of the vehicle at time th; X t is the distribution of vehicle demand power transfer probability at time t; Q i is the matrix of the i-th order transition probability corresponding to the current high-order Markov chain model; X t-i is the distribution of vehicle demand power transfer probability at time ti.

3. The hybrid off-road vehicle energy management method based on model-based reinforcement learning according to claim 2, characterized in that: The weight λ of the i-th order i The method for determining , specifically including: Among them, λ is the weight parameter; is the estimate of the current transition probability matrix; ||·|| ∞ is the infinity norm of the matrix.

4. The hybrid off-road vehicle energy management method based on model-based reinforcement learning according to claim 1, characterized in that: Based on the current high-order Markov chain model, the agent is trained using reinforcement learning to obtain the energy management control strategy at the current moment, which includes: Based on the current high-order Markov chain model, the reinforcement learning method is used to train and iteratively update the agent based on the quaternion to realize the interaction between the current high-order Markov chain model and the agent, and obtain the energy management control strategy at the current moment; the quaternion is (S t ,A t ,R t ,S t+1 ); Among them, training and iterative updating of the intelligent agent include: At each time step, we perform: Determine the state S based on the current high-order Markov chain model t ; Status S t It is determined based on the state quantity; The agent is based on the state S t Choose Action A t ; Based on Action A t Determine the result of the execution. At this time, the current high-order Markov chain model is converted to the new state S t+1 , and generates a reward R and passes it to the agent.

5. The hybrid off-road vehicle energy management method based on model-based reinforcement learning according to claim 4, characterized in that: The energy management control strategy at the current moment is determined by the estimated value of the action value function in each state; the expression of the energy management control strategy at the current moment is: Among them, a is the action; s is the state; π * (a|s) is the optimal strategy for performing action a under state s; q * (s,a) is the state-action q-value function obtained by performing action a under state s; A is the action set of the intelligent agent.

6. The hybrid off-road vehicle energy management method based on model-based reinforcement learning according to claim 1, characterized in that: The recursive method is used to update the current high-order Markov chain model online to obtain the updated high-order Markov chain model, which specifically includes: The recursive method is used to determine the recursive result of the average frequency; the calculation formula of the recursive result is: Obtaining an updated transition probability according to the recursive result; The updated transition probability matrix is ​​determined based on the updated transition probability to obtain an updated high-order Markov chain model; the expression of the updated transition probability is: Where L is the length of the time series; A constant representing whether a transition occurs from state k to state j at time t. A value of 1 indicates that a transition occurs, and a value of 0 indicates that no transition occurs. A constant representing whether a transition occurs from state k at time t. A value of 1 indicates that a transition occurs, and a value of 0 indicates that no transition occurs. is the total number of transitions from state k to state j; is the total number of transitions from state k to state j at the previous moment; is a constant representing whether a transition occurs from state k to state j under the time series length L; is the attenuation factor; is the total number of transitions starting from state k; is the total number of transitions starting from state k at the previous moment; is the updated transition probability from the kth vehicle state quantity to the jth vehicle state quantity at the i-th order.

7. The hybrid off-road vehicle energy management method based on model-based reinforcement learning according to claim 1, characterized in that: The expression of the induced matrix norm is: Where IMN(Q||Q′) is the induced matrix norm of Q and Q′; Q is the state transition matrix at the current moment; Q′ is the new state transition matrix; ||QQ′||2 is the matrix norm difference; ε i (QQ′) is the eigenvalue between Q and Q′; T is the transpose.

8. The hybrid off-road vehicle energy management method based on model-based reinforcement learning according to claim 7, characterized in that: To determine whether the matrix norm difference is greater than a set threshold, the calculation formula used is: Among them, h1 is the order of the high-order Markov chain model; λ i is the weight of the i-th order; Q i is the matrix of the i-th order transition probability corresponding to the current high-order Markov chain model; Q i ′ is the matrix of the i-th order transition probability corresponding to the updated high-order Markov chain model; μ is the threshold; IMN(Q i ||Q i ′) is about Q i With Q i ′ is the induced matrix norm.

Images