Energy management optimization method for plug-in hybrid electric vehicle based on deep reinforcement learning

Abstract

This invention discloses a method for optimizing energy management for plug-in hybrid electric vehicles based on deep reinforcement learning. The method comprises the following steps: Step 1: Collecting data and generating a state feature dataset; Step 2: Constructing a potential energy tensor graph; Step 3: Constructing a modal jump graph and generating a modal jump path structure; Step 4: Inputting the modal jump path structure into a piecewise policy jump learner and outputting jump policy samples; Step 5: Using a jump Bellman function to estimate the value of state-mode pairs, embedding jump policy samples into the modal subspace tensor graph, and dynamically modeling the modal mapping relationship using a variable structure neural network to obtain a jump policy model; Step 6: Invoking the jump policy model to generate a smoothed modal control trajectory; and Step 7: Applying the smoothed modal control trajectory to the vehicle control system. This invention combines the jump Bellman function with a variable structure neural network to achieve efficient modal policy decision-making and energy-optimized control for plug-in hybrid vehicles.

Description

Technical Field

[0001] The present invention relates to the field of new energy vehicle control and energy management technology, and in particular to a plug-in hybrid vehicle energy management optimization method based on deep reinforcement learning. Background Art

[0002] With the increasing intelligence of new energy vehicles, energy management and mode switching control for plug-in hybrid vehicles have become key research directions for energy-saving optimization. Existing energy management methods are mostly based on preset rules, model predictive control, or continuous strategy reinforcement learning. However, the following problems are common in practical applications:

[0003] Traditional methods fail to fully consider the discreteness and structural constraints of modal transitions, and their policy space expression capabilities are limited, making it difficult to adapt to complex road environments and multi-modal coupling decision-making needs; existing reinforcement learning methods generally adopt continuous policy modeling, ignoring the jumping characteristics of modal switching behavior, resulting in unstable policies or frequent jitters in modal execution; there is a lack of clear modeling paths for the cost between states and modes, the policy learning objectives are disconnected from the actual energy consumption of the vehicle, and the optimization results lack physical explanations; at the same time, the relationship between modal transition frequency and energy consumption fluctuations has not been explicitly modeled, which often leads to problems such as high-frequency switching, system response delays, and increased overall energy consumption.

[0004] Therefore, how to provide a plug-in hybrid vehicle energy management optimization method based on deep reinforcement learning is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] One objective of the present invention is to propose an energy management optimization method for plug-in hybrid electric vehicles based on deep reinforcement learning. The present invention integrates the jump Bellman function and the variable structure neural network, constructs a discontinuous strategy space and introduces a modal transition gating mechanism to realize the modal control decision and energy distribution optimization of the plug-in hybrid electric vehicle under complex operating conditions. It has the advantages of precise strategy response, smooth modal transition and low vehicle energy consumption, and significantly improves the vehicle's adaptive energy management capability in a multi-modal coupling environment.

[0006] A plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to an embodiment of the present invention includes the following steps:

[0007] Step 1: Collect the operating status data of the plug-in hybrid vehicle and generate a status feature data set;

[0008] Step 2: constructing a dynamic potential energy field model based on the state characteristic data set to obtain a potential energy tensor diagram covering a multi-dimensional state space, wherein each state point corresponds to a potential energy value;

[0009] Step 3: Constructing a modal jump graph based on the potential energy tensor graph, defining the vehicle operation mode as a discrete node, establishing an inter-modal jump path based on the potential energy gradient of the state transition, and generating a modal jump path structure;

[0010] Step 4: Input the modal jump path structure into the segmented policy jump learner to construct a discontinuous policy space, and limit the modal jump action to be executed only when the corresponding state meets the potential energy transition threshold, and output the jump policy sample;

[0011] Step 5: Use the jump Bellman function to estimate the value of the state-mode pair, embed the jump strategy sample into the modal subspace tensor graph, and use a variable structure neural network to dynamically model the modal mapping relationship to obtain a trained jump strategy model;

[0012] Step 6: Calling the jump strategy model, inferring the real-time operating state of the vehicle, outputting a modal jump sequence, and smoothing the modal jump sequence based on the modal switching frequency and energy consumption changes to generate a smoothed modal control trajectory;

[0013] Step 7: Apply the smooth modal control trajectory to the vehicle control system to achieve mode switching and energy distribution of the plug-in hybrid vehicle under different operating conditions.

[0014] Optionally, the operating status data includes battery state of charge, electric drive power, engine output power, vehicle speed, longitudinal acceleration, geographic location information, road slope, traffic flow level, traffic light cycle, and driver's accelerator pedal input amplitude and frequency.

[0015] Optionally, the state feature data set is generated by performing timestamp alignment, missing value interpolation and noise filtering on the operating state data, and segmenting the continuous data based on a set time sliding window to extract statistical features, rate of change features, extreme value features and frequency domain features from each segment of data, forming feature vectors according to a unified coding format, and constructing a state feature data set.

[0016] Optionally, the step 2 is specifically as follows:

[0017] According to the state characteristic data set, a dynamic potential energy field model is constructed using a mapping relationship between vehicle energy consumption and state variables;

[0018] Each sample in the state feature data set is regarded as a state point in the multidimensional state space, and the energy consumption cost value of the corresponding state point is calculated by combining the battery power change value, engine fuel consumption change value and mechanical kinetic energy loss change value per unit time corresponding to the state point;

[0019] Normalizing the energy consumption cost value and converting it into a potential energy value to reflect the energy consumption trend and the difficulty of the drive system mode transition at the state point;

[0020] By performing grid interpolation construction on all state points, a continuous potential energy tensor diagram is generated in the multidimensional state space, where each state point serves as a grid point in the potential energy tensor diagram and corresponds to a unique potential energy value.

[0021] Optionally, the step three is specifically as follows:

[0022] The operating modes of the plug-in hybrid vehicle are defined as discrete nodes of the modal jump diagram, wherein the operating modes include pure electric drive mode, engine drive mode, hybrid drive mode and energy recovery mode;

[0023] Mapping each state point in the potential energy tensor graph to the discrete nodes, and determining the mapping relationship between the state points and the discrete nodes using the potential energy value corresponding to each state point under each discrete node;

[0024] For each state point in the potential energy tensor graph, calculating the potential energy difference between the state point at the current discrete node and other discrete nodes, and determining the potential energy gradient of each state point between modes;

[0025] A potential energy gradient threshold is set, and the potential energy gradient of the state point between discrete nodes is compared with the potential energy gradient threshold. When the potential energy gradient is less than the potential energy gradient threshold, a jump path between the discrete nodes is constructed as a directed edge of the modal jump graph. The direction of the directed edge represents the allowable modal transition direction, and the weight of the directed edge represents the corresponding potential energy gradient during the modal transition process. A modal jump path structure containing potential energy gradient information is generated to characterize the allowable modal transition relationship of the plug-in hybrid vehicle under different states.

[0026] Optionally, the segmented policy jump learner is used to construct a discontinuous policy space and output jump policy samples that meet the jump conditions, specifically:

[0027] A segmented modal-action mapping structure is established, in which each pair of modal transition combinations in the modal jump path structure is defined as a discrete modal transition action and mapped to an independent strategy subspace, forming multiple discontinuous state-action mapping segments.

[0028] Define the modal transition gating function to activate the discrete modal transition action in any state. In modal With modal The difference in potential energy under and preset potential energy transition threshold Compare, when satisfied , the discrete modal transition action is allowed to be activated and executed; otherwise, the discrete modal transition action is disabled in the current state;

[0029] According to the current state, the effective modal jump action that meets the potential energy transition threshold condition is screened out from all discrete modal jump actions, and is combined with the current state to form a jump strategy sample, which is saved in the sample buffer.

[0030] Optionally, the jump Bellman function is used to estimate the value of the state-mode pair, and the jump strategy sample is embedded in the modal subspace tensor graph, specifically:

[0031] The long-term energy cost of the state and mode combination is estimated using the jump Bellman function. The jump Bellman function takes the state, current mode, target mode, and modal transition cost in each jump strategy sample as input data and calculates the total expected energy consumption brought about by the future modal sequence when adopting a given modal jump behavior in the current state. The modal transition cost is the difference in potential energy value of the state between the current mode and the target mode.

[0032] In the jump Bellman function calculation, for each combination of state and current mode, all target modes that are allowed to jump are traversed, and the corresponding mode transition cost in the jump strategy sample is extracted as the immediate cost. Combined with the value information contained in the jump strategy sample of the next state corresponding to the target mode, the long-term cost of the state-mode combination is estimated through a weighted accumulation method. The long-term cost reflects the overall energy consumption level caused by the vehicle executing the mode and mode sequence in the current state;

[0033] All state-mode combinations in the jump strategy sample are constructed as nodes in the modal subspace tensor graph. Each node contains state characteristics, current modal category, immediate modal transition cost and corresponding long-term cost value. The connection relationship between nodes is constructed according to the modal jump pairs actually occurring in the jump strategy sample. Each connecting edge represents an executed modal transition action, and the edge weight represents the cost level and frequency of the modal transition action in the actual execution process.

[0034] Optionally, the variable structure neural network includes a neural ordinary differential equation modeling module and a self-attention mechanism modeling module;

[0035] The input of the neural ordinary differential equation modeling module is the eigenvector corresponding to each node in the modal subspace tensor graph, and the eigenvector includes the potential energy value, state characteristics, modal category label and long-term cost value calculated according to the jump Bellman function of the state corresponding to the node;

[0036] The neural ordinary differential equation modeling module uses node feature vectors as state variables in continuous time and time as the external driving dimension, and adopts numerical methods to approximate the ordinary differential equation model to obtain the evolution trajectory of the node long-term cost value at different time steps, realizes the modeling of the long-term cost value evolution process of the state-mode combination over time, and outputs the predicted long-term cost value sequence of each node in the next several time steps;

[0037] The input of the self-attention mechanism modeling module is the feature vectors and connection edge information of all nodes in the modal subspace tensor graph. The connection edge records the actual modal transition behavior between nodes in the historical jump path and contains the corresponding transition frequency and cost as weight information;

[0038] The self-attention mechanism modeling module calculates the attention weight between each node and all connected nodes, dynamically adjusts the response degree of each node to the adjacent modal jump path, and realizes modal transition relationship modeling based on historical structure information;

[0039] The jump strategy samples are input into the variable structure neural network in batches, and the deviation between the predicted long-term cost value and the calculated value of the jump Bellman function is minimized as the loss target. Repeated training is performed until the loss function converges, and the trained jump strategy model is output.

[0040] Optionally, the step six is ​​specifically as follows:

[0041] The trained jump strategy model is called, and the vehicle's current real-time operating state is used as input. The model infers the modal jump actions under the real-time operating state, forming a state-modal sequence output, where each record represents the vehicle's recommended modal jump path in a continuous time period, generating a modal jump sequence.

[0042] Counting the mode switching frequencies in continuous time periods in the mode hopping sequence, and calculating the number of mode transitions per unit time;

[0043] Evaluate the energy consumption change rate before and after each mode transition as a measure of the impact of mode switching behavior on the energy consumption of the plug-in hybrid vehicle;

[0044] A mode switching suppression factor is set. The mode switching suppression factor includes two sub-items: a mode transition frequency threshold and an energy consumption fluctuation tolerance threshold. The mode transition frequency threshold is defined as the maximum number of mode jumps allowed within a unit time window, and the energy consumption fluctuation tolerance threshold is defined as the maximum allowable range of the rate of change of energy consumption per unit time before and after a mode transition. In the mode jump sequence, if the number of mode jumps in a continuous time period exceeds the mode transition frequency threshold, or the rate of change of energy consumption is greater than the energy consumption fluctuation tolerance threshold, the time period is marked as an abnormal transition period.

[0045] Performing a window sliding average smoothing process on the abnormal transition segment, specifically comprising: sliding a modal value sequence on the modal jump sequence with a preset window length, calculating the mode of the modal value occurrence frequency in each window, and replacing the modal instruction at the center of the window with the mode, thereby replacing and delaying frequent transition behaviors, and replacing the original path by locally searching for a modal sequence with the minimum energy consumption cost;

[0046] The smoothed modal jump sequence is used as the execution instruction of the vehicle control system to generate the final smooth modal control trajectory.

[0047] Optionally, the step seven of applying the smooth modal control trajectory to a vehicle control system to achieve mode switching and energy distribution of the plug-in hybrid vehicle under different operating states is as follows:

[0048] Inputting the smooth modal control trajectory into a vehicle control system as a time series modal command sequence, wherein the smooth modal control trajectory includes target modes and control command parameters corresponding to each time step;

[0049] In each control cycle, the vehicle control system reads the target mode of the corresponding time step in the smooth modal control trajectory according to the current real-time state information of the vehicle;

[0050] According to the target mode, the engine control unit, the electric drive control unit and the energy recovery system are dynamically called to coordinate the drive source power output and the energy flow path according to the planning content of the modal control trajectory;

[0051] When the target mode is pure electric drive mode, the vehicle control system turns off the engine and only controls the motor output to meet the driving demand power;

[0052] When the target mode is the engine drive mode, the vehicle control system uses the engine to directly drive the vehicle, and the electric drive system remains disengaged or unloaded;

[0053] When the target mode is hybrid drive mode, the vehicle control system coordinates the engine and motor outputs to jointly drive to meet the vehicle power requirements;

[0054] When the target mode is the energy recovery mode, the vehicle control system controls the motor to operate in the power generation state, converting the mechanical energy during braking or coasting into electrical energy and feeding it back to the battery system.

[0055] The beneficial effects of the present invention are:

[0056] The present invention introduces a jump Bellman function and a piecewise policy jump learning structure to address the problems of policy discontinuity, frequent jumps, and fuzzy cost assessment in the modal transition control of plug-in hybrid vehicles under complex operating conditions. By constructing a discontinuous policy space with potential energy constraints, the present invention precisely defines the modal transition boundaries based on a multi-modal potential energy tensor graph, and filters state-dependent transition behaviors through a gating function to ensure the physical feasibility and energy rationality of policy activation. In the policy modeling process, a modal subspace tensor graph embedding structure is combined with a jump Bellman value estimation to perform structure-aware dynamic modeling of the long-term energy consumption cost of each state-modal combination. Furthermore, a variable structure neural network consisting of a neural ordinary differential equation and a self-attention mechanism is introduced to capture the temporal evolution of the modal transition value and the strength of cross-modal dependencies, respectively, to achieve fine generalization of the jump strategy in a high-dimensional state space. In the control output stage, a mode switching suppression factor is set and integrated with a local sliding average smoothing strategy to effectively suppress modal jitter and invalid transitions, ultimately improving the response stability and energy optimization performance of the vehicle's modal control. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0058] Figure 1 This is the overall flow chart of the plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning proposed by the present invention;

[0059] Figure 2 This is a flowchart of the jump Bellman function estimation of the plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning proposed by the present invention;

[0060] Figure 3 This is a schematic diagram of the structure of the variable structure neural network of the plug-in hybrid vehicle energy management optimization method based on deep reinforcement learning proposed in the present invention. DETAILED DESCRIPTION

[0061] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0062] refer to Figure 1-Figure 3 The energy management optimization method for plug-in hybrid electric vehicles based on deep reinforcement learning includes the following steps:

[0063] Step 1: Collect the operating status data of the plug-in hybrid vehicle and generate a status feature data set;

[0064] Step 2: constructing a dynamic potential energy field model based on the state characteristic data set to obtain a potential energy tensor diagram covering a multi-dimensional state space, wherein each state point corresponds to a potential energy value;

[0065] Step 3: Constructing a modal jump graph based on the potential energy tensor graph, defining the vehicle operation mode as a discrete node, establishing an inter-modal jump path based on the potential energy gradient of the state transition, and generating a modal jump path structure;

[0066] Step 4: Input the modal jump path structure into the segmented policy jump learner to construct a discontinuous policy space, and limit the modal jump action to be executed only when the corresponding state meets the potential energy transition threshold, and output the jump policy sample;

[0067] Step 5: Use the jump Bellman function to estimate the value of the state-mode pair, embed the jump strategy sample into the modal subspace tensor graph, and use a variable structure neural network to dynamically model the modal mapping relationship to obtain a trained jump strategy model;

[0068] Step 6: Calling the jump strategy model, inferring the real-time operating state of the vehicle, outputting a modal jump sequence, and smoothing the modal jump sequence based on the modal switching frequency and energy consumption changes to generate a smoothed modal control trajectory;

[0069] Step 7: Apply the smooth modal control trajectory to the vehicle control system to achieve mode switching and energy distribution of the plug-in hybrid vehicle under different operating conditions.

[0070] In this embodiment, the operating status data includes battery state of charge, electric drive power, engine output power, vehicle speed, longitudinal acceleration, geographic location information, road slope, traffic flow level, traffic light cycle, and driver accelerator pedal input amplitude and frequency.

[0071] In this embodiment, the state feature data set is generated by performing timestamp alignment, missing value interpolation and noise filtering on the operating state data, and segmenting the continuous data based on a set time sliding window to extract statistical features, rate of change features, extreme value features and frequency domain features from each segment of data, forming feature vectors in a unified coding format, and constructing a state feature data set.

[0072] Statistical characteristics, including mean, standard deviation, and median, indicate the central tendency and dispersion of each signal;

[0073] Rate of change characteristics, including first-order difference and acceleration slope, indicate the speed of signal change;

[0074] Extreme value features, including maximum and minimum values, represent the upper and lower limits of the signal;

[0075] Frequency domain features, including fast Fourier transform (FFT) of the accelerator pedal input and vehicle total power signals, extracting the amplitude and energy distribution of the first N orders of main frequencies.

[0076] In this embodiment, the step 2 is specifically as follows:

[0077] According to the state characteristic data set, a dynamic potential energy field model is constructed using a mapping relationship between vehicle energy consumption and state variables;

[0078] Each sample in the state feature data set is regarded as a state point in the multidimensional state space, and the energy consumption cost value of the corresponding state point is calculated by combining the battery power change value, engine fuel consumption change value and mechanical kinetic energy loss change value per unit time corresponding to the state point;

[0079] ;

[0080] in, Indicates the The energy consumption cost value corresponding to each state point is in kWh. Indicates the The battery charge change value per unit time at each state point. A positive value indicates discharge. The unit is kWh. Indicates the The change in engine fuel consumption per unit time at each state point is in kWh. Indicates the change in mechanical kinetic energy loss due to acceleration and deceleration per unit time at the state point, in kWh. 、 and Represents the weight coefficient, satisfying ;

[0081] Normalizing the energy consumption cost value and converting it into a potential energy value to reflect the energy consumption trend and the difficulty of the drive system mode transition at the state point;

[0082] ;

[0083] in, Indicates the The potential energy value corresponding to each state point in the potential energy field is: Represents the minimum energy consumption cost value among all state points, Represents the maximum energy consumption cost value among all state points;

[0084] By performing grid interpolation construction on all state points, a continuous potential energy tensor diagram is generated in the multidimensional state space, where each state point serves as a grid point in the potential energy tensor diagram and corresponds to a unique potential energy value.

[0085] In this embodiment, the step three is specifically as follows:

[0086] The operating modes of the plug-in hybrid vehicle are defined as discrete nodes of the modal jump diagram, wherein the operating modes include pure electric drive mode, engine drive mode, hybrid drive mode and energy recovery mode;

[0087] Mapping each state point in the potential energy tensor graph to the discrete nodes, and determining the mapping relationship between the state points and the discrete nodes using the potential energy value corresponding to each state point under each discrete node;

[0088] For each state point in the potential energy tensor graph, calculating the potential energy difference between the state point at the current discrete node and other discrete nodes, and determining the potential energy gradient of each state point between modes;

[0089] By simulating the execution of different modal operation strategies at the same state point, the corresponding energy consumption cost value is calculated and converted into the potential energy value under each mode;

[0090] A potential energy gradient threshold is set, and the potential energy gradient of the state point between discrete nodes is compared with the potential energy gradient threshold. When the potential energy gradient is less than the potential energy gradient threshold, a jump path between the discrete nodes is constructed as a directed edge of the modal jump graph. The direction of the directed edge represents the allowable modal transition direction, and the weight of the directed edge represents the corresponding potential energy gradient during the modal transition process. A modal jump path structure containing potential energy gradient information is generated to characterize the allowable modal transition relationship of the plug-in hybrid vehicle under different states.

[0091] In this embodiment, the segmented policy jump learner is used to construct a discontinuous policy space and output jump policy samples that meet the jump conditions, specifically:

[0092] A segmented modal-action mapping structure is established, in which each pair of modal transition combinations in the modal jump path structure is defined as a discrete modal transition action and mapped to an independent strategy subspace, forming multiple discontinuous state-action mapping segments.

[0093] Define the modal transition gating function to activate the discrete modal transition action in any state. In modal With modal The difference in potential energy under and preset potential energy transition threshold Compare, when satisfied , the discrete modal transition action is allowed to be activated and executed; otherwise, the discrete modal transition action is disabled in the current state;

[0094] According to the current state, the effective modal jump action that meets the potential energy transition threshold condition is screened out from all discrete modal jump actions, and is combined with the current state to form a jump strategy sample, which is saved in the sample buffer.

[0095] In this embodiment, the jump Bellman function is used to estimate the value of the state-mode pair, and the jump strategy sample is embedded in the modal subspace tensor graph, specifically:

[0096] The long-term energy cost of the state and mode combination is estimated using the jump Bellman function. The jump Bellman function takes the state, current mode, target mode, and modal transition cost in each jump strategy sample as input data and calculates the total expected energy consumption brought about by the future modal sequence when adopting a given modal jump behavior in the current state. The modal transition cost is the difference in potential energy value of the state between the current mode and the target mode.

[0097] In the jump Bellman function calculation, for each combination of state and current mode, all target modes that are allowed to jump are traversed, and the corresponding mode transition cost in the jump strategy sample is extracted as the immediate cost. Combined with the value information contained in the jump strategy sample of the next state corresponding to the target mode, the long-term cost of the state-mode combination is estimated through a weighted accumulation method. The long-term cost reflects the overall energy consumption level caused by the vehicle executing the mode and mode sequence in the current state;

[0098] ;

[0099] in, Indicates status Lower mode The long-term cost value, Indicates that the modal Transition to Mode The jumping action, represents the current policy distribution, represents the mathematical expectation, represents the discount factor, Indicates the execution of modal transition action When in state The instantaneous modal transition cost generated under , is equivalent to the potential energy difference between modes, Indicates the next state The minimum long-term cost value among all target modes is, Indicates the next state after executing the jump action. Indicates the next state The set of target modes allowed under

[0100] All state-mode combinations in the jump strategy sample are constructed as nodes in the modal subspace tensor graph. Each node contains state characteristics, current modal category, immediate modal transition cost and corresponding long-term cost value. The connection relationship between nodes is constructed according to the modal jump pairs actually occurring in the jump strategy sample. Each connecting edge represents an executed modal transition action, and the edge weight represents the cost level and frequency of the modal transition action in the actual execution process.

[0101] In this embodiment, the variable structure neural network includes a neural ordinary differential equation modeling module and a self-attention mechanism modeling module;

[0102] The input of the neural ordinary differential equation modeling module is the eigenvector corresponding to each node in the modal subspace tensor graph, and the eigenvector includes the potential energy value, state characteristics, modal category label and long-term cost value calculated according to the jump Bellman function of the state corresponding to the node;

[0103] The neural ordinary differential equation modeling module uses node feature vectors as state variables in continuous time and time as the external driving dimension, and adopts numerical methods to approximate the ordinary differential equation model to obtain the evolution trajectory of the node long-term cost value at different time steps, realizes the modeling of the long-term cost value evolution process of the state-modal combination over time, and outputs the predicted long-term cost value sequence of each node in the next several time steps to support the value evaluation of cross-time modal jump paths;

[0104] The input of the self-attention mechanism modeling module is the feature vectors and connection edge information of all nodes in the modal subspace tensor graph. The connection edge records the actual modal transition behavior between nodes in the historical jump path and contains the corresponding transition frequency and cost as weight information;

[0105] The self-attention mechanism modeling module calculates the attention weight between each node and all connected nodes, dynamically adjusts the response degree of each node to the adjacent modal jump path, and realizes modal transition relationship modeling based on historical structure information;

[0106] In the present invention, in order to achieve detailed modeling of the modal transition structure of the plug-in hybrid vehicle, a self-attention mechanism modeling module is designed to process the node relationships in the modal subspace tensor graph. Its core function is to automatically identify and adjust the degree of attention of each state-modal node to its adjacent jump path, thereby realizing structural perception modeling of the modal transition relationship.

[0107] In the self-attention mechanism modeling module, each state-modality node uses its feature vector as the query vector (Query) and the feature vectors of its neighboring nodes as the key (Key) and value (Value) inputs. An attention weight matrix is ​​generated through a dot-product attention calculation. The attention weights are used to measure the current node's response strength to its adjacent modal jump paths. The features of all neighboring nodes are weighted and summed according to the attention weights to serve as the node's structurally enhanced feature representation. The updated node features retain the original state and modal information while also incorporating the modal jump history structure into the node's importance assessment, thereby achieving the goal of dynamically adjusting the degree of response to modal transition paths.

[0108] The jump strategy samples are input into the variable structure neural network in batches, and the deviation between the predicted long-term cost value and the calculated value of the jump Bellman function is minimized as the loss target. Repeated training is performed until the loss function converges, and the trained jump strategy model is output.

[0109] In this embodiment, the step six is ​​specifically as follows:

[0110] The trained jump strategy model is called, and the current real-time operating state of the vehicle is used as input. The modal jump action under the real-time operating state is inferred, forming a state-modal sequence output, where each record represents the modal jump path recommended by the vehicle in a continuous time period, generating a modal jump sequence.

[0111] The modal jump sequence is smoothed, comprising:

[0112] Counting the mode switching frequencies in continuous time periods in the mode hopping sequence, and calculating the number of mode transitions per unit time;

[0113] Evaluate the energy consumption change rate before and after each mode transition as a measure of the impact of mode switching behavior on the energy consumption of the plug-in hybrid vehicle;

[0114] A mode switching suppression factor is set. The mode switching suppression factor includes two sub-items: a mode transition frequency threshold and an energy consumption fluctuation tolerance threshold. The mode transition frequency threshold is defined as the maximum number of mode jumps allowed within a unit time window, and the energy consumption fluctuation tolerance threshold is defined as the maximum allowable range of the rate of change of energy consumption per unit time before and after a mode transition. In the mode jump sequence, if the number of mode jumps in a continuous time period exceeds the mode transition frequency threshold, or the rate of change of energy consumption is greater than the energy consumption fluctuation tolerance threshold, the time period is marked as an abnormal transition period.

[0115] A window sliding average smoothing process is performed on the abnormal transition segment, specifically including: sliding a modal value sequence on the modal jump sequence with a preset window length, calculating the mode of the modal value occurrence frequency in each window, and replacing the modal instruction at the center of the window with the mode to achieve replacement and delay of frequent transition behaviors; when there is a high-energy consumption transition in the window, further replacing the original path by locally searching for the modal sequence with the minimum energy consumption cost;

[0116] The smoothed mode jump sequence is used as the execution instruction of the vehicle control system to generate the final smooth mode control trajectory, which is used to guide the operation mode switching and energy distribution execution of the plug-in hybrid vehicle.

[0117] In this embodiment, the step seven of applying the smoothed modal control trajectory to the vehicle control system to achieve mode switching and energy distribution of the plug-in hybrid vehicle under different operating states is specifically as follows:

[0118] Inputting the smooth modal control trajectory into a vehicle control system as a time series modal command sequence, wherein the smooth modal control trajectory includes target modes and control command parameters corresponding to each time step;

[0119] In each control cycle, the vehicle control system reads the target mode of the corresponding time step in the smooth modal control trajectory according to the current real-time state information of the vehicle;

[0120] According to the target mode, the engine control unit, the electric drive control unit and the energy recovery system are dynamically called to coordinate the drive source power output and the energy flow path according to the planning content of the modal control trajectory;

[0121] When the target mode is pure electric drive mode, the vehicle control system turns off the engine and only controls the motor output to meet the driving demand power;

[0122] When the target mode is the engine drive mode, the vehicle control system uses the engine to directly drive the vehicle, and the electric drive system remains disengaged or unloaded;

[0123] When the target mode is hybrid drive mode, the vehicle control system coordinates the engine and motor outputs to jointly drive to meet the vehicle power requirements;

[0124] When the target mode is energy recovery mode, the vehicle control system controls the motor to operate in a power generation state, converting mechanical energy during braking or coasting into electrical energy and feeding it back to the battery system;

[0125] The vehicle control system implements corresponding control instructions based on the duration and transfer sequence of each modal segment in the smooth modal control trajectory, realizing flexible mode switching and dynamic energy distribution during vehicle operation, thereby improving overall energy efficiency and ensuring smooth response of the vehicle's power system.

[0126] Example 1:

[0127] To verify the feasibility of the present invention, it was applied to the energy management system of a plug-in hybrid SUV being tested by a new energy vehicle company. This vehicle is equipped with a 1.5T engine, an electric drive system, and a 12.8kWh battery pack, and has four operating modes: pure electric, engine, hybrid, and energy recovery. Due to the hilly terrain, complex traffic, and frequent operating condition fluctuations in City A, traditional energy management algorithms driven by rules or static policy tables suffer from frequent mode switching, delayed drive source response, and high overall energy consumption, making it difficult to meet the dual needs of urban commuting and energy conservation and emission reduction.

[0128] In this example, vehicle operating state data is first collected, including state-of-the-art (SOC), electric drive power, engine output power, vehicle speed, longitudinal acceleration, geographic location, road gradient, traffic flow level, traffic light cycle, and the driver's accelerator pedal input amplitude and frequency. Through timestamp alignment, missing value filling, and filtering, a state feature dataset that meets unified standards is generated. A kinetic potential energy tensor graph is then constructed. Energy consumption values ​​at each state point are estimated using metrics such as battery charge change, fuel consumption change, and kinetic energy loss. This data is then standardized to form a multidimensional potential energy tensor graph.

[0129] A modal jump graph is constructed based on the potential energy tensor graph, with the four driving modes as nodes. The potential energy differences between states determine whether to establish jump paths, thereby defining a feasible transition structure for the system. In a piecewise policy jump learner, whether to allow a mode jump is determined based on the current state and a threshold value. Jump policy samples are generated, and the long-term cost of the state-mode combination is estimated using the jump Bellman function. The jump policy samples are then used to construct a modal subspace tensor graph.

[0130] During model training, a neural ordinary differential equation module is used to learn the temporal evolution of the long-term cost value of each modal node. Simultaneously, a self-attention mechanism module extracts the historical dependency strength between modal transitions, enabling awareness of modal structure and improving the interpretability and stability of the transition strategy. The resulting trained jump strategy model is deployed in an actual vehicle, and a modal switching smoothing module is used to suppress fluctuations in transition frequency and energy consumption, preventing modal jitter and ineffective switching.

[0131] In real-world urban road testing, the test vehicle traveled approximately 6,400 kilometers, spanning spring and winter, and encompassing typical urban conditions such as peak traffic, mountain roads, and highways. To validate the performance of the proposed system, it was compared with existing rule-driven strategies (Baseline-Rule) and deep Q-network strategies (Baseline-DQN). The test results are shown in Table 1 below.

[0132] Table 1 Comparison of mode switching frequency and energy consumption

[0133]

[0134] As can be seen from Table 1, the Baseline-Rule strategy has the highest average mode switching frequency, reaching 1.83 times / km. This indicates that frequent switching between different driving modes can easily lead to system response delays and energy consumption fluctuations. In contrast, the Baseline-DQN strategy introduces a learning mechanism, reducing the mode switching frequency to 1.21 times / km, demonstrating a certain degree of optimization. The proposed method significantly reduces the mode switching frequency to only 0.78 times / km, demonstrating its more stable and effective control of mode transitions and avoiding unnecessary frequent switching.

[0135] In terms of average unit energy consumption, the proposed method reduces energy consumption to 13.09 kWh / 100 km, a reduction of approximately 16.7% compared to the Baseline-Rule's 15.72 kWh / 100 km, demonstrating significant energy-saving potential in actual operation. The proposed method's standard deviation of energy consumption, a measure of system energy consumption volatility, is only 0.65 kWh, significantly lower than the Baseline-Rule's 1.14 kWh and the Baseline-DQN's 0.92 kWh. This demonstrates that the proposed modal smoothing mechanism is more robust in controlling energy consumption fluctuations, resulting in smoother and more efficient overall system operation.

[0136] In addition, 50 typical acceleration and deceleration phases and traffic light conditions were analyzed, and their modal response delays and modal transition efficiencies were statistically analyzed. The results are shown in Table 2 below.

[0137] Table 2 Comparison of modal response and transition efficiency under typical working conditions

[0138]

[0139] It can be seen from the data in Table 2 that the present invention has significant advantages over the prior art in terms of dynamic operating response and modal transition efficiency.

[0140] Under the typical working condition of starting at a traffic light, the modal response delay of the Baseline-Rule strategy is 2.15 seconds, the effective transition rate is only 67.4%, and the instantaneous energy consumption changes by as much as 8.7% after the transition, indicating that it is slow to respond and has rough control in complex traffic scenarios, which easily leads to energy consumption fluctuations. Although the Baseline-DQN strategy using deep Q learning has improved in terms of delay and transition rate, there are still obvious energy consumption disturbances. In contrast, the modal response delay of the method of the present invention is reduced to 0.89 seconds, the transition efficiency reaches 91.8%, and the instantaneous energy consumption changes by only 2.3% after the transition, which significantly improves the system's responsiveness and energy stability in traffic signal control scenarios.

[0141] In urban deceleration and congestion conditions, the Baseline-Rule strategy has a response delay of 1.87 seconds and an effective transition rate of 71.6%, while the instantaneous energy consumption after the transition still fluctuates by 6.9%. The Baseline-DQN strategy performs slightly better, but the energy consumption variation is still 3.6%. The modal response delay of the present invention in this scenario is further shortened to 0.75 seconds, the transition rate is increased to 94.2%, and the energy consumption variation is reduced to 1.7%, indicating that it can achieve faster, more accurate and smoother modal control in complex dynamic road conditions.

[0142] In this embodiment, by introducing the jump Bellman function, the modal subspace tensor graph and the variable structure neural network, efficient modal decision-making and energy distribution control of the plug-in hybrid vehicle under complex dynamic conditions are achieved. Compared with the traditional rule-driven strategy and the ordinary deep reinforcement learning method, the present invention can significantly improve the response speed and decision-making accuracy of the modal transition, reduce the modal switching frequency and energy consumption fluctuations, and enhance the robustness and economy of the system in a non-stationary environment. At the same time, through potential energy modeling and the construction of discontinuous strategy space, frequent invalid transitions and energy waste are effectively avoided, making the energy management of the whole vehicle more in line with the dynamic characteristics and intelligent requirements in real driving scenarios. Overall, the present invention achieves the goals of more accurate modal selection, more intelligent strategy and more flexible control, and has good prospects for promotion and application.

[0143] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning, characterized by: The steps include: Step 1: Collect the operating status data of the plug-in hybrid vehicle and generate a status feature data set; Step 2: constructing a dynamic potential energy field model based on the state characteristic data set to obtain a potential energy tensor diagram covering a multi-dimensional state space, wherein each state point corresponds to a potential energy value; Step 3: Constructing a modal jump graph based on the potential energy tensor graph, defining the vehicle operation mode as a discrete node, establishing an inter-modal jump path based on the potential energy gradient of the state transition, and generating a modal jump path structure; Step 4: Input the modal jump path structure into the segmented policy jump learner to construct a discontinuous policy space, and limit the modal jump action to be executed only when the corresponding state meets the potential energy transition threshold, and output the jump policy sample; Step 5: Use the jump Bellman function to estimate the value of the state-mode pair, embed the jump strategy sample into the modal subspace tensor graph, and use a variable structure neural network to dynamically model the modal mapping relationship to obtain a trained jump strategy model; Step 6: Calling the jump strategy model, inferring the real-time operating state of the vehicle, outputting a modal jump sequence, and smoothing the modal jump sequence based on the modal switching frequency and energy consumption changes to generate a smoothed modal control trajectory; Step 7: Apply the smooth modal control trajectory to the vehicle control system to achieve mode switching and energy distribution of the plug-in hybrid vehicle under different operating conditions.

2. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The operating status data includes battery state of charge, electric drive power, engine output power, vehicle speed, longitudinal acceleration, geographic location information, road slope, traffic flow level, traffic light cycle, and driver accelerator pedal input amplitude and frequency.

3. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The state feature data set is generated by performing timestamp alignment, missing value interpolation and noise filtering on the operating state data, and segmenting the continuous data based on a set time sliding window to extract statistical features, rate of change features, extreme value features and frequency domain features from each segment of data, forming feature vectors according to a unified coding format, and constructing a state feature data set.

4. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1, characterized in that: The step 2 is specifically as follows: According to the state characteristic data set, a dynamic potential energy field model is constructed using a mapping relationship between vehicle energy consumption and state variables; Each sample in the state feature data set is regarded as a state point in the multidimensional state space, and the energy consumption cost value of the corresponding state point is calculated by combining the battery power change value, engine fuel consumption change value and mechanical kinetic energy loss change value per unit time corresponding to the state point; Normalizing the energy consumption cost value and converting it into a potential energy value to reflect the energy consumption trend and the difficulty of the drive system mode transition at the state point; By performing grid interpolation construction on all state points, a continuous potential energy tensor diagram is generated in the multidimensional state space, where each state point serves as a grid point in the potential energy tensor diagram and corresponds to a unique potential energy value.

5. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1, characterized in that: The step three is specifically as follows: The operating modes of the plug-in hybrid vehicle are defined as discrete nodes of the modal jump diagram, wherein the operating modes include pure electric drive mode, engine drive mode, hybrid drive mode and energy recovery mode; Mapping each state point in the potential energy tensor graph to the discrete nodes, and determining the mapping relationship between the state points and the discrete nodes using the potential energy value corresponding to each state point under each discrete node; For each state point in the potential energy tensor graph, calculating the potential energy difference between the state point at the current discrete node and other discrete nodes, and determining the potential energy gradient of each state point between modes; A potential energy gradient threshold is set, and the potential energy gradient of the state point between discrete nodes is compared with the potential energy gradient threshold. When the potential energy gradient is less than the potential energy gradient threshold, a jump path between the discrete nodes is constructed as a directed edge of the modal jump graph. The direction of the directed edge represents the allowable modal transition direction, and the weight of the directed edge represents the corresponding potential energy gradient during the modal transition process. A modal jump path structure containing potential energy gradient information is generated to characterize the allowable modal transition relationship of the plug-in hybrid vehicle under different states.

6. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1, characterized in that: The segmented policy jump learner is used to construct a discontinuous policy space and output jump policy samples that meet the jump conditions, specifically: A segmented modal-action mapping structure is established, in which each pair of modal transition combinations in the modal jump path structure is defined as a discrete modal transition action and mapped to an independent strategy subspace, forming multiple discontinuous state-action mapping segments. Define the modal transition gating function to activate the discrete modal transition action in any state. In modal With modal The difference in potential energy under and preset potential energy transition threshold Compare, when satisfied When , discrete modal transition actions are allowed to be activated and executed; Otherwise, the discrete modal transition action is disabled in the current state; According to the current state, the effective modal jump action that meets the potential energy transition threshold condition is screened out from all discrete modal jump actions, and is combined with the current state to form a jump strategy sample, which is saved in the sample buffer.

7. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1, characterized in that: The jump Bellman function is used to estimate the value of the state-mode pair, and the jump strategy sample is embedded in the modal subspace tensor graph, specifically: The long-term energy cost of the state and mode combination is estimated using the jump Bellman function. The jump Bellman function takes the state, current mode, target mode, and modal transition cost in each jump strategy sample as input data and calculates the total expected energy consumption brought about by the future modal sequence when adopting a given modal jump behavior in the current state. The modal transition cost is the difference in potential energy value of the state between the current mode and the target mode. In the jump Bellman function calculation, for each combination of state and current mode, all target modes that are allowed to jump are traversed, and the corresponding mode transition cost in the jump strategy sample is extracted as the immediate cost. Combined with the value information contained in the jump strategy sample of the next state corresponding to the target mode, the long-term cost of the state-mode combination is estimated through a weighted accumulation method. The long-term cost reflects the overall energy consumption level caused by the vehicle executing the mode and mode sequence in the current state; All state-mode combinations in the jump strategy sample are constructed as nodes in the modal subspace tensor graph. Each node contains state characteristics, current modal category, immediate modal transition cost and corresponding long-term cost value. The connection relationship between nodes is constructed according to the modal jump pairs actually occurring in the jump strategy sample. Each connecting edge represents an executed modal transition action, and the edge weight represents the cost level and frequency of the modal transition action in the actual execution process.

8. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1, characterized in that: The variable structure neural network includes a neural ordinary differential equation modeling module and a self-attention mechanism modeling module; The input of the neural ordinary differential equation modeling module is the eigenvector corresponding to each node in the modal subspace tensor graph, and the eigenvector includes the potential energy value, state characteristics, modal category label and long-term cost value calculated according to the jump Bellman function of the state corresponding to the node; The neural ordinary differential equation modeling module uses node feature vectors as state variables in continuous time and time as the external driving dimension, and adopts numerical methods to approximate the ordinary differential equation model to obtain the evolution trajectory of the node long-term cost value at different time steps, realizes the modeling of the long-term cost value evolution process of the state-mode combination over time, and outputs the predicted long-term cost value sequence of each node in the next several time steps; The input of the self-attention mechanism modeling module is the feature vectors and connection edge information of all nodes in the modal subspace tensor graph. The connection edge records the actual modal transition behavior between nodes in the historical jump path and contains the corresponding transition frequency and cost as weight information; The self-attention mechanism modeling module calculates the attention weight between each node and all connected nodes, dynamically adjusts the response degree of each node to the adjacent modal jump path, and realizes modal transition relationship modeling based on historical structure information; The jump strategy samples are input into the variable structure neural network in batches, and the deviation between the predicted long-term cost value and the calculated value of the jump Bellman function is minimized as the loss target. Repeated training is performed until the loss function converges, and the trained jump strategy model is output.

9. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1, characterized in that: The step six is ​​specifically as follows: The trained jump strategy model is called, and the current real-time operating state of the vehicle is used as input. The modal jump action under the real-time operating state is inferred, forming a state-modal sequence output, where each record represents the modal jump path recommended by the vehicle in a continuous time period, generating a modal jump sequence. Counting the mode switching frequencies in continuous time periods in the mode hopping sequence, and calculating the number of mode transitions per unit time; Evaluate the energy consumption change rate before and after each mode transition as a measure of the impact of mode switching behavior on the energy consumption of the plug-in hybrid vehicle; A mode switching suppression factor is set. The mode switching suppression factor includes two sub-items: a mode transition frequency threshold and an energy consumption fluctuation tolerance threshold. The mode transition frequency threshold is defined as the maximum number of mode jumps allowed within a unit time window, and the energy consumption fluctuation tolerance threshold is defined as the maximum allowable range of the rate of change of energy consumption per unit time before and after a mode transition. In the mode jump sequence, if the number of mode jumps in a continuous time period exceeds the mode transition frequency threshold, or the rate of change of energy consumption is greater than the energy consumption fluctuation tolerance threshold, the time period is marked as an abnormal transition period. Performing a window sliding average smoothing process on the abnormal transition segment, specifically comprising: sliding a modal value sequence on the modal jump sequence with a preset window length, calculating the mode of the modal value occurrence frequency in each window, and replacing the modal instruction at the center of the window with the mode, thereby replacing and delaying frequent transition behaviors, and replacing the original path by locally searching for a modal sequence with the minimum energy consumption cost; The smoothed modal jump sequence is used as the execution instruction of the vehicle control system to generate the final smooth modal control trajectory.

10. The plug-in hybrid electric vehicle energy management optimization method based on deep reinforcement learning according to claim 1, characterized in that: The step seven is specifically as follows: Inputting the smooth modal control trajectory into a vehicle control system as a time series modal command sequence, wherein the smooth modal control trajectory includes target modes and control command parameters corresponding to each time step; In each control cycle, the vehicle control system reads the target mode of the corresponding time step in the smooth modal control trajectory according to the current real-time state information of the vehicle; According to the target mode, the engine control unit, the electric drive control unit and the energy recovery system are dynamically called to coordinate the drive source power output and the energy flow path according to the planning content of the modal control trajectory; When the target mode is pure electric drive mode, the vehicle control system turns off the engine and only controls the motor output to meet the driving demand power; When the target mode is the engine drive mode, the vehicle control system uses the engine to directly drive the vehicle, and the electric drive system remains disengaged or unloaded; When the target mode is hybrid drive mode, the vehicle control system coordinates the engine and motor outputs to jointly drive to meet the vehicle power requirements; When the target mode is the energy recovery mode, the vehicle control system controls the motor to operate in the power generation state, converting the mechanical energy during braking or coasting into electrical energy and feeding it back to the battery system.

Images