Safety assessment and application methods of reinforcement learning control strategies in high-dimensional space

Abstract

The present invention relates to a safety assessment and application method of a reinforcement learning control strategy in a high-dimensional space, which belongs to the intersection of new energy vehicles and artificial intelligence. The method includes: S1: defining a normalized state space, an action space and a reward function, guiding an intelligent agent to master a near-optimal or optimal control strategy in a current training environment, and saving training samples generated in real time; S2: building a high-dimensional known state space corresponding to a neural network near-optimal or optimal control strategy model at this stage and maintaining stable control reliability; S3: taking a high-dimensional historical state space as a benchmark and taking a clustering result of a single cluster as a center, proposing the concepts of judgment window and membership, and clarifying under which state tensor data is driven with reliability and meets the optimal control effect; S4: strengthening the state transition samples of the acquisition boundary and unfamiliar areas, and building a sample set for a real and unfamiliar environment, for improving the applicability and safety of deep reinforcement learning strategies.

Description

Technical Field

[0001] The present invention belongs to the intersection of new energy vehicles and artificial intelligence, and relates to a safety assessment and application method of a reinforcement learning control strategy in a high-dimensional space. Background Art

[0002] New energy vehicles are seen as a key measure to promote energy transformation and alleviate the energy crisis. Mainstream and emerging automakers have launched pure electric vehicles, hybrid vehicles and fuel cell vehicles. Overall, pure electric vehicles have attracted the attention of consumers due to their low charging costs, environmentally friendly driving methods and the ability to meet urban travel needs. Fuel cell vehicles use hydrogen instead of gasoline to generate electricity and drive motors, and are considered to be the main power system for future commercial vehicles in many places. Hybrid vehicles have the most mature technology level, which can meet the requirements of driving range, convenient energy replenishment, energy conservation and emission reduction. They are also ideal transitional products and are expected to occupy a long-term sales share in the new energy vehicle market.

[0003] The development path of hybrid vehicle technology begins with the selection and parameter matching of the powertrain, and the energy management strategy is one of the core technologies for achieving energy conservation, emission reduction and improving fuel economy. Its main principle is to reasonably distribute power flow among multiple power sources under the premise of meeting the power system requirements and constraints. In recent years, with the development of machine learning, especially the application of deep learning and reinforcement learning, learning-based energy management strategies have attracted the attention of many researchers. Simply put, reinforcement learning is to give reward information to the output action of the agent through the environment, and then the agent gradually explores the corresponding optimal action under each state with the goal of maximizing the cumulative reward. At this time, once the environment space used for training has complex time-varying characteristics and needs to be described by a higher-dimensional state space, it is not only difficult for offline training, but also whether it can be effectively used for online real space testing even after the training is completed. Because for some learning tasks, such as autonomous driving, etc., the simulation environment of offline training cannot cover all possible driving states. Therefore, no matter how much training effect can be achieved, if the scope of application of the deep reinforcement learning control strategy cannot be determined, there will always be considerable safety hazards in the online environment. Therefore, on the one hand, the security of learning strategies needs to be clarified, and on the other hand, the applicability of learning control strategies in online environments should be gradually improved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a security assessment and application method of a reinforcement learning control strategy in a high-dimensional space, which can clarify the security of the learning strategy and gradually improve the applicability of the learning control strategy in an online environment.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A safety assessment and application method of a reinforcement learning control strategy in a high-dimensional space, specifically comprising the following steps:

[0007] S1: Offline pre-training of deep reinforcement learning control strategies. On the one hand, by defining the state space, action space and reward function, the normalization indicators for different variables are clarified to guide the deep reinforcement learning agent to master the near-optimal or optimal control strategy in the current training environment. On the other hand, during the training process, the training samples generated in real time and directly used to update the neural network are saved to build a high-dimensional historical state space as a benchmark.

[0008] S2: Before conducting online testing, all historical samples in the experience pool need to be preprocessed to build a high-dimensional known state space that corresponds to the neural network type near-optimal or optimal control strategy model at this stage and maintains stable control reliability;

[0009] S3: During the online test, we use the historical state space as a benchmark and the clustering results of a single cluster as the center to conduct real-time safety assessment on multi-dimensional state tensors with high-dimensional time-varying characteristics, propose the concepts of judgment window and membership, and clarify under what state tensor data-driven conditions the control strategy model based on deep reinforcement learning is reliable and meets the optimal control effect when facing new and unfamiliar test scenarios;

[0010] S4: After the online test, the frequency of collecting Markov state transition samples is strengthened for the boundaries of high-dimensional state space and unexpected state unfamiliar areas, and a sample set for real and unfamiliar driving environments is constructed. Then, based on the "concatenation" of new environmental samples and the exploration and learning by the new sub-strategy network, the main strategy network and the sub-strategy network are merged and unified, gradually realizing the online and safe expansion of the state space, and gradually improving the applicability and safety of deep reinforcement learning strategies.

[0011] Furthermore, in step S1, the offline pre-training of the deep reinforcement learning control strategy specifically includes the following steps:

[0012] S11: In an offline simulation environment, we are committed to obtaining a more reliable deep reinforcement learning control strategy in advance through simulation;

[0013] S12: During offline training, corresponding historical experience pools are established for different control strategies, and training samples used directly for loss function and gradient calculation are saved in a timely manner.<s(t),a(t),r(t),s’(t)> , where s is the current state, a is the current action, r is the current reward, and s' is the state at the next moment.

[0014] Further, step S11 specifically includes: taking intelligent hybrid vehicles as the research object, taking deep deterministic policy gradient equipped with an actor-critic network structure as the dominant algorithm, and taking energy-saving cruise driving that synergistically optimizes economy and safety as the expected goal, defining the required state space S, action space A, reward function R, neural network structure, and hyperparameters required for training, etc., and then obtaining the initialized neural network policy model after completing multiple rounds of iterative training; at the same time, all variables in the state space need to be normalized, and the normalized threshold boundary of each state variable value is clearly defined.

[0015] Furthermore, in step S12, for the Adaptive Cruise Control (ACC) strategy, the correct control command needs to be output at any time when the speed is not equal to 0 m / s to maintain the normal cruising speed; for the Energy Management Strategy (EMS), power allocation only acts on the driving charging mode and the hybrid drive mode, so the training samples generated in the pure electric drive or regenerative braking modes do not need to be stored in the historical experience pool of the energy management strategy (EMS); at this time, after completing the offline training and obtaining the initialized neural network strategy model, all historical state samples witnessed by the neural network in the offline environment can also be obtained.

[0016] Further, step S2 specifically includes: by collecting all historical samples, for example, for an adaptive cruise control (ACC) strategy, the state space will include the current vehicle speed, target vehicle speed, transmission gear, required power, control action at the previous moment, road curvature and road slope, etc.; at this time, a total of seven-dimensional historical state space is formed for the adaptive cruise control (ACC) strategy, which means that the entire neural network fits the optimal control function between seven independent states and one dependent action; by performing N-dimensional convex hull calculation Convhulln and single cluster clustering calculation Kmeans on a large number of seven-dimensional state samples, the envelope boundary corresponding to the outside of the seven-dimensional state space and the sample midpoint corresponding to the inside are obtained, thereby determining the boundary and center of the entire sample space; at the same time, principal component analysis PCA can be used to reduce the dimensionality of the seven-dimensional boundary points after the envelope calculation, thereby achieving a visualization effect in three-dimensional space.

[0017] Further, step S3 specifically includes the following steps:

[0018] S31: Taking the dimension of the online state tensor as an indicator, the spatial range formed by the historical state tensor of each dimension is obtained one by one, especially the threshold boundary, thereby forming a loop architecture for each dimension in the outermost layer;

[0019] S32: The concept of JudgeWindow is proposed as the width range for filtering adjacent samples layer by layer; after calculating the absolute deviation AbsDiff between the surrounding tensor and the real-time state tensor in the current dimension, all historical tensors with absolute deviation AbsDiff within the judgment window are extracted as the evaluation basis for the next dimension; after the above processing of all dimensions one by one, the step-by-step "slicing" operation of the historical state tensor space can be realized;

[0020] S33: Through the above-mentioned iterative evaluation of each dimension, if the value of each dimension on the online time-varying state tensor is in the high-dimensional historical state space, then it can be determined that the current state belongs to the historical training sample set, and the control action output by the deep reinforcement learning control strategy model at this time also has a certain degree of reliability and safety, which is completely guaranteed by the degree of offline training, that is, in an offline environment, as long as the optimal action point corresponding to each high-dimensional state tensor point in the known historical state space can be mastered, then in the online test, as long as the new state tensor is still included in the historical state space, it can be considered that the current strategy is still reliable;

[0021] S34: According to the real-time changing online state tensor, the nearest Euclidean distance relative to the envelope boundary and the Euclidean distance relative to the center of the space are calculated respectively, and the familiarity or danger index of the online state relative to the current neural network strategy model is evaluated by the proximity of the real-time seven-dimensional space tensor point to the boundary, and then the control percentage of the deep reinforcement learning control strategy and the rule-based control strategy is determined based on this index; this means that when the online state tensor point changes randomly in the known seven-dimensional space, the closer it is to the center, the more reliable the effect of learning control is in the current state; conversely, when the dangerous space boundary has been touched, 100% rule-based control commands are used to ensure safety.

[0022] Further, step S4 specifically includes the following steps:

[0023] S41: Based on the changing characteristics and familiarity evaluation index of the dynamically changing high-dimensional state tensor in the online test environment, the online collection frequency of Markov state transition samples in the spatial boundary and unknown area is strengthened, and an online learning sample data set for actual extreme working conditions and unfamiliar driving environments is constructed. In particular, special focus is placed on the collection of state samples that significantly exceed the known space, so as to further expand the scope of application of the optimal learning control strategy in the online environment;

[0024] S42: A "concatenation" operation is performed on the unfamiliar state samples that have been identified online and newly collected to form a training case specifically for the unfamiliar state boundary space. At the same time, a new sub-strategy network is created specifically for exploring and learning the optimal strategy under this case. The training at this time does not affect the application of the original main strategy network.

[0025] S43: Modify the training environment, changing the original closed-loop environmental dynamics model driven by a small number of independent states into an open-loop environmental dynamics model driven entirely by training samples, and strive to determine the reward response and state transition response characteristics that each new sample can generate after performing different actions; then after completing the policy learning for boundaries and unfamiliar states, integrate the main policy network for conventional historical space and the sub-policy network for unfamiliar samples, and through the "apprentice architecture" and "knowledge distillation" training system, realize the merger and unification of the two types of policy networks, narrow the unknown state field of the deep reinforcement learning control policy network, and improve the online update scheme that takes into account safety and applicability.

[0026] The beneficial effects of the present invention are as follows: the present invention can clarify the security of the learning strategy and at the same time improve the applicability of the learning control strategy in an online environment.

[0027] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 It is an overall flow chart of the safety assessment and application method of the reinforcement learning control strategy in high-dimensional space of the present invention;

[0030] Figure 2 It is the visualization result of high-dimensional envelope and principal component method dimensionality reduction;

[0031] Figure 3 is the historical known state space of ACC and EMS;

[0032] Figure 4 It is a schematic diagram of the membership determination steps in high-dimensional space. DETAILED DESCRIPTION

[0033] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0034] See also Figure 1 to Figure 4 The present invention provides a safety assessment and application method of a reinforcement learning control strategy in a high-dimensional space, the process is as follows Figure 1 As shown, it can be mainly divided into the following stages:

[0035] S1: Offline pre-training of deep reinforcement learning control strategies. On the one hand, by defining the state space S, action space A and reward function R, the normalized indicators for different variables are clearly defined to guide the deep reinforcement learning agent to master the near-optimal or optimal control strategy in the current training environment; on the other hand, during the training process, the training samples generated in real time and directly used to update the neural network are also saved to build a high-dimensional historical state space as a benchmark.

[0036] S11: In an offline simulation environment, we are committed to obtaining a more reliable deep reinforcement learning control strategy in advance through simulation. For example, we take smart hybrid vehicles as the research object, use the deep deterministic policy gradient equipped with an actor-critic network structure as the dominant algorithm, and take the realization of energy-saving cruise driving that synergistically optimizes economy and safety as the expected goal. We will define the required state space S, action space A, reward function R, neural network structure Network, and hyperparameters required for training, and then obtain the initialized neural network policy model after completing multiple rounds of iterative training. At the same time, all variables in the state space need to be normalized, and the normalized threshold boundary of each state variable value must be clearly defined;

[0037] S12: During offline training, corresponding historical experience pools are established for different control strategies, and training samples used directly for loss function and gradient calculation are saved in a timely manner.<s(t),a(t),r(t),s’(t)> , where s is the current state, a is the current action, r is the current reward, and s' is the state at the next moment. In particular, for the Adaptive Cruise Control (ACC) strategy, the correct control command needs to be output at any time when the speed is not equal to 0m / s to maintain the normal cruising speed; for the Energy Management Strategy (EMS) strategy, power allocation only acts on the driving charging mode and the hybrid drive mode, so the training samples generated in the pure electric drive, regenerative braking and other modes do not need to be stored in the historical experience pool of the EMS. At this time, after completing the offline training and obtaining the initialized neural network policy model, all historical state samples witnessed by the neural network in the offline environment can also be obtained.

[0038] S2: Before conducting online testing, all historical samples in the experience pool need to be preprocessed to build a high-dimensional known state space that corresponds to the neural network type near-optimal / optimal control strategy model at this stage and maintains stable control reliability.

[0039] The construction of a high-dimensional known state space is carried out as follows: by collecting all historical samples, for example, for the ACC strategy, the state space will include the current vehicle speed VelSpd, the target vehicle speed TgtSpd, the transmission gear TransGear, the required power WhlTrq, the control action ActBef at the previous moment, the road curvature RodCuv and the road slope RodSlp, etc. At this time, a total of seven dimensions of historical state space are formed for the ACC strategy, which means that the entire neural network fits the optimal control function between seven independent states and one dependent action. By performing N-dimensional convex hull calculation Convhulln and single cluster clustering calculation Kmeans on a large number of seven-dimensional state samples, the envelope boundary corresponding to the outside of the seven-dimensional state space and the corresponding sample midpoint inside are obtained, thereby determining the boundary and center of the entire sample space. At the same time, principal component analysis PCA can be used to reduce the dimensionality of the seven-dimensional boundary points after the envelope calculation, thereby achieving the following Figure 2 The visualization effect in three-dimensional space is shown.

[0040] S3: During the online test, based on the historical state space and centered on the clustering results of a single cluster, a real-time safety assessment is performed on the seven-dimensional state tensor with high-dimensional time-varying characteristics. The concepts of judgment window and membership are proposed to clarify under what state tensor data the policy model based on deep reinforcement learning is reliable and meets the optimal control effect when facing a new and unfamiliar test scenario.

[0041] The safety and membership are determined according to the following steps:

[0042] S31: First, using the dimension of the online state tensor as an indicator, we need to obtain the following Figure 3 The spatial range of the historical state tensor of each dimension shown, in particular the threshold boundary, thereby forming a recurrent architecture for each dimension at the outermost layer.

[0043] S32: Secondly, the concept of JudgeWindow is proposed as the width range for filtering adjacent samples layer by layer. After calculating the absolute deviation AbsDiff between the surrounding tensor and the real-time state tensor in the current dimension, all historical tensors with absolute deviation AbsDiff within the judgment window are extracted as the evaluation basis for the next dimension. After the above processing of all dimensions one by one, it can be achieved, for example Figure 4 The figure shows the step-by-step “slicing” operation on the history state tensor space.

[0044] S33: Finally, through the above-mentioned iterative evaluation dimension by dimension, if the values ​​of each dimension on the online time-varying state tensor are in the high-dimensional historical state space, then it can be determined that the current state belongs to the historical training sample set, and the control action output by the deep reinforcement learning strategy model at this time also has a certain degree of reliability and security, which is completely guaranteed by the degree of offline training. That is, in an offline environment, as long as the optimal action point corresponding to each high-dimensional state tensor point in the known historical state space can be mastered, then under online testing, as long as the new state tensor is still included in the historical state space, the current strategy can be considered to be still reliable.

[0045] S34: According to the real-time changing online state tensor, the nearest Euclidean distance to the envelope boundary and the Euclidean distance to the center of the space are calculated respectively, and the familiarity or danger index of the online state relative to the current neural network strategy model is evaluated by the closeness of the real-time seven-dimensional space tensor point to the boundary, and then the control percentage of the learning control strategy and the rule-based control strategy is determined by this index. This means that when the online state tensor point changes randomly in the known seven-dimensional space, the closer to the center, the more reliable the effect of learning control in the current state; conversely, when the dangerous space boundary has been touched, 100% rule-based control commands are used to ensure safety.

[0046] S4: After the online test, facing the high-dimensional state space boundary and the unexpected state unfamiliar area, the frequency of collecting Markov state transition samples will be strengthened, and a sample set for real and unfamiliar driving environment will be constructed. Then, according to the "series" of new environment samples, the new sub-strategy network will explore and learn, and then the main strategy network and the sub-strategy network will be merged and unified, gradually realizing the online and safe expansion of the state space, and gradually improving the applicability and safety of deep reinforcement learning strategies. The specific steps are as follows:

[0047] S41: First, based on the changing characteristics and familiarity evaluation indicators of the dynamically changing high-dimensional state tensor in the online test environment, the online collection frequency of Markov state transition samples in the spatial boundaries and unknown areas is strengthened, and an online learning sample data set for actual extreme working conditions and unfamiliar driving environments is constructed. In particular, special focus is placed on the collection of state samples that significantly exceed the known space, so as to further expand the scope of application of the optimal learning control strategy in the online environment.

[0048] S42: Secondly, a "concatenation" operation is performed on the unfamiliar state samples that have been identified online and newly collected to form training cases specifically for the unfamiliar state boundary space. At the same time, a new sub-strategy network is created specifically for exploring and learning the optimal strategy under this case. The training at this time does not affect the application of the original main strategy network.

[0049] S43: Finally, the training environment is modified, and the original closed-loop environmental dynamics model driven by a small number of independent states is changed to an open-loop environmental dynamics model driven entirely by training samples, aiming to determine the reward response and state transition response characteristics that each new sample can generate after performing different actions. After completing the policy learning for boundaries and unfamiliar states, the main policy network for conventional historical space and the sub-policy network for unfamiliar samples are integrated, and the two types of policy networks are merged and unified through the "apprentice architecture" and "knowledge distillation" training system, narrowing the unknown state field of the deep reinforcement learning control policy network and improving the online update scheme that takes into account safety and applicability.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A safety assessment and application method for reinforcement learning control strategy in high-dimensional space, characterized in that: The method specifically comprises the following steps: S1: For offline pre-training of deep reinforcement learning control strategies, on the one hand, by defining the state space, action space and reward function, clarifying the normalized indicators for different variables, and guiding the deep reinforcement learning agent to master the near-optimal or optimal control strategy in the current training environment; on the other hand, during the training process, save the training samples generated in real time and directly used to update the neural network, and use them to build a high-dimensional historical state space as a benchmark; in step S1, the offline pre-training of the deep reinforcement learning control strategy specifically includes the following steps: S11: In an offline simulation environment, a reliable deep reinforcement learning control strategy is obtained in advance through simulation, specifically including: taking intelligent hybrid vehicles as the research object, taking the deep deterministic policy gradient equipped with the actor-critic network structure as the dominant algorithm, and taking the energy-saving cruise driving that synergistically optimizes economy and safety as the expected goal, the required state space S, action space A, reward function R, neural network structure Network and the hyperparameters required for training will be defined respectively, and then the initialization neural network policy model will be obtained after completing multiple rounds of iterative training; at the same time, all variables in the state space need to be normalized, and the normalization threshold boundary of each state variable value must be clearly defined; S12: During offline training, corresponding historical experience pools are established for different control strategies, and training samples used directly for loss function and gradient calculation are saved in a timely manner.<s(t),a(t),r(t),s’(t)> , where s is the current state, a is the current action, r is the current reward, and s' is the state at the next moment; The training samples generated in pure electric drive or regenerative braking mode do not need to be stored in the historical experience pool of the energy management strategy; at this time, after completing the offline training and obtaining the initialized neural network strategy model, all historical state samples witnessed by the neural network in the offline environment can also be obtained; S2: Before conducting the online test, all historical samples in the experience pool are preprocessed to build a high-dimensional known state space that corresponds to the neural network type near-optimal or optimal control strategy model at this stage and maintains stable control reliability; Step S2 specifically includes: by collecting all historical samples, for the adaptive cruise control strategy, the state space will include the current vehicle speed, target vehicle speed, transmission gear, required power, control action at the previous moment, road curvature and road slope; at this time, a total of seven-dimensional historical state space is formed for the adaptive cruise control strategy, which means that the entire neural network fits the optimal control function between seven independent states and one dependent action; by performing N-dimensional convex hull calculation Convhulln and single cluster clustering calculation Kmeans on a large number of seven-dimensional state samples, the envelope boundary corresponding to the outside of the seven-dimensional state space and the sample midpoint corresponding to the inside are obtained, thereby determining the boundary and center of the entire sample space; at the same time, the seven-dimensional boundary points that have been calculated by the envelope are reduced in dimension using principal component analysis PCA, thereby achieving visualization in three-dimensional space; S3: During the online test, based on the historical state space and centered on the clustering results of a single cluster, a real-time safety assessment is performed on the multidimensional state tensor with high-dimensional time-varying characteristics, and the concepts of judgment window and membership are proposed to clarify the state tensor data-driven reliability and optimal control effect of the control strategy model based on deep reinforcement learning when facing a new and unfamiliar test scenario; specifically, the following steps are included: S31: Taking the dimension of the online state tensor as an indicator, obtain the spatial range formed by the historical state tensor of each dimension one by one, thereby forming a loop architecture for each dimension in the outermost layer; S32: The concept of JudgeWindow is proposed as the width range for filtering adjacent samples layer by layer; after calculating the absolute deviation AbsDiff between the surrounding tensor and the real-time state tensor in the current dimension, all historical tensors with absolute deviation AbsDiff within the judgment window are extracted as the evaluation basis for the next dimension; after the above processing of all dimensions one by one, the step-by-step "slicing" operation of the historical state tensor space is realized; S33: Through the above-mentioned iterative evaluation of each dimension, if the value of each dimension of the online time-varying state tensor is in the high-dimensional historical state space, then the current state is considered to belong to the historical training sample set, and in the offline environment, as long as the optimal action point corresponding to each high-dimensional state tensor point in the known historical state space can be mastered, then in the online test, as long as the new state tensor is still included in the historical state space, the current strategy is considered to be still reliable; S34: According to the online state tensor that changes in real time, the nearest Euclidean distance to the envelope boundary and the Euclidean distance to the center of the space are calculated respectively, and the familiarity or danger index of the online state relative to the current neural network strategy model is evaluated by the proximity of the real-time seven-dimensional space tensor point to the boundary, and then the control percentage of the deep reinforcement learning control strategy and the rule-based control strategy is determined by this index; when the online state tensor point changes randomly in the known seven-dimensional space, the closer to the center, the more reliable the effect of learning control is in the current state; on the contrary, when the dangerous space boundary has been touched, 100% rule-based control commands are used to ensure safety issues; S4: After the online test, the frequency of collecting Markov state transition samples is strengthened for the high-dimensional state space boundary and unexpected state unfamiliar areas, and a sample set for real and unfamiliar driving environment is constructed. Then, new environment samples are "connected in series" and explored and learned by the new sub-strategy network. Then, the main strategy network and the sub-strategy network are merged and unified, and the online and safe expansion of the state space is gradually realized, and the applicability and safety of the deep reinforcement learning strategy are gradually improved. The specific steps include: S41: Based on the changing characteristics and familiarity evaluation index of the dynamically changing high-dimensional state tensor in the online test environment, the online collection frequency of Markov state transition samples in the spatial boundary and unknown area is strengthened, and an online learning sample data set for actual extreme working conditions and unfamiliar driving environments is constructed. Special focus is placed on the collection of state samples that significantly exceed the known space, so as to further expand the scope of application of the optimal learning control strategy in the online environment; S42: A "concatenation" operation is performed on the unfamiliar state samples that have been identified online and newly collected to form a training case specifically for the unfamiliar state boundary space. At the same time, a new sub-strategy network is created specifically for exploring and learning the optimal strategy under the training case. The training at this time does not affect the application of the original main strategy network. S43: Modify the training environment, changing the original closed-loop environmental dynamics model driven by a small number of independent states to an open-loop environmental dynamics model driven entirely by training samples, and strive to determine the reward response and state transition response characteristics that each new sample can generate after performing different actions; then after completing the policy learning for boundaries and unfamiliar states, integrate the main policy network for conventional historical space and the sub-policy network for unfamiliar samples, and through the "apprentice architecture" and "knowledge distillation" training system, realize the merger and unification of the two types of policy networks, narrow the unknown state field of the deep reinforcement learning control policy network, and improve the online update scheme that considers safety and applicability.

Images