Unfamiliar environment state cognitive model training method for online driving scene

Abstract

The invention relates to an unfamiliar environment state cognitive model training method for an online driving scene, and belongs to the crossing field of new energy automobiles and artificial intelligence. The method comprises the steps that S1, a three-dimensional driving scene model is loaded, a hybrid electric vehicle model is built, and multi-angle driving pictures of a driving position at each moment are collected; s2, obtaining Camera BEV features of the pure visual feature extraction stream, and combining the Camera BEV features with an original two-dimensional variable state tensor; the method comprises the following steps of: forming an Actor-Critic-Guard network architecture by using a near-end strategy optimization algorithm based on an Actor-Critic architecture as an intelligent agent; s3, constructing a loss function containing loss values about Actor, Critic and Guard and an L2 regular term to train the network; and S4, loading a test environment similar to a real world, and evaluating the Guard environment cognitive network in a dynamic driving scene.

Description

Technical Field

[0001] The present invention belongs to the intersection field of new energy vehicles and artificial intelligence, and relates to a method for training a cognitive model of an unfamiliar environment state in an online driving scenario. Background Art

[0002] New energy vehicles are seen as an important measure to promote energy transformation and alleviate the energy crisis, especially the development of hybrid electric vehicles (HEV). As the world pays more attention to sustainable development and low-carbon economy, many mainstream and emerging automakers have launched pure electric vehicles, hybrid vehicles and fuel cell vehicles. However, hybrid vehicles play a key role in the transition period because they can combine the advantages of internal combustion engines and electric drive systems, meeting the needs of long-distance driving and achieving lower emissions in cities.

[0003] At the same time, the development of artificial intelligence technology, especially deep learning and reinforcement learning in machine learning, has gradually attracted the attention of the scientific research community and the industrial community, promoting the development of the AI ​​era. Data- or sample-driven deep reinforcement learning strategies face three key obstacles: the difference between simulation and reality, credibility, and sample efficiency. In a real driving environment, the time-varying and high-dimensional characteristics of the state make the update process lengthy. Regarding sample efficiency, by timely editing the Markov decision process and connecting key states, the information in the training data is made denser. When facing real physical systems, the dynamic residual model can alleviate the impact of simulation and real dynamics and sensor noise.

[0004] Usually, when learning strategies are deployed, it is necessary to implement backup strategies to ensure safety control. Therefore, some studies have introduced concepts such as uncertainty or confidence to achieve switching between these two types of strategies. However, a complete solution should include clarifying the scope of application of data-driven strategies based on historical training samples, evaluating risks or unknown states that occur during testing, and enhancing their applicability through updates to the online environment. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a method for training an unfamiliar environment state cognitive model for an online driving scenario to improve the reliability and safety of a deep reinforcement learning-based hybrid electric vehicle control strategy.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for training a cognitive model of an unfamiliar environment state in an online driving scenario specifically comprises the following steps:

[0008] S1: In the offline training scenario, adaptive cruise control and lane keeping assist are the primary learning tasks. By loading the three-dimensional driving scene model, a hybrid vehicle model is built, and the corresponding RGB camera model is arranged according to the camera internal and external parameter matrix of the nuScenes dataset, the multi-angle driving picture acquisition of each driving position is realized;

[0009] S2: Before the simulation training environment is started, the Camera BEV features of the pure visual feature extraction stream are obtained based on the real-time driving images collected by six RGB cameras with different viewing angles, and combined with the original two-dimensional variable state tensor; then, the proximal policy optimization algorithm based on the Actor-Critic architecture is used as the intelligent agent, and the concept of "Guard environmental cognition network" is proposed to form an "Actor-Critic-Guard" network architecture that can fully realize the three functions of policy learning-policy evaluation-environmental cognition;

[0010] S3: During the training of the simulated driving scenario, the traditional Actor-Critic architecture training process continues according to the original plan for the data-driven deep reinforcement learning hybrid vehicle control strategy. At the same time, in order to effectively train the Guard environment recognition network, the loss values ​​and L2 regularization terms of the Actor, Critic, and Guard are synchronously included in the loss function. Therefore, after the Actor and Critic networks have achieved a stable fit to the current training environment, they will begin to recognize the training environment state characteristics corresponding to the current optimal strategy.

[0011] S4: After the offline simulation training is completed, a more complex and changeable path trajectory that is close to the real world is loaded as a test condition, and a large and diverse state deviation is introduced to the test environment, such as trajectory characteristics, lighting conditions, weather changes and vehicle status. Then, the "Guard Environmental Cognition Network" is evaluated in a dynamic driving scenario to determine the scope of application that needs to be improved and enhanced.

[0012] Furthermore, in step S1, the modeling process of offline training specifically includes the following steps:

[0013] S11: Based on the training method of deep reinforcement learning control strategy, the state space S, action space A and reward function R of adaptive cruise control strategy and lane keeping assist strategy are set;

[0014] S ACC =(VehSpd,TgtSpd,TrsGear,WhlTrq,CurVat,Slope)

[0015] A ACC =[-1,1]Activation function = tanh

[0016] R ACC = -1×[abs(A ACC (t)-A Hybrid (t-1))+abs(VehSpd(t)-TgtSpd(t))]

[0017] S LKA =(VehSpd,AngDif,YawRat,PreLka,LatOff,CurVat)

[0018] A LKA =[-π / 2,π / 2]Activation function = tanh×π / 2

[0019] R LKA = -1×[abs(A LKA (t)-A PreLka (t))+abs(LatOff)]

[0020] Among them, S ACC is the state of the adaptive cruise control strategy, VehSpd is the current vehicle speed, TgtSpd is the target vehicle speed, TrsGear is the transmission gear, WhlTrq is the torque demand, CurVat is the road curvature, Slope is the road slope, A ACC is the output action of deep reinforcement learning, R ACC is the reward for the adaptive cruise control strategy, A Hybrid is the hybrid action that actually controls the vehicle, AngDif is the real-time angle difference, YawRat is the yaw rate, PreLka is the expected steering angle, LatOff is the lateral offset relative to the center lane, S LKA is the state of lane keeping assist, A LKA It is the action of lane keeping assist, R LKA is a bonus for lane keeping assist, A PreLka It is the expected steering action predicted based on the characteristics of the road ahead;

[0021] S12: Load the scenario for offline training in the autonomous driving 3D simulation software CARLA, and load the Citroen C3 used to characterize the body model. At the same time, build a parallel hybrid vehicle model in the Simulink environment, which mainly includes the driver model, controller model (engine, transmission, hybrid control), power system and dynamics model (engine, motor, power battery, transmission, body, tire, suspension, etc.);

[0022] S13: RGB cameras are arranged at the six camera extrinsic parameter matrix positions corresponding to the Citroen C3 body model to collect real-time scene images of the vehicle driving in a three-dimensional scene, including: front view, front left view, front right view, rear view, rear left view and rear right view, and an image with a size of 1600×900 is obtained as a perception feature.

[0023] Further, step S2 specifically includes the following steps:

[0024] S21: Before the simulation training environment is started, six RGB cameras with different viewing angles are used to collect driving images in real time. These images are processed by the BEV Fusion algorithm to generate BEV features with dimensions of 1×80×128×128. This process fuses the visual information from multiple cameras into a unified bird's-eye view, making subsequent processing more efficient and accurate. In order to further optimize feature extraction, a convolutional neural network is used to extract the driving image from the S21. BEV Perform dimensionality reduction and then compare with S ACC and S LKA Finally, these states are integrated using a fully connected network to provide multimodal input information for the training of deep reinforcement learning control strategies.

[0025] S22: Deep reinforcement learning algorithm based on Actor-Critic architecture - proximal policy optimization As the intelligent agent learns the energy-saving driving strategy of hybrid electric vehicles, the optimal control strategy of the current training environment is fitted by Actor, and the optimality of the current strategy is evaluated by Critic; at the same time, in order to further enhance the environmental adaptability and recognition ability of the intelligent agent, actively distinguish the unfamiliar state that may appear at any time and the dangerous actions that may be caused by it, the concept of "Guard environmental cognition network" is proposed; the core purpose is to analyze the state differences caused by the dynamic environment in real time, and take timely and safe takeover measures after approaching the boundary of the known space; in this regard, an "Actor-Critic-Guard" network architecture combining strategy learning-strategy evaluation-environmental cognition is constructed.

[0026] Further, in step S3, the training steps of the simulated driving scene are specifically as follows:

[0027] S31: In the training of the control strategy of hybrid electric vehicles based on data-driven deep reinforcement learning, the traditional Actor-Critic architecture training process calculates the updated gradient according to the loss function shown in the following formula, where the goal of the Actor is to maximize the expected return, and the goal of the Critic is to accurately estimate the value function of the current strategy. In order to prevent the neural network from overfitting, enhance the generalization of the neural network, and suppress the plasticity loss of the neural network, an L2 regularization term is added to the loss function of the original value function;

[0028]

[0029] L Critic =E[(V θ (s)-R) 2 ]+L Critic-L2

[0030] Among them, L Actor is the loss value of the actor network, L Critic is the loss of the critic network, π θ (a|s) is the probability distribution of the current strategy, is the probability distribution of the old strategy, is the advantage estimate, ε is the clipping parameter, V θ (s) is the output of the Critic’s value function, R is the return value calculated based on the actual reward and advantage calculation, and L Critic-L2 With L Actor-L2 It is the L2 regularization term about the weights of the Critic and Actor networks;

[0031] S32: In order to effectively train the Guard Environment Cognition Network, the loss function includes the loss values ​​and L2 regularization terms of the Actor, Critic, and Guard. Therefore, only after the Actor and Critic networks have achieved stable learning of the current training environment, will they begin to recognize the training environment state characteristics corresponding to the current optimal strategy. The training of the Guard Environment Cognition Network is mainly based on the following formula L Guard As shown, it specifically includes comprehensive indicators of multiple loss functions:

[0032] L Guard =L Critic +L Actor +L Guard-Label +L Guard-L2

[0033] Among them, L Guard is the loss value of the guardian network, L Guard-Label is the loss that guides Guard to learn the environment cognitive label, and L Guard-L2 It is the L2 regularization term about the Guard network weight.

[0034] Further, step S4 specifically includes the following steps:

[0035] S41: After the offline simulation training is completed, in order to fully highlight and verify the discrimination effect of the Guard environmental cognition network, a more complex and changeable path trajectory that is close to the real world is loaded as a test condition, and a state deviation with large differences and diversity is introduced to the test environment, such as: trajectory characteristics (more changeable road properties, steeper slopes, sharper curves, etc.), lighting conditions (night, early morning, noon, dusk, etc.), weather changes (sunny, rainy, snowy, foggy, haze, sandstorms, etc.) and vehicle status (higher speed, larger charge and discharge amplitude, more drastic torque and speed change range);

[0036] S42: In the test scenario for dynamic driving environment, verify the evaluation results of the "Guard Environmental Cognitive Network" on the random state tensor, and observe the control effect of the Actor network on the hybrid vehicle; once the real-time state tensor changes outside the range of the historical sample space, promptly start the takeover mode of the safety strategy, and continue to collect unfamiliar samples to complete the continuous learning of the optimal strategy corresponding to the new state in the test scenario. Among them, the most critical is whether the "Guard Environmental Cognitive Network" is accurate and reliable in the dynamic analysis of the current environment tensor, which directly affects the reliability and safety indicators of the deep reinforcement learning hybrid vehicle control strategy.

[0037] The beneficial effect of the present invention is that the "Guard Environmental Cognitive Network" model constructed by the present invention can provide continuous learning of the optimal strategy after inputting more complex and changeable path trajectories that are close to the real world as test conditions and diversified test environments, that is, the present invention greatly improves the reliability and safety of the deep reinforcement learning hybrid vehicle control strategy.

[0038] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0040] Figure 1 It is an overall flow chart of the unfamiliar environment state recognition model training method for online driving scenarios provided by the present invention;

[0041] Figure 2 The Citroen C3 car body model of the unfamiliar environment state cognitive model training method of the present invention;

[0042] Figure 3 It is a multi-camera perception picture of the unfamiliar environment state recognition model training method of the present invention;

[0043] Figure 4 It is the Camera BEV feature of the unfamiliar environment state recognition model training method of the present invention;

[0044] Figure 5 This is a test scenario for introducing state differences in the unfamiliar environment state cognitive model training method of the present invention. DETAILED DESCRIPTION

[0045] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0046] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0047] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0048] See also Figure 1 to Figure 5 The present invention provides a method for training a cognitive model of an unfamiliar environment state in an online driving scenario, the process is as follows Figure 1 As shown, it can be mainly divided into the following stages:

[0049] S1: In the offline training scenario, adaptive cruise control and lane keeping assist are the primary learning tasks. By loading the three-dimensional driving scene model, building a hybrid vehicle model, and arranging the corresponding RGB camera model according to the camera internal and external parameter matrix of the nuScenes dataset, multi-angle driving picture acquisition of the driving position at each moment is achieved.

[0050] Step S1 specifically includes the following steps:

[0051] S11: Based on the training method of deep reinforcement learning control strategy, the state space S, action space A and reward function R of adaptive cruise control strategy and lane keeping assist strategy are set;

[0052] S ACC =(VehSpd,TgtSpd,TrsGear,WhlTrq,CurVat,Slope)

[0053] A ACC =[-1,1]Activation function = tanh

[0054] R ACC = -1×[abs(A ACC (t)-A Hybrid (t-1))+abs(VehSpd(t)-TgtSpd(t))]

[0055] S LKA =(VehSpd,AngDif,YawRat,PreLka,LatOff,CurVat)

[0056] A LKA =[-π / 2,π / 2]Activation function = tanh×π / 2

[0057] R LKA = -1×[abs(A LKA (t)-A PreLka (t))+abs(LatOff)]

[0058] Among them, S ACC is the state of the adaptive cruise control strategy, VehSpd is the current vehicle speed, TgtSpd is the target vehicle speed, TrsGear is the transmission gear, WhlTrq is the torque demand, CurVat is the road curvature, Slope is the road slope, A ACC is the output action of deep reinforcement learning, R ACC is the reward for the adaptive cruise control strategy, A Hybridis the hybrid action that actually controls the vehicle, AngDif is the real-time angle difference, YawRat is the yaw rate, PreLka is the expected steering angle, LatOff is the lateral offset relative to the center lane, S LKA is the state of lane keeping assist, A LKA It is the action of lane keeping assist, R LKA is a bonus for lane keeping assist, A PreLka It is the expected steering action predicted based on the characteristics of the road ahead;

[0059] S12: Load the scene for offline training in the autonomous driving 3D simulation software CARLA, and load Figure 2 The Citroen C3 shown is used to characterize the body model. At the same time, a parallel hybrid vehicle model is built in the Simulink environment, which mainly includes the driver model, controller model (engine, transmission, hybrid control), power system and dynamics model (engine, motor, power battery, transmission, body, tire, suspension, etc.).

[0060] S13: Arrange RGB cameras at the six camera extrinsic parameter matrix positions corresponding to the Citroen C3 body model to collect real-time scene images of the vehicle driving in a three-dimensional scene, including: front view, front left view, front right view, rear view, rear left view and rear right view, and obtain Figure 3 Images of size 1600 × 900 are shown as perceptual features.

[0061] S2: Before the simulation training environment is started, the Camera BEV features of the pure visual feature extraction stream are obtained based on the real-time driving images collected by 6 RGB cameras with different viewing angles, and combined with the original two-dimensional variable state tensor. Subsequently, the proximal policy optimization algorithm based on the Actor-Critic architecture is used as the intelligent agent, and the concept of the Guard environmental cognition network is proposed to form an Actor-Critic-Guard network architecture that can fully realize the three functions of policy learning-policy evaluation-environmental cognition.

[0062] Step S2 specifically includes the following steps:

[0063] S21: Before the simulation training environment is started, six RGB cameras with different viewing angles are used to collect driving images in real time. These images are processed by the BEV Fusion algorithm to generate Figure 4 The BEV features shown are of dimensions 1×80×128×128. This process fuses the visual information from multiple cameras into a unified bird’s-eye view, making subsequent processing more efficient and accurate. To further optimize feature extraction, a convolutional neural network is used to extract the S BEVPerform dimensionality reduction and then compare with S ACC and S LKA Finally, a fully connected network is used to integrate these states to provide multimodal input information for the training of deep reinforcement learning control strategies.

[0064] S22: Deep reinforcement learning algorithm based on Actor-Critic architecture - proximal policy optimization As an intelligent agent learning the energy-saving driving strategy of hybrid electric vehicles, the optimal control strategy of the current training environment is fitted by the Actor, and the Critic is used to evaluate the optimality of the current strategy. At the same time, in order to further enhance the environmental adaptability and recognition ability of the intelligent agent, actively distinguish the unfamiliar state that may appear at any time and the dangerous actions that may be caused by it, the concept of "Guard environmental cognition network" is proposed. The core purpose is to analyze the state differences caused by the dynamic environment in real time, and take over measures in a timely and safe manner after approaching the boundary of the known space. In this regard, an innovative "Actor-Critic-Guard" network architecture combining strategy learning-strategy evaluation-environmental cognition is constructed.

[0065] S3: During the training of the simulated driving scenario, the traditional Actor-Critic architecture training process continues according to the original plan for the data-driven deep reinforcement learning hybrid vehicle control strategy. At the same time, in order to effectively train the Guard environment recognition network, the loss value and L2 regularization term of the Actor, Critic and Guard are simultaneously included in the loss function, so that the Actor and Critic networks will not begin to recognize the training environment state characteristics corresponding to the current optimal strategy until they have achieved a stable fit to the current training environment.

[0066] Step S3 specifically includes the following steps:

[0067] S31: In the training of the control strategy of hybrid electric vehicles based on data-driven deep reinforcement learning, the traditional Actor-Critic architecture training process calculates the updated gradient according to the loss function shown in the following formula, where the goal of the Actor is to maximize the expected return and the goal of the Critic is to accurately estimate the value function of the current strategy. In order to prevent the neural network from overfitting, enhance the generalization of the neural network and suppress the plasticity loss of the neural network, the L2 regularization term is added to the loss function of the original value function;

[0068]

[0069] L Critic =E[(V θ (s)-R) 2 ]+L Critic-L2

[0070] Among them, L Actor is the loss value of the actor network, L Critic is the loss of the critic network, π θ (a|s) is the probability distribution of the current strategy, is the probability distribution of the old strategy, is the advantage estimate, ε is the clipping parameter, V θ (s) is the output of the Critic’s value function, R is the return value calculated based on the actual reward and advantage calculation, and L Critic-L2 With L Actor-L2 It is the L2 regularization term about the weights of the Critic and Actor networks.

[0071] S32: In order to effectively train the Guard Environment Cognition Network, the loss function includes the loss values ​​and L2 regularization terms of the Actor, Critic, and Guard. Therefore, only after the Actor and Critic networks have achieved stable learning of the current training environment, will they begin to recognize the training environment state characteristics corresponding to the current optimal strategy. The training of the Guard Environment Cognition Network is mainly based on the following formula L Guard As shown, it specifically includes comprehensive indicators of multiple loss functions:

[0072] L Guard =L Critic +L Actor +L Guard-Label +L Guard-L2

[0073] Among them, L Guard is the loss value of the critic network, L Guard-Label is the loss that guides Guard to learn the environment cognitive label, and L Guard-L2 It is the L2 regularization term about the Guard network weight.

[0074] S4: After the offline simulation training is completed, a more complex and changeable path trajectory that is close to the real world is loaded as a test condition, and a large and diverse state deviation is introduced to the test environment, such as trajectory characteristics, lighting conditions, weather changes and vehicle status. Then, the Guard environmental cognition model is evaluated in a dynamic driving scenario to determine the scope of application that needs to be improved and enhanced.

[0075] Step S4 specifically includes the following steps:

[0076] S41: After the offline simulation training is completed, in order to fully highlight and verify the discrimination effect of the Guard environment cognition network, a more complex and changeable path trajectory that is close to the real world is loaded as a test condition, and a state deviation with large differences and diversity is introduced into the test environment, such as Figure 5 As shown: trajectory characteristics (more variable road properties, steeper slopes, sharper curves, etc.), lighting conditions (night, early morning, noon, dusk, etc.), weather changes (sunny, rainy, snowy, foggy, haze, dust, etc.) and vehicle status (higher speed, larger charge and discharge amplitude, more drastic torque and speed change range).

[0077] S42: In a test scenario for a dynamic driving environment, verify the evaluation results of the "Guard Environmental Cognition Model" on the random state tensor, and observe the control effect of the Actor network on the hybrid vehicle. Once the real-time state tensor changes outside the range of the historical sample space, promptly turn on the takeover mode of the safety strategy, and continue to collect unfamiliar samples to complete the continuous learning of the optimal strategy corresponding to the new state in the test scenario. Among them, the most critical thing is whether the "Guard Environmental Cognition Network" is accurate and reliable in its dynamic analysis of the current environment tensor, which directly affects the reliability and safety indicators of the deep reinforcement learning hybrid vehicle control strategy.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for training a cognitive model of unfamiliar environment state in an online driving scenario, characterized in that: The method specifically comprises the following steps: S1: In the offline training scenario, adaptive cruise control and lane keeping assist are the primary learning tasks. By loading the three-dimensional driving scene model, a hybrid vehicle model is built, and the corresponding RGB camera model is arranged according to the camera internal and external parameter matrix of the nuScenes dataset, the multi-angle driving picture acquisition of each driving position is realized; S2: Before the simulation training environment is started, the Camera BEV features of the pure visual feature extraction stream are obtained based on the real-time driving images collected by six RGB cameras with different viewing angles, and combined with the original two-dimensional variable state tensor; then, the proximal policy optimization algorithm based on the Actor-Critic architecture is used as the intelligent agent, and the concept of "Guard environmental cognition network" is proposed to form an "Actor-Critic-Guard" network architecture that can fully realize the three functions of policy learning-policy evaluation-environmental cognition; S3: During the training of the simulated driving scenario, the traditional Actor-Critic architecture training process continues according to the original plan for the data-driven deep reinforcement learning hybrid vehicle control strategy. At the same time, in order to effectively train the Guard environment recognition network, the loss values ​​and L2 regularization terms of the Actor, Critic, and Guard are synchronously included in the loss function. Therefore, after the Actor and Critic networks have achieved a stable fit to the current training environment, they will begin to recognize the training environment state characteristics corresponding to the current optimal strategy. S4: After the offline simulation training is completed, complex and changeable path trajectories that are close to the real world are loaded as test conditions, and different and diverse state deviations are introduced to the test environment. Then, the "Guard Environmental Cognition Network" is evaluated in dynamic driving scenarios to determine the scope of application that needs to be improved and enhanced.

2. The method for training a cognitive model of an unfamiliar environment state in an online driving scenario according to claim 1, characterized in that: In step S1, the modeling process of offline training specifically includes the following steps: S11: Based on the training method of deep reinforcement learning control strategy, the state space S, action space A and reward function R of adaptive cruise control strategy and lane keeping assist strategy are set; S ACC =(VehSpd,TgtSpd,TrsGear,WhlTrq,CurVat,Slope) A ACC =[-1,1]Activation function = tanh R ACC =-1×[abs(A ACC (t)-A Hybrid (t-1))+abs(VehSpd(t)-TgtSpd(t))] S LKA =(VehSpd,AngDif,YawRat,PreLka,LatOff,CurVat) A LKA =[-π / 2,π / 2]Activation function = tanh×π / 2 R LKA =-1×[abs(A LKA (t)-A PreLka (t))+abs(LatOff)] Among them, S ACC is the state of the adaptive cruise control strategy, VehSpd is the current vehicle speed, TgtSpd is the target vehicle speed, TrsGear is the transmission gear, WhlTrq is the torque demand, CurVat is the road curvature, Slope is the road slope, A ACC is the output action of deep reinforcement learning, R ACC is the reward for the adaptive cruise control strategy, A Hybrid is the hybrid action that actually controls the vehicle, AngDif is the real-time angle difference, YawRat is the yaw rate, PreLka is the expected steering angle, LatOff is the lateral offset relative to the center lane, S LKA is the state of lane keeping assist, A LKA It is the action of lane keeping assist, R LKA is a bonus for lane keeping assist, A PreLka It is the expected steering action predicted based on the characteristics of the road ahead; S12: Load the scenario for offline training in the autonomous driving 3D simulation software CARLA, and load the Citroen C3 used to characterize the body model. At the same time, build a parallel hybrid vehicle model in the Simulink environment, including the driver model, controller model, power system, and dynamics model; S13: RGB cameras are arranged at the six camera extrinsic parameter matrix positions corresponding to the Citroen C3 body model to collect real-time scene images of the vehicle driving in a three-dimensional scene, including: front view, front left view, front right view, rear view, rear left view and rear right view, and an image with a size of 1600×900 is obtained as a perception feature.

3. The method for training a cognitive model of an unfamiliar environment state in an online driving scenario according to claim 2, characterized in that: Step S2 specifically includes the following steps: S21: Before the simulation training environment is started, six RGB cameras with different viewing angles are used to collect driving images in real time. These images are processed by the BEV Fusion algorithm to generate BEV features with dimensions of 1×80×128×128. The convolutional neural network is used to generate the S BEV Perform dimensionality reduction and then compare with S ACC and S LKA The variable state tensors are combined; finally, these states are integrated using a fully connected network to provide multimodal input information for the training of deep reinforcement learning control strategies; S22: Deep reinforcement learning algorithm based on Actor-Critic architecture - proximal policy optimization As an intelligent agent, learn the energy-saving driving strategy of hybrid electric vehicles, fit the optimal control strategy of the current training environment through Actor, and use Critic to evaluate the optimality of the current strategy; at the same time, actively distinguish the unfamiliar state that may appear at any time and the dangerous actions that may be caused by it, and propose the concept of "Guard environmental cognition network"; analyze the state differences caused by the dynamic environment in real time, and take timely and safe takeover measures after approaching the boundary of the known space; in this regard, construct the "Actor-Critic-Guard" network architecture that combines strategy learning-strategy evaluation-environmental cognition.

4. The method for training a cognitive model of an unfamiliar environment state in an online driving scenario according to claim 1, characterized in that: In step S3, the training steps of the simulated driving scene are specifically as follows: S31: In the training of the data-driven deep reinforcement learning hybrid electric vehicle control strategy, the traditional Actor-Critic architecture training process calculates the updated gradient according to the loss function shown in the following formula, where the goal of the Actor is to maximize the expected return, and the goal of the Critic is to accurately estimate the value function of the current strategy. The L2 regularization term is added to the loss function of the original value function; Among them, L Actor is the loss value of the actor network, L Critic is the loss of the critic network, π θ (a|s) is the probability distribution of the current strategy, is the probability distribution of the old strategy, is the advantage estimate, ε is the clipping parameter, V θ (s) is the output of the Critic’s value function, R is the return value calculated based on the actual reward and advantage calculation, and L Critic-L2 With L Actor-L2 It is the L2 regularization term about the weights of the Critic and Actor networks; S32: In order to effectively train the "Guard Environment Cognition Network", the loss function simultaneously includes the loss values ​​and L2 regularization terms of the Actor, Critic, and Guard. Therefore, only after the Actor and Critic networks have achieved stable learning of the current training environment, will they begin to recognize the training environment state characteristics corresponding to the current optimal strategy. The training of the "Guard Environment Cognition Network" is based on the following formula L Guard As shown, it specifically includes comprehensive indicators of multiple loss functions: L Guard =L Critic +L Actor +L Guard-Label +L Guard-L2 Among them, L Guard is the loss value of the guardian network, L Guard-Label is the loss that guides Guard to learn the environment cognitive label, and L Guard-L2 It is the L2 regularization term about the Guard network weight.

5. The method for training a cognitive model of an unfamiliar environment state in an online driving scenario according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41: After the offline simulation training is completed, a complex and changeable path trajectory that is close to the real world is loaded as a test condition, and a state deviation with large differences and diversity is introduced into the test environment; S42: In a test scenario for a dynamic driving environment, verify the evaluation results of the "Guard Environmental Cognition Network" on the random state tensor, and observe the control effect of the Actor network on the hybrid vehicle; once the real-time state tensor changes outside the range of the historical sample space, promptly start the takeover mode of the safety policy, and continue to collect unfamiliar samples to complete the continuous learning of the optimal strategy corresponding to the new state in the test scenario.

Images