Deep reinforcement learning type body-equipped intelligent method for hybrid electric vehicle

Abstract

The invention relates to a deep reinforcement learning type intelligent method with a body for a hybrid electric vehicle, and belongs to the crossing field of new energy vehicles and artificial intelligence. The method comprises the following steps: S1, customizing an offline training and modeling online test scheme; s2, constructing a real-time driving environment BEV sensing scheme; loading an offline training scene and a self-defined test map in the automatic driving simulation software, and configuring a camera according to the nuScenes data set; after scene, vehicle and sensor modeling is completed, key traffic elements are segmented and detected based on a BEV Fusion algorithm; s3, a high-dimensional Camera BEV tensor is obtained through a BEV Fusion algorithm, and an Alpha HEV body intelligent model is trained by using a deep reinforcement learning algorithm; and S4, the Alpha HEV body intelligent model is verified.

Description

Technical Field

[0001] The present invention belongs to the intersection of new energy vehicles and artificial intelligence, and relates to a deep reinforcement learning-based embodied intelligence method for hybrid vehicles. Background Art

[0002] New energy vehicles, especially hybrid vehicles, play an important role in promoting energy transformation and alleviating the energy crisis. Hybrid vehicles combine the advantages of internal combustion engines and electric drive systems, meeting long-distance driving needs while achieving lower emissions in cities.

[0003] Artificial intelligence technology, especially the progress of deep learning and reinforcement learning, has promoted the development of the AI ​​era, but in real driving environments, deep reinforcement learning faces challenges such as the gap between simulation and reality, credibility, and sample efficiency. At the same time, as a new concept, embodied intelligence refers to an intelligent system that closely integrates the perception and action of an intelligent agent with its physical body and environment. Unlike traditional artificial intelligence, embodied intelligence emphasizes interaction and learning in the real world, using sensors to obtain environmental information, and providing feedback and adjustments through actual actions. It usually integrates multimodal perception capabilities, enabling intelligent agents to make real-time decisions in dynamic environments, thus playing an important role in robotics, autonomous driving, intelligent manufacturing and other fields. In this way, embodied intelligence will be able to better adapt to complex environments and enhance its ability to collaborate with humans.

[0004] Therefore, a deep reinforcement learning-based embodied intelligence method is urgently needed to enhance the intelligent driving technology of hybrid vehicles. The "Alpha HEV Embodied Intelligence Model" is trained by deep reinforcement learning algorithms, focusing on energy-saving driving, and promoting the development of intelligent energy-saving driving technology by improving environmental perception and optimizing driving decisions. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a deep reinforcement learning-based embodied intelligence method for hybrid vehicles, aiming at the coordinated promotion and development of new energy hybrid vehicles and artificial intelligence technology, training the "Alpha HEV embodied intelligence model" through a deep reinforcement learning algorithm, focusing on energy-saving driving, and promoting the development of intelligent energy-saving driving technology by improving environmental perception and optimizing driving decisions.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles, comprising the following steps:

[0008] S1: Customized offline training and modeling online testing solutions;

[0009] The vehicle is equipped with a global positioning system, an inertial measurement unit, and a monocular camera to collect road trajectory data from the real world. Then, a waypoint file in XDOR format is generated to create a custom map as a simulated test scenario for the digital twin. At the same time, 80% of the actual slope is used for the official Curved Road map of MATLAB as a training scenario.

[0010] S2: Build a real-time driving environment BEV perception solution;

[0011] Load offline training scenarios and custom test maps in the autonomous driving simulation software, and configure cameras based on the nuScenes dataset. After completing the scene, vehicle, and sensor modeling, segment and detect key traffic elements based on the BEV Fusion algorithm to improve environmental perception and detection accuracy. The perception results provide basic features for deep reinforcement learning-based embodied intelligence.

[0012] S3: Training the “Alpha HEV Embodied Intelligence Model”;

[0013] The high-dimensional Camera BEV tensor is obtained through the BEV Fusion algorithm, and the "Alpha HEV Embodied Intelligence Model" is trained using a deep reinforcement learning algorithm; the model focuses on four core learning tasks for vehicle energy-saving driving, including adaptive cruise control, lane keeping assist, energy management strategy, and transmission shifting strategy;

[0014] S4: Verify the “Alpha HEV Embodied Intelligence Model”;

[0015] After completing offline training, the applicability and reliability of the "Alpha HEV Embodied Intelligence Model" are directly verified in the test scenario, while maintaining the online update effect of the critic network; by creating a guardian network to cope with unfamiliar environments, and testing its real-time status recognition and reliability assessment; at the same time, after scene reconstruction in the cloud, imitation learning is used to quickly fit control commands to enhance training for new scenarios.

[0016] Furthermore, in step S1, the offline training and modeling online testing scheme is customized, specifically including the following steps:

[0017] S11: Through vehicles equipped with sensing equipment such as global positioning systems, inertial measurement units, and monocular cameras, the real-world trajectory is actually collected, that is, the longitude and latitude, altitude, and yaw angle ψ, pitch angle θ, and roll angle φ in the target path during driving;

[0018] For example, for the target path from Area A of Chongqing University in Shapingba District, Chongqing to Huxi Campus of Chongqing University, data on latitude and longitude, altitude, and yaw angle ψ, pitch angle θ, and roll angle φ during driving are obtained;

[0019] S12: Based on the measured target path data, after data processing and coordinate system conversion, the actual data in meters is obtained; since GPS has obvious and serious measurement deviations in artificially modified terrains such as tunnels and bridges, the slope data originally required is replaced by the pitch angle trajectory data, and MATLAB is used to generate a waypoint file in XDOR format;

[0020] S13: Using RoadRunner modeling software, we added extra lanes, defined road types, set road boundaries, and drew lane lines and other road features. Then, we used Unreal Engine to draw terrain and other landmarks. After packaging, we loaded it into the autonomous driving simulation software CARLA as a custom map, and used it as a test scenario for the deep reinforcement learning-based embodied intelligence solution for hybrid vehicles. At the same time, we loaded 80% of the real slope into the path trajectory of the MALAB official map Curved Road, and used it as a training scenario. At this point, the difference between the training scenario and the test scenario is mainly reflected in the state characteristics of the road slope, curvature and maximum speed.

[0021] Further, in step S2, a real-time driving environment BEV perception solution is constructed, which specifically includes the following steps:

[0022] S21: In CARLA, not only the custom map model from the real world is loaded, but also the camera installation position is arranged according to the sensor acquisition equipment specifications of the nuScenes dataset and the internal and external parameters of the monocular RGB camera in the dataset. The specific external parameters are shown in the following formula; if there is a need for research and application of equipment such as depth cameras, lidars, and millimeter-wave radars, they can be placed in the corresponding positions according to the reference position of the vehicle coordinate system, so as to facilitate the subsequent development of multi-modal fusion perception algorithms;

[0023]

[0024] S22: After completing the modeling and loading of scenes, vehicles, and sensors, the key traffic elements in the current driving scene are segmented and detected based on the BEV Fusion algorithm. The BEV Fusion algorithm can effectively integrate information from multiple perspectives by mapping data from different sensors to a bird's-eye view, providing a more comprehensive environmental perception. This method can not only improve the understanding of the surrounding environment, but also accurately identify and locate roads, obstacles, pedestrians, and other traffic participants. The processed data is subjected to feature extraction and segmentation by a deep learning model to separate key traffic elements from the background. The feature fusion strategy in this process enables the algorithm to overcome the limitations of single sensor data and achieve higher detection accuracy and robustness. Ultimately, the generated perception results will serve as the basic input features of the subsequent deep reinforcement learning control strategy model. These features include not only dynamic information such as position and speed, but also changes in traffic signs, lane lines, and the surrounding environment, providing more accurate and rich decision-making basis for the intelligent driving system, thereby improving the safety and reliability of the embodied intelligent model.

[0025] Further, in step S3, training the "Alpha HEV embodied intelligence model" specifically includes the following steps:

[0026] S31: After the BEV Fusion algorithm completes environmental perception, a four-dimensional space tensor is obtained, in which each dimension represents the number of batch samples, the number of feature layers, the length of the feature layer, and the width of the feature layer. At this time, the deep reinforcement learning algorithm is used as a development tool, the intelligent hybrid vehicle is used as the research and development object, and the whole vehicle energy-saving driving is used as the learning task to train the "Alpha HEV embodied intelligent model" that integrates AI experienced drivers and AI professional engineers. This includes four core tasks: adaptive cruise control (ACC), lane keeping assist (LKA), energy management strategy (EMS) and transmission shift strategy (TSS), etc.

[0027] S32: The training concept of adaptive cruise control based on deep reinforcement learning uses a hybrid action strategy, which is combined with a proportional-integral-derivative (PID)-based controller. The speed deviation is used as a weight to make the deep reinforcement learning agent maintain a stable following speed as much as possible. If speed loss occurs, especially when the deviation exceeds 1m / s, the PID-based controller will quickly stabilize the actual speed.

[0028] R ACC = -1×[abs(A ACC (t)-A Hybrid (t-1))+abs(VehSpd(t)-TgtSpd(t))]

[0029] Among them, R ACC is a bonus for adaptive cruise control, A ACC is the action output by reinforcement learning, A Hybrid is the mixed action that actually controls the vehicle, VehSpd is the vehicle speed, and TgtSpd is the target speed;

[0030] S33: The deep reinforcement learning-based lane keeping assist training concept uses a desired steering angle based on angular deviation and lateral deviation, where the output action directly controls the steering wheel angle; angular deviation refers to the difference between the vehicle heading and the direction of the road ahead at a short distance, while lateral deviation is converted to an angle value based on the same longitudinal distance and lateral offset distance; by combining these two correction angles, the goal of reducing both angular deviation and lateral deviation is achieved; in addition, based on the reference angle, an additional optimization term for lateral deviation is added to aim to stay as close to the center of the lane as possible while maintaining the correct driving direction;

[0031] R LKA = -1×[abs(A LKA (t)-A PreLka (t))+abs(LatOff)]

[0032] Among them, R LKA is a bonus for lane keeping assist, A LKA is the steering action actually output by reinforcement learning, A PreLka It is the steering action predicted based on the characteristics of the road ahead, and LatOff is the lateral offset;

[0033] S34: The energy management strategy training concept based on deep reinforcement learning is committed to ensuring that the engine torque is maintained in an efficient fuel economy zone by controlling the throttle opening, and making the operating point as close as possible to the pre-calibrated optimal operating line; at the same time, the energy management strategy must maintain the battery charge state within a stable and reliable range, especially between 40% and 80%, which helps to significantly extend the battery life while actively preventing phenomena such as overcharging and over-discharging; therefore, under the influence of rewards, when hybrid power distribution is required, the desired engine operating area will be concentrated on the efficient side of the optimal operating curve;

[0034]

[0035] Among them, R EMS is the reward of the energy management strategy, BSFC is the brake specific fuel consumption, T eng is the engine torque, T maxeng and T mineng are the maximum and minimum engine torque, T OOLis the calibrated optimal operating line of the engine, and ε is a minimum constant to prevent division by zero;

[0036] S35: The concept of training a transmission shift strategy based on deep reinforcement learning is very straightforward, involving continuous adjustment of the gear ratio to keep the power source operating at about 2500 rpm; it should be noted that previous studies have concluded that deep reinforcement learning is not suitable for learning intermittent discrete strategies, such as the shift strategy for a step-variable transmission and the mode switching strategy for a multi-mode hybrid system, due to the instability of its own exploration-exploitation characteristics;

[0037] R TSS = -1×[abs(MotSpd(t)-MotSpd target )]

[0038] Among them, R TSS is the reward of the transmission shift strategy, MotSpd is the real-time motor speed, MotSpd target is the target motor speed, which can be defined as 2500 rpm here by combining the engine efficiency characteristic diagram and the motor efficiency characteristic diagram.

[0039] Further, in step S4, verifying the "Alpha HEV embodied intelligence model" specifically includes the following steps:

[0040] S41: After offline training, we obtain the "Alpha HEV embodied intelligence model", which is the optimal control strategy fitted by the neural network parameters of the actor network. All policy models are directly loaded into the test scenario from the real world. Under the condition of a small learning rate, the deep reinforcement learning agent is always in update mode. In this way, on the one hand, the applicability and generalization ability of the actor network in a new and unfamiliar test environment are verified, and on the other hand, the continuous evaluation and update effect of the critic network on the actor network in an online environment is evaluated.

[0041] S42: For the aforementioned offline training environment, a guardian network is created and trained for the recognition and identification of unfamiliar environments; in a new test environment, the guardian network is mainly tested for its ability to discriminate the real-time state tensor of the deviation and historical sample space range, as well as the evaluation results of membership and reliability; once the evaluation result is lower than the default reliability threshold, the energy-saving driving strategy of the deep reinforcement learning-based intelligent hybrid vehicle will gradually become invalid, and will be replaced by a "bottom line" control strategy with safety control; in addition, for new and unfamiliar state characteristics, the imitation learning method is first used to fit reliable control commands as quickly as possible, and then the training of new scenarios is enhanced through cloud-based reconstruction of unfamiliar scenarios.

[0042] The beneficial effects of the present invention are as follows: the present invention trains the "Alpha HEV embodied intelligence model" through a deep reinforcement learning algorithm, focuses on energy-saving driving, and promotes the development of intelligent energy-saving driving technology by improving environmental perception and optimizing driving decisions.

[0043] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 It is an overall flow chart of the deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles of the present invention;

[0046] Figure 2 It is the real vehicle collection picture of the method of the present invention;

[0047] Figure 3 It is an offline training scenario of the method of the present invention;

[0048] Figure 4 is a custom test map of the method of the present invention;

[0049] Figure 5 It is the Camera BEV feature of the method of the present invention. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0052] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0053] See also Figure 1 to Figure 5 The present invention provides a deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles, the process is as follows Figure 1 As shown, it can be mainly divided into the following stages:

[0054] S1: Road trajectory data from the real world is collected by a vehicle equipped with a global positioning system, an inertial measurement unit, and a monocular camera. Then a waypoint file in XDOR format is generated to create a custom map as a simulated test scenario for the digital twin. At the same time, an 80% real slope is used for the official Curved Road map of MATLAB as a training scenario.

[0055] Step S1 specifically includes the following steps:

[0056] S11: First, through Figure 2 The vehicle shown is equipped with a global positioning system, an inertial measurement unit, a monocular camera and other sensor equipment, and actually collects trajectories from the real world. For example, a target path from Chongqing University Area A in Shapingba District, Chongqing to Chongqing University Huxi Campus, and then obtains data on latitude and longitude, altitude, and real-time yaw angle ψ, pitch angle θ and roll angle φ during vehicle driving.

[0057] S12: Then, based on the measured target path data, after data processing and coordinate system conversion, the actual data in meters is obtained. Since GPS has obvious and serious measurement deviations in artificially modified terrains such as tunnels and bridges, the pitch angle trajectory data is low-pass filtered to replace the originally required slope data, and MATLAB is used to generate a waypoint file in XDOR format.

[0058] S13: Finally, the RoadRunner modeling software was used to completely add additional lanes, define road surface types, set road boundaries, and draw road features such as lane lines. Subsequently, Unreal Engine was used to draw the terrain and other landmark references, including the Jialing River, University Town Tunnel, Xianglushan Tunnel, Chongqing University Area A campus features, and Chongqing University Huxi campus features. After the packaging operation, it was loaded into the autonomous driving simulation software CARLA as a custom map, and used as a test scenario for the deep reinforcement learning-based embodied intelligence solution for hybrid vehicles. At the same time, an 80% real slope was loaded into the path trajectory of the MALAB official map Curved Road, and used as a training scenario. At this point, Figure 3 The training scenario shown is similar to Figure 4 The differences between the test scenarios shown are mainly reflected in important state characteristics such as road slope, road curvature and maximum vehicle speed.

[0059] S2: Load offline training scenarios and custom test maps in the autonomous driving simulation software, and configure the camera according to the nuScenes dataset. After completing the scene, vehicle, and sensor modeling, segment and detect key traffic elements based on BEV Fusion to improve environmental perception and detection accuracy. The perception results provide basic features for deep reinforcement learning-based embodied intelligence.

[0060] Step S2 specifically includes the following steps:

[0061] S21: First, in CARLA, not only the custom map model from the real world is loaded, but also the camera installation position is arranged according to the sensor acquisition equipment specifications of the nuScenes dataset and the internal and external parameters of the monocular RGB camera in the dataset. The specific external parameters are shown in the following formula. If there is a need for research and application of equipment such as depth cameras, lidars, and millimeter-wave radars, they can be placed in the corresponding positions according to the reference position of the vehicle coordinate system, so as to facilitate the subsequent development of multi-modal fusion perception algorithms;

[0062]

[0063] S22: After completing the modeling and loading of scenes, vehicles, and sensors, the key traffic elements in the current driving scene are segmented and detected based on the BEV Fusion algorithm. The BEV Fusion algorithm can effectively fuse information from multiple perspectives by mapping data from different sensors to a bird's-eye view, providing a more comprehensive environmental perception. This method can not only improve the understanding of the surrounding environment, but also accurately identify and locate roads, obstacles, pedestrians, and other traffic participants. The processed data is subjected to feature extraction and segmentation by a deep learning model to separate key traffic elements from the background. The feature fusion strategy in this process enables the algorithm to overcome the limitations of single sensor data and achieve higher detection accuracy and robustness. Ultimately, the generated perception results will serve as the basic input features of the subsequent deep reinforcement learning control strategy model. These features include not only dynamic information such as position and speed, but also changes in traffic signs, lane lines, and the surrounding environment, providing more accurate and rich decision-making basis for the intelligent driving system, thereby improving the safety and reliability of the embodied intelligent model.

[0064] S3: The high-dimensional Camera BEV tensor is obtained through the BEV Fusion algorithm, and the "Alpha HEV Embodied Intelligence Model" is trained using a deep reinforcement learning algorithm. The model focuses on four core learning tasks for energy-saving driving of the vehicle, covering adaptive cruise control, lane keeping assist, energy management strategy, and transmission shifting strategy.

[0065] Step S3 specifically includes the following steps:

[0066] S31: Obtained after the BEV Fusion algorithm completes environmental perception Figure 5 The four-dimensional space tensor shown in the figure, where each dimension represents the number of batch samples, the number of feature layers, the length of the feature layer and the width of the feature layer. At this time, the deep reinforcement learning algorithm is used as the development tool, the intelligent hybrid vehicle is used as the research and development object, and the energy-saving driving of the whole vehicle is used as the learning task to train the "Alpha HEV embodied intelligent model" that integrates AI experienced drivers and AI professional engineers. Among them, energy-saving driving mainly includes 4 core tasks: Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Energy Management Strategy (EMS), Transmission Shifting Strategy (TSS), etc.;

[0067] S32: The training concept of adaptive cruise control based on deep reinforcement learning uses a hybrid action strategy, which is combined with a proportional-integral-derivative (PID)-based controller. The speed deviation is used as a weight to make the deep reinforcement learning agent maintain a stable following speed as much as possible. If speed loss occurs, especially when the deviation exceeds 1m / s, the PID-based controller will quickly stabilize the actual speed;

[0068] R ACC = -1×[abs(A ACC (t)-A Hybrid (t-1))+abs(VehSpd(t)-TgtSpd(t))]

[0069] Among them, A ACC is the action output by reinforcement learning, A Hybrid is the hybrid action that actually controls the vehicle, VehSpd is the vehicle speed, and TgtSpd is the target speed.

[0070] S33: The deep reinforcement learning-based lane keeping assist training concept uses a desired steering angle based on angular deviation and lateral deviation, where the output action directly controls the steering wheel angle. Angular deviation refers to the difference between the vehicle heading and the direction of the short distance ahead, while lateral deviation is converted to an angle value based on the same longitudinal distance and lateral offset distance. By combining these two correction angles, the goal of reducing both angular deviation and lateral deviation is achieved. In addition, on the basis of following the reference angle, additional optimization items for lateral deviation are added, aiming to stay as close to the center of the lane as possible while maintaining the correct driving direction;

[0071] R LKA = -1×[abs(A LKA (t)-A PreLka (t))+abs(LatOff)]

[0072] Among them, A LKA is the steering action actually output by reinforcement learning, A PreLka It is a steering action predicted based on the characteristics of the road ahead, and LatOff is a lateral offset.

[0073] S34: The energy management strategy training concept based on deep reinforcement learning is committed to ensuring that the engine torque is maintained in an efficient fuel economy zone by controlling the throttle opening, and making the operating point as close as possible to the pre-calibrated optimal operating line. At the same time, the energy management strategy must maintain the battery state of charge within a stable and reliable range, especially between 40% and 80%, which helps to significantly extend the battery life while actively preventing phenomena such as overcharging and over-discharging. Therefore, under the influence of rewards, when hybrid power distribution is required, the desired engine operating area will be concentrated on the efficient side of the optimal operating curve;

[0074]

[0075] Where BSFC is the brake specific fuel consumption, T eng is the engine torque, T maxeng and T mineng are the maximum and minimum engine torque, T OOL is the calibrated optimal operating line of the engine, and ε is a minimum constant that prevents division by zero.

[0076] S35: The concept of training a transmission shift strategy based on deep reinforcement learning is very intuitive, involving continuous adjustment of the gear ratio to keep the power source operating at about 2500 rpm. It should be noted that previous studies have concluded that deep reinforcement learning is not suitable for learning intermittent discrete strategies, such as the shift strategy for a step-variable transmission and the mode switching strategy for a multi-mode hybrid system, and the reason for this is the instability of its own exploration-exploitation characteristics;

[0077] R TSS = -1×[abs(MotSpd(t)-MotSpd target )]

[0078] Among them, MotSpd is the real-time motor speed, MotSpd target is the target motor speed, which can be defined as 2500 rpm here by combining the engine efficiency characteristic diagram and the motor efficiency characteristic diagram.

[0079] S4: After completing offline training, directly verify the applicability and reliability of the "Alpha HEV Embodied Intelligence Model" in the test scenario while maintaining the online update effect of the critic network. Create a guardian network to deal with unfamiliar environments and test its discrimination and reliability assessment of real-time status. At the same time, after scene reconstruction in the cloud, use imitation learning to quickly fit control commands to enhance training for new scenarios.

[0080] Step S4 specifically includes the following steps:

[0081] S41: After offline training, the "Alpha HEV embodied intelligence model" was obtained, which is the optimal control strategy fitted by the neural network parameters of the actor network. All policy models were directly loaded into the test scene from the real world. Under the condition of a small learning rate, the deep reinforcement learning agent was always in update mode. In this way, on the one hand, the applicability and generalization ability of the actor network in a new and unfamiliar test environment were verified, and on the other hand, the continuous evaluation and update effect of the critic network on the actor network in an online environment was evaluated.

[0082] S42: For the aforementioned offline training environment, a guardian network for unfamiliar environment cognition and identification was created and trained. In the new test environment, the guardian network is mainly tested for its ability to discriminate the real-time state tensor of the deviation and historical sample space range, as well as the evaluation results of membership and reliability. Once the evaluation result is lower than the default reliability threshold, the energy-saving driving strategy of the deep reinforcement learning-based intelligent hybrid vehicle will gradually become ineffective, and will be replaced by a "bottom-line" control strategy with safety control. In addition, for new and unfamiliar state features, the imitation learning method is first used to fit reliable control commands as quickly as possible, and then the training of new scenes is enhanced by cloud-based reconstruction of unfamiliar scenes.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles, characterized in that: The method specifically comprises the following steps: S1: Customized offline training and modeling online testing solutions; The vehicle is equipped with a global positioning system, an inertial measurement unit, and a monocular camera to collect road trajectory data from the real world. Then, a waypoint file in XDOR format is generated to create a custom map as a simulated test scenario for the digital twin. At the same time, 80% of the actual slope is used for the official Curved Road map of MATLAB as a training scenario. S2: Build a real-time driving environment BEV perception solution; Load offline training scenarios and custom test maps in the autonomous driving simulation software, and configure cameras based on the nuScenes dataset. After completing the scene, vehicle, and sensor modeling, segment and detect key traffic elements based on the BEV Fusion algorithm to improve environmental perception and detection accuracy. The perception results provide basic features for deep reinforcement learning-based embodied intelligence. S3: Training the "Alpha HEV Embodied Intelligence Model"; The high-dimensional Camera BEV tensor is obtained through the BEV Fusion algorithm, and the "AlphaHEV Embodied Intelligence Model" is trained using a deep reinforcement learning algorithm; the model focuses on four core learning tasks for energy-saving driving of the entire vehicle, including adaptive cruise control, lane keeping assist, energy management strategy, and transmission shifting strategy; S4: Verify the "Alpha HEV Embodied Intelligence Model"; After completing offline training, the applicability and reliability of the "Alpha HEV Embodied Intelligence Model" are directly verified in the test scenario, while maintaining the online update effect of the critic network; by creating a guardian network to cope with unfamiliar environments, and testing its real-time status recognition and reliability assessment; at the same time, after scene reconstruction in the cloud, imitation learning is used to quickly fit control commands to enhance training for new scenarios.

2. The deep reinforcement learning embodied intelligence method according to claim 1, characterized in that: In step S1, the offline training and modeling online testing scheme is customized, which specifically includes the following steps: S11: Through a vehicle equipped with a global positioning system, an inertial measurement unit, and a monocular camera, the real-world trajectory is actually collected, that is, the latitude and longitude, altitude, and yaw angle ψ, pitch angle θ, and roll angle φ in the target path during driving; S12: Based on the measured target path data, after data processing and coordinate system conversion, the actual data in meters is obtained; the slope data originally required is replaced by the pitch angle trajectory data, and the waypoint file in XDOR format is generated by MATLAB; S13: Using RoadRunner modeling software, we added extra lanes, defined road types, set road boundaries, and drew lane lines. Then, we used Unreal Engine to draw terrain and other landmarks. After packaging, we loaded it into the autonomous driving simulation software CARLA as a custom map and used it as a test scenario for the deep reinforcement learning-based embodied intelligence solution for hybrid vehicles. At the same time, we loaded the 80% real slope into the path trajectory of the Curved Road, the official MALAB map, and used it as a training scenario.

3. The deep reinforcement learning embodied intelligence method according to claim 1, characterized in that: In step S2, a real-time driving environment BEV perception solution is constructed, which specifically includes the following steps: S21: In CARLA, not only the custom map model from the real world is loaded, but also the camera installation positions are arranged according to the sensor acquisition device specifications of the nuScenes dataset and the internal and external parameters of the monocular RGB camera in the dataset. S22: After completing the modeling and loading of the scene, vehicle, and sensor, the key traffic elements in the current driving scene are segmented and detected based on the BEV Fusion algorithm; the BEV Fusion algorithm maps the data from different sensors to a bird's-eye view; the processed data is subjected to feature extraction and segmentation by a deep learning model to separate the key traffic elements from the background; finally, the generated perception results will serve as the basic input features of the subsequent deep reinforcement learning control strategy model.

4. The deep reinforcement learning embodied intelligence method according to claim 1, characterized in that: In step S3, training the "Alpha HEV embodied intelligence model" specifically includes the following steps: S31: After the BEV Fusion algorithm completes environmental perception, a four-dimensional space tensor is obtained, in which each dimension represents the number of batch samples, the number of feature layers, the length of the feature layer, and the width of the feature layer. At this time, the deep reinforcement learning algorithm is used as a development tool, the intelligent hybrid vehicle is used as the research and development object, and the whole vehicle energy-saving driving is used as the learning task to train the "Alpha HEV embodied intelligent model" that integrates AI experienced drivers and AI professional engineers. This includes four core tasks: adaptive cruise control ACC, lane keeping assist LKA, energy management strategy EMS, and transmission shift strategy TSS. S32: The training concept of adaptive cruise control based on deep reinforcement learning uses a hybrid action strategy and combines it with a PID-based controller. The speed deviation is used as a weight to enable the deep reinforcement learning agent to maintain a stable following speed. If speed loss occurs, especially when the deviation exceeds 1m / s, the PID-based controller quickly stabilizes the actual speed. R ACC =-1×[abs(A ACC (t)-A Hybrid (t-1))+abs(VehSpd(t)-TgtSpd(t))] Among them, R ACC is a bonus for adaptive cruise control, A ACC is the action output by reinforcement learning, A Hybrid is the mixed action that actually controls the vehicle, VehSpd is the vehicle speed, and TgtSpd is the target speed; S33: The lane keeping assist training concept based on deep reinforcement learning uses a desired steering angle based on angular deviation and lateral deviation, where the output action directly controls the steering wheel angle; angular deviation refers to the difference between the vehicle heading and the direction of the road ahead at a short distance, while lateral deviation is converted to an angle value based on the same longitudinal distance and lateral offset distance; by combining these two correction angles, the goal of reducing both angular deviation and lateral deviation is achieved; in addition, based on following the reference angle, an additional optimization term for lateral deviation is added to aim to stay close to the center of the lane while maintaining the correct driving direction; R LKA =-1×[abs(A LKA (t)-A PreLka (t))+abs(LatOff)] Among them, R LKA is the reward for lane keeping assist, A LKA is the steering action actually output by reinforcement learning, A PreLka It is the steering action predicted based on the characteristics of the road ahead, and LatOff is the lateral offset; S34: The energy management strategy training concept based on deep reinforcement learning is committed to ensuring that the engine torque is maintained in an efficient fuel economy zone by controlling the throttle opening and keeping the operating point close to the pre-calibrated optimal operating line; at the same time, the energy management strategy must maintain the battery state of charge within a stable and reliable range, especially between 40% and 80%, which helps to significantly extend the battery life while actively preventing overcharging and over-discharging; therefore, under the influence of rewards, when hybrid power distribution is required, the desired engine operating area will be concentrated on the efficient side of the optimal operating curve; Among them, R EMS is the reward value of the energy management strategy, BSFC is the brake specific fuel consumption, T eng is the engine torque, T maxeng and T mineng are the maximum and minimum engine torque, T OOL is the calibrated optimal operating line of the engine, and ε is a minimum constant to prevent division by zero; S35: The concept of training the transmission shift strategy based on deep reinforcement learning is very intuitive and involves continuous adjustment of the gear ratio to keep the operating speed of the power source at about 2500 rpm; R TSS =-1×[abs(MotSpd(t)-MotSpd target )] Among them, R TSS is the reward of the transmission shift strategy, MotSpd is the real-time motor speed, MotSpd target is the target motor speed, which can be defined as 2500 rpm here by combining the engine efficiency characteristic diagram and the motor efficiency characteristic diagram.

5. The deep reinforcement learning embodied intelligence method according to claim 1, characterized in that: In step S4, verifying the "Alpha HEV embodied intelligence model" specifically includes the following steps: S41: After offline training, the "Alpha HEV embodied intelligence model" is obtained, that is, the optimal control strategy fitted by the neural network parameters of the actor network; all policy models are directly loaded into the test scene from the real world, and under the condition of a small learning rate, the deep reinforcement learning agent is always in update mode S42: For offline training environments, create and train a guardian network for unfamiliar environment recognition and identification; in a new test environment, test the guardian network's ability to discriminate real-time state tensors within the range of deviation and historical sample space, as well as the evaluation results of membership and reliability; once the evaluation result is lower than the default reliability threshold, the energy-saving driving strategy of the deep reinforcement learning-based intelligent hybrid vehicle will gradually become ineffective, and will be replaced by a "bottom-line" control strategy with safety control; in addition, for new and unfamiliar state features, first use imitation learning to fit reliable control commands as quickly as possible, and then enhance training for new scenarios through cloud-based reconstruction of unfamiliar scenarios.

Images