Deep reinforcement learning-based embodied intelligence approach for hybrid electric vehicles

Abstract

This invention relates to a deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles (HEVs), which falls within the interdisciplinary field of new energy vehicles and artificial intelligence. The method includes: S1: customizing offline training and modeling online testing solutions; S2: constructing a BEV perception solution for real-time driving environments; loading offline training scenarios and custom test maps into autonomous driving simulation software, and configuring cameras based on the nuScenes dataset; after completing scene, vehicle, and sensor modeling, segmenting and detecting key traffic elements using the BEV Fusion algorithm; S3: obtaining a high-dimensional Camera BEV tensor using the BEV Fusion algorithm, and training an "Alpha HEV Embodied Intelligence Model" using a deep reinforcement learning algorithm; and S4: validating the "Alpha HEV Embodied Intelligence Model."

Description

Technical Field

[0001] The present invention belongs to the intersection of new energy vehicles and artificial intelligence, and relates to a deep reinforcement learning-based embodied intelligence method for hybrid vehicles. Background Art

[0002] New energy vehicles, especially hybrid vehicles, play an important role in promoting energy transformation and alleviating the energy crisis. Hybrid vehicles combine the advantages of internal combustion engines and electric drive systems, meeting long-distance driving needs while achieving lower emissions in urban areas.

[0003] Advances in artificial intelligence (AI), particularly deep learning and reinforcement learning, are driving the AI ​​era. However, in real-world driving environments, deep reinforcement learning faces challenges such as the gap between simulation and reality, credibility, and sample efficiency. Meanwhile, embodied intelligence, a newer concept, refers to intelligent systems that closely integrate the perception and actions of an intelligent agent with its physical body and environment. Unlike traditional AI, embodied intelligence emphasizes interaction and learning in the real world, using sensors to acquire environmental information and providing feedback and adjustments through actual actions. It often integrates multimodal perception capabilities, enabling agents to make real-time decisions in dynamic environments, thus playing a vital role in fields such as robotics, autonomous driving, and intelligent manufacturing. In this way, embodied intelligence will be able to better adapt to complex environments and enhance its ability to collaborate with humans.

[0004] Therefore, a deep reinforcement learning-based embodied intelligence approach is urgently needed to enhance the intelligent driving technology of hybrid vehicles. The "Alpha HEV Embodied Intelligence Model," trained through a deep reinforcement learning algorithm, focuses on energy-efficient driving. This model, by improving environmental perception and optimizing driving decisions, will promote the development of intelligent, energy-efficient driving technology. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a deep reinforcement learning-based embodied intelligence method for hybrid vehicles. Aiming at the coordinated promotion and development of new energy hybrid vehicles and artificial intelligence technologies, the "Alpha HEV Embodied Intelligence Model" is trained through a deep reinforcement learning algorithm, focusing on energy-saving driving, and promoting the development of intelligent energy-saving driving technology by improving environmental perception and optimizing driving decisions.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles includes the following steps:

[0008] S1: Customize offline training and modeling online testing solutions;

[0009] Real-world road trajectory data is collected using a vehicle equipped with a global positioning system, an inertial measurement unit, and a monocular camera. Then, a waypoint file in XDOR format is generated to create a custom map, which serves as a realistic test scenario for the digital twin. At the same time, an 80% real-world slope is used in the official Curved Road map in MATLAB as a training scenario.

[0010] S2: Build a real-time driving environment BEV perception solution;

[0011] Offline training scenarios and custom test maps are loaded into the autonomous driving simulation software, and cameras are configured based on the nuScenes dataset. After completing the scene, vehicle, and sensor modeling, key traffic elements are segmented and detected using the BEV Fusion algorithm to improve environmental perception and detection accuracy. The perception results provide the foundational features for deep reinforcement learning-based embodied intelligence.

[0012] S3: Training the Alpha HEV Embodied Intelligence Model;

[0013] The high-dimensional Camera BEV tensor is obtained through the BEV Fusion algorithm, and the "Alpha HEV Embodied Intelligence Model" is trained using a deep reinforcement learning algorithm. This model focuses on four core learning tasks for vehicle energy-saving driving, including adaptive cruise control, lane keeping assist, energy management strategy, and transmission shift strategy.

[0014] S4: Verify the “Alpha HEV Embodied Intelligence Model”;

[0015] After completing offline training, the applicability and reliability of the "Alpha HEV Embodied Intelligence Model" are directly verified in the test scenario, while maintaining the online update effect of the critic network; by creating a guardian network to cope with unfamiliar environments, and testing its judgment of real-time status and reliability assessment; at the same time, after scene reconstruction is achieved in the cloud, imitation learning is used to quickly fit control commands to enhance training for new scenarios.

[0016] Furthermore, in step S1, a customized offline training and modeling online testing scheme is specifically comprised of the following steps:

[0017] S11: Using a vehicle equipped with sensors such as a global positioning system, an inertial measurement unit, and a monocular camera, it collects real-world trajectories, including data on the latitude and longitude, altitude, and yaw angles ψ, pitch angles θ, and roll angles φ along the target path.

[0018] For example, for a target route from Area A of Chongqing University in Shapingba District, Chongqing to the Huxi Campus of Chongqing University, data on latitude and longitude, altitude, and the yaw angle ψ, pitch angle θ, and roll angle φ during driving are obtained;

[0019] S12: Based on the measured target path data, after data processing and coordinate system conversion, actual data in meters is obtained. Because GPS has obvious and serious measurement errors in artificially modified terrain such as tunnels and bridges, the required slope data is replaced by pitch angle trajectory data, and MATLAB is used to generate a waypoint file in XDOR format.

[0020] S13: Using RoadRunner modeling software, the team added additional lanes, defined road surface types, set road boundaries, and drew road features such as lane lines. Unreal Engine was then used to draw terrain and other landmarks. After packaging, the resulting map was loaded into the autonomous driving simulation software CARLA as a custom map, serving as a test scenario for a deep reinforcement learning-based embodied intelligence solution for hybrid electric vehicles. Furthermore, an 80% realistic slope was added to the path trajectory of the official MALAB map, Curved Road, and used as a training scenario. The differences between the training and test scenarios primarily reflected road slope, curvature, and maximum speed.

[0021] Furthermore, in step S2, a real-time driving environment BEV perception solution is constructed, specifically including the following steps:

[0022] S21: In CARLA, not only are custom map models from the real world loaded, but camera installation locations are also arranged according to the sensor acquisition device specifications of the nuScenes dataset and the intrinsic and extrinsic parameters of the monocular RGB camera in the dataset. The specific extrinsic parameters are shown in the following formula. If there are research and application requirements for devices such as depth cameras, lidars, and millimeter-wave radars, they can be placed in the corresponding positions based on the reference position of the vehicle coordinate system, facilitating the subsequent development of multimodal fusion perception algorithms.

[0023]

[0024] S22: After modeling and loading the scene, vehicle, and sensor data, the BEV Fusion algorithm segments and detects key traffic elements in the current driving scenario. By mapping data from various sensors to a bird's-eye view, the BEV Fusion algorithm effectively integrates information from multiple perspectives, providing a more comprehensive environmental perception. This approach not only improves understanding of the surrounding environment but also accurately identifies and locates roads, obstacles, pedestrians, and other traffic participants. A deep learning model extracts and segments the processed data, separating key traffic elements from the background. This feature fusion strategy enables the algorithm to overcome the limitations of single-sensor data and achieve higher detection accuracy and robustness. Ultimately, the resulting perception results serve as the fundamental input features for the subsequent deep reinforcement learning-based control strategy model. These features include not only dynamic information such as position and speed, but also changes in traffic signs, lane markings, and the surrounding environment. This provides the intelligent driving system with more accurate and comprehensive decision-making information, thereby enhancing the safety and reliability of the embodied intelligence model.

[0025] Furthermore, in step S3, the "Alpha HEV Embodied Intelligence Model" is trained, specifically including the following steps:

[0026] S31: After the BEV Fusion algorithm completes environmental perception, it generates a four-dimensional spatial tensor, where each dimension represents the number of batch samples, the number of feature layers, the length of a feature layer, and the width of a feature layer. Using deep reinforcement learning algorithms as development tools, intelligent hybrid vehicles as research targets, and vehicle-wide energy-saving driving as learning tasks, the "Alpha HEV Embodied Intelligence Model" is trained. This model integrates an experienced AI driver and a professional AI engineer. It includes four core tasks: Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Energy Management Strategy (EMS), and Transmission Shift Strategy (TSS).

[0027] S32: The deep reinforcement learning-based adaptive cruise control training concept uses a hybrid action strategy combined with a proportional-integral-derivative (PID)-based controller. Speed ​​deviation is weighted to ensure that the deep reinforcement learning agent maintains a stable following speed. If speed loss occurs, especially when the deviation exceeds 1 m / s, the PID-based controller quickly stabilizes the actual speed.

[0028] R ACC =-1×[abs(A ACC (t)-A Hybrid (t-1))+abs(VehSpd(t)-TgtSpd(t))]

[0029] Among them, R ACC is a bonus for adaptive cruise control, A ACC is the action output by reinforcement learning, A Hybrid is the hybrid action that actually controls the vehicle, VehSpd is the vehicle speed, and TgtSpd is the target speed;

[0030] S33: The deep reinforcement learning-based lane keeping assist training concept uses a desired steering angle based on angular deviation and lateral deviation, where the output action directly controls the steering wheel angle. Angular deviation refers to the difference between the vehicle's heading and the direction of the road ahead over a short distance, while lateral deviation is converted to an angle value based on the same longitudinal distance and lateral offset distance. By combining these two correction angles, the goal of simultaneously reducing angular deviation and lateral deviation is achieved. Furthermore, in addition to adhering to the reference angle, an additional optimization term for lateral deviation is added, aiming to keep the vehicle as close to the center of the lane as possible while maintaining the correct driving direction.

[0031] R LKA =-1×[abs(A LKA (t)-A PreLka (t))+abs(LatOff)]

[0032] Among them, R LKA Is the reward for lane keeping assist, A LKA is the steering action actually output by reinforcement learning, A PreLka It is the steering action predicted based on the characteristics of the road ahead, and LatOff is the lateral offset;

[0033] S34: The deep reinforcement learning-based energy management strategy training concept is dedicated to ensuring that the engine torque is maintained in the efficient fuel economy zone by controlling the throttle opening, and the operating point is kept as close as possible to the pre-calibrated optimal operating line. At the same time, the energy management strategy must maintain the battery state of charge within a stable and reliable range, especially between 40% and 80%, which helps significantly extend the battery life while actively preventing phenomena such as overcharging and excessive discharge. Therefore, under the influence of rewards, when hybrid power distribution is required, the desired engine operating area will be concentrated on the efficient side of the optimal operating curve.

[0034]

[0035] Among them, R EMS is the reward of the energy management strategy, BSFC is the brake specific fuel consumption, T eng is the engine torque, T maxeng and T mineng are the maximum and minimum engine torque, T OOLis the calibrated optimal operating line of the engine, and ε is a minimum constant that prevents division by zero;

[0036] S35: The concept of training transmission shift strategies based on deep reinforcement learning is straightforward, involving continuous adjustment of gear ratios to maintain the power source's operating speed at approximately 2500 rpm. It is important to note that previous research has concluded that deep reinforcement learning is not suitable for learning intermittent discrete strategies, such as shift strategies for stepped transmissions and mode switching strategies for multi-mode hybrid powertrains, due to the inherent instability of its exploration-exploitation characteristics.

[0037] R TSS = -1×[abs(MotSpd(t)-MotSpd target )]

[0038] Among them, R TSS is the reward of the transmission shift strategy, MotSpd is the real-time motor speed, MotSpd target is the target motor speed, which can be defined as 2500 rpm based on the engine efficiency characteristic map and the motor efficiency characteristic map.

[0039] Furthermore, in step S4, the "Alpha HEV Embodied Intelligence Model" is verified, which specifically includes the following steps:

[0040] S41: After offline training, we obtained the "Alpha HEV Embodied Intelligence Model," which is the optimal control policy fitted by the actor network's neural network parameters. All policy models were directly loaded into a real-world test scenario, and the deep reinforcement learning agent was always in update mode under a small learning rate. This not only verified the actor network's applicability and generalization ability in a new and unfamiliar test environment, but also evaluated the critic network's continuous evaluation and update of the actor network in an online environment.

[0041] S42: For the aforementioned offline training environment, a guardian network is created and trained for the recognition and identification of unfamiliar environments. In a brand-new test environment, the guardian network's discrimination effect on the real-time state tensor of the deviation and historical sample space range, as well as the evaluation results of membership and reliability, are mainly tested. Once the evaluation result is lower than the default reliability threshold, the energy-saving driving strategy of the deep reinforcement learning-based intelligent hybrid vehicle will gradually become invalid, and will be replaced by a "bottom-line" control strategy with safety control. In addition, for new and unfamiliar state characteristics, imitation learning is first used to fit reliable control commands as quickly as possible, and then the training of new scenarios is enhanced through cloud-based reconstruction of unfamiliar scenarios.

[0042] The beneficial effects of the present invention are: the present invention trains the "Alpha HEV embodied intelligence model" through a deep reinforcement learning algorithm, focuses on energy-saving driving, and promotes the development of intelligent energy-saving driving technology by improving environmental perception and optimizing driving decisions.

[0043] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0045] Figure 1 This is an overall flow chart of the deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles of the present invention;

[0046] Figure 2 This is the actual vehicle collection picture of the method of the present invention;

[0047] Figure 3 This is the offline training scenario of the method of the present invention;

[0048] Figure 4 is a custom test map for the method of the present invention;

[0049] Figure 5 It is the Camera BEV feature of the method of the present invention. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0052] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0053] See also Figures 1 to 5 The present invention provides a deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles. The process is as follows: Figure 1 As shown, it can be mainly divided into the following stages:

[0054] S1: Real-world road trajectory data was collected using a vehicle equipped with a global positioning system, an inertial measurement unit, and a monocular camera. Waypoint files in XDOR format were then generated and a custom map was created to serve as a realistic test scenario for the digital twin. An 80% realistic slope was used in the official Curved Road map in MATLAB as a training scenario.

[0055] Step S1 specifically includes the following steps:

[0056] S11: First, through Figure 2 The vehicle shown is equipped with sensors such as a global positioning system, an inertial measurement unit, and a monocular camera. It collects real-world trajectories, such as a target path from Area A of Chongqing University in Shapingba District, Chongqing, to the Huxi Campus of Chongqing University. This data is then used to obtain data on latitude and longitude, altitude, and the vehicle's real-time yaw angle ψ, pitch angle θ, and roll angle φ during driving.

[0057] S12: Based on the measured target path data, actual data in meters is obtained after data processing and coordinate system conversion. Because GPS has significant and serious measurement errors in artificially modified terrain such as tunnels and bridges, the pitch angle trajectory data is low-pass filtered to replace the originally required slope data. MATLAB is then used to generate a waypoint file in XDOR format.

[0058] S13: Finally, the RoadRunner modeling software was used to completely add additional lanes, define road surface types, set road boundaries, and draw lane lines and other road features. Subsequently, Unreal Engine was used to draw the terrain and other landmark references, including the Jialing River, University Town Tunnel, Xianglushan Tunnel, Chongqing University Area A campus features, and Chongqing University Huxi campus features. After the packaging operation, it was loaded into the autonomous driving simulation software CARLA as a custom map, and used as a test scenario for the deep reinforcement learning-based embodied intelligence solution for hybrid vehicles. At the same time, the 80% real slope was loaded into the path trajectory of the MALAB official map Curved Road, and used as a training scenario. At this point, Figure 3 The training scenario shown is similar to Figure 4 The differences between the test scenarios shown are mainly reflected in important state characteristics such as road slope, road curvature and maximum vehicle speed.

[0059] S2: Load offline training scenarios and custom test maps into the autonomous driving simulation software and configure cameras based on the nuScenes dataset. After completing the scene, vehicle, and sensor modeling, segment and detect key traffic elements based on the BEV Fusion, improving environmental perception and detection accuracy. These perception results provide the foundational features for deep reinforcement learning-based embodied intelligence.

[0060] Step S2 specifically includes the following steps:

[0061] S21: First, in CARLA, not only is a custom map model from the real world loaded, but the camera installation locations are also arranged according to the sensor acquisition device specifications of the nuScenes dataset and the internal and external parameters of the monocular RGB camera in the dataset. The specific external parameters are shown in the following formula. If there are research and application needs for devices such as depth cameras, lidars, and millimeter-wave radars, they can be placed in the corresponding positions based on the reference position of the vehicle coordinate system to facilitate the subsequent development of multimodal fusion perception algorithms.

[0062]

[0063] S22: After completing the modeling and loading of the scene, vehicle, and sensors, the BEV Fusion algorithm is used to segment and detect key traffic elements in the current driving scenario. The BEV Fusion algorithm effectively integrates information from multiple perspectives by mapping data from different sensors to a bird's-eye view, providing a more comprehensive environmental perception. This approach not only improves understanding of the surrounding environment, but also accurately identifies and locates roads, obstacles, pedestrians, and other traffic participants. The processed data is subjected to feature extraction and segmentation by a deep learning model, separating key traffic elements from the background. The feature fusion strategy in this process enables the algorithm to overcome the limitations of single sensor data and achieve higher detection accuracy and robustness. Ultimately, the generated perception results will serve as the basic input features for the subsequent deep reinforcement learning control strategy model. These features include not only dynamic information such as position and speed, but also changes in traffic signs, lane lines, and the surrounding environment, providing the intelligent driving system with more accurate and rich decision-making basis, thereby improving the safety and reliability of the embodied intelligence model.

[0064] S3: Using the BEV Fusion algorithm to generate a high-dimensional Camera BEV tensor, the Alpha HEV Embodied Intelligence Model is trained using a deep reinforcement learning algorithm. This model focuses on four core learning tasks for vehicle energy-efficient driving: adaptive cruise control, lane keeping assist, energy management strategy, and transmission shifting strategy.

[0065] Step S3 specifically includes the following steps:

[0066] S31: Obtained after the BEV Fusion algorithm completes environmental perception Figure 5 The four-dimensional space tensor shown in the figure represents the number of batch samples, the number of feature layers, the length of the feature layer, and the width of the feature layer. At this time, the deep reinforcement learning algorithm is used as the development tool, the intelligent hybrid vehicle is used as the research and development object, and the energy-saving driving of the whole vehicle is used as the learning task to train the "Alpha HEV embodied intelligent model" that integrates AI experienced drivers and AI professional engineers. Among them, energy-saving driving mainly includes 4 core tasks: Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Energy Management Strategy (EMS), Transmission Shifting Strategy (TSS), etc.

[0067] S32: The deep reinforcement learning-based adaptive cruise control training concept uses a hybrid action strategy combined with a proportional-integral-derivative (PID)-based controller. Speed ​​deviation is weighted to ensure that the deep reinforcement learning agent maintains a stable following speed. If speed loss occurs, especially when the deviation exceeds 1m / s, the PID-based controller quickly stabilizes the actual speed.

[0068] R ACC =-1×[abs(A ACC (t)-A Hybrid (t-1))+abs(VehSpd(t)-TgtSpd(t))]

[0069] Among them, A ACC is the action output by reinforcement learning, A Hybrid is the hybrid action that actually controls the vehicle, VehSpd is the vehicle speed, and TgtSpd is the target speed.

[0070] S33: The deep reinforcement learning-based lane keeping assist training concept uses a desired steering angle based on angular deviation and lateral deviation, where the output action directly controls the steering wheel angle. Angular deviation refers to the difference between the vehicle's heading and the direction of the road ahead at a short distance, while lateral deviation is converted to an angle value based on the same longitudinal distance and lateral offset distance. By combining these two correction angles, the goal of reducing both angular deviation and lateral deviation is achieved. In addition, based on the reference angle, an additional optimization term for lateral deviation is added, aiming to stay as close to the center of the lane as possible while maintaining the correct driving direction;

[0071] R LKA =-1×[abs(A LKA (t)-A PreLka (t))+abs(LatOff)]

[0072] Among them, A LKA is the steering action actually output by reinforcement learning, A PreLka It is a steering action predicted based on the characteristics of the road ahead, and LatOff is a lateral offset.

[0073] S34: The energy management strategy training concept based on deep reinforcement learning is committed to ensuring that the engine torque is maintained in the efficient fuel economy zone by controlling the throttle opening and keeping the operating point as close as possible to the pre-calibrated optimal operating line. At the same time, the energy management strategy must maintain the battery state of charge within a stable and reliable range, especially between 40% and 80%, which helps to significantly extend the battery life while actively preventing phenomena such as overcharging and over-discharging. Therefore, under the influence of rewards, when hybrid power distribution is required, the desired engine operating area will be concentrated on the efficient side of the optimal operating curve;

[0074]

[0075] Where BSFC is brake specific fuel consumption, T eng is the engine torque, T maxeng and T mineng are the maximum and minimum engine torque, T OOL is the calibrated optimal operating line of the engine, and ε is a minimum constant that prevents division by zero.

[0076] S35: The concept of training a transmission shift strategy based on deep reinforcement learning is straightforward, involving continuous adjustment of the gear ratio to maintain the power source's operating speed at approximately 2500 rpm. It's important to note that previous research has concluded that deep reinforcement learning is not suitable for learning intermittent discrete strategies, such as shift strategies for step-variable transmissions and mode switching strategies for multi-mode hybrid powertrains, due to the inherent instability of its exploration-exploitation characteristics.

[0077] R TSS = -1×[abs(MotSpd(t)-MotSpd target )]

[0078] Among them, MotSpd is the real-time motor speed, MotSpd target is the target motor speed, which can be defined as 2500 rpm based on the engine efficiency characteristic map and the motor efficiency characteristic map.

[0079] S4: After completing offline training, the applicability and reliability of the Alpha HEV Embodied Intelligence Model are directly verified in test scenarios while maintaining the online updates of the critic network. A guardian network is created to navigate unfamiliar environments and its ability to discern real-time states and assess reliability is tested. Furthermore, after reconstructing the scene in the cloud, imitation learning is used to rapidly fit control commands to enhance training in new scenarios.

[0080] Step S4 specifically includes the following steps:

[0081] S41: After offline training, we obtained the "Alpha HEV Embodied Intelligence Model," the optimal control policy fitted by the actor network's neural network parameters. All policy models were directly loaded into real-world test scenarios. Under a low learning rate, the deep reinforcement learning agent was constantly in update mode. This not only validated the actor network's applicability and generalization capabilities in a new and unfamiliar test environment, but also evaluated the effectiveness of the critic network's continuous evaluation and update of the actor network in an online environment.

[0082] S42: For the aforementioned offline training environment, a guardian network for unfamiliar environment recognition and identification was created and trained. In this new test environment, the main test is the guardian network's ability to discriminate the real-time state tensor of the deviation and historical sample space range, as well as the evaluation results of membership and reliability. Once the evaluation result is lower than the default reliability threshold, the energy-saving driving strategy of the deep reinforcement learning-based intelligent hybrid vehicle will gradually become ineffective, and will be replaced by a "bottom-line" control strategy with safety control. In addition, for new and unfamiliar state characteristics, imitation learning is first used to fit reliable control commands as quickly as possible, and then the training of new scenarios is enhanced by cloud-based reconstruction of unfamiliar scenarios.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A deep reinforcement learning-based embodied intelligence method for hybrid electric vehicles, characterized by: The method specifically comprises the following steps: S1: Customize offline training and modeling online testing solutions; Real-world road trajectory data is collected using a vehicle equipped with a global positioning system, an inertial measurement unit, and a monocular camera. Then, a waypoint file in XDOR format is generated to create a custom map, which serves as a realistic test scenario for the digital twin. At the same time, an 80% real-world slope is used in the official Curved Road map in MATLAB as a training scenario. S2: Build a real-time driving environment BEV perception solution; Offline training scenarios and custom test maps are loaded into the autonomous driving simulation software, and cameras are configured based on the nuScenes dataset. After completing the scene, vehicle, and sensor modeling, key traffic elements are segmented and detected using the BEV Fusion algorithm to improve environmental perception and detection accuracy. The perception results provide the foundational features for deep reinforcement learning-based embodied intelligence. S3: Training the Alpha HEV Embodied Intelligence Model; The high-dimensional Camera BEV tensor is obtained through the BEV Fusion algorithm, and the "AlphaHEV Embodied Intelligence Model" is trained using a deep reinforcement learning algorithm. This model focuses on four core learning tasks for vehicle energy-saving driving, including adaptive cruise control, lane keeping assist, energy management strategy, and transmission shift strategy. S4: Verify the "Alpha HEV Embodied Intelligence Model"; After completing offline training, the applicability and reliability of the "Alpha HEV Embodied Intelligence Model" are directly verified in the test scenario, while maintaining the online update effect of the critic network; by creating a guardian network to cope with unfamiliar environments, and testing its real-time status judgment and reliability assessment; at the same time, after scene reconstruction in the cloud, imitation learning is used to quickly fit control commands to enhance training for new scenarios.

2. The deep reinforcement learning-based embodied intelligence method according to claim 1, characterized in that: In step S1, a customized offline training and modeling online testing plan is provided, specifically including the following steps: S11: Using a vehicle equipped with a global positioning system, an inertial measurement unit, and a monocular camera, we collect real-world trajectories, i.e., the latitude and longitude, altitude, and yaw angle 𝜓, pitch angle 𝜃, and roll angle 𝜙 of the target path during driving. S12: Based on the measured target path data, after data processing and coordinate system conversion, the actual data in meters is obtained; the slope data originally required is replaced by the pitch angle trajectory data, and the waypoint file in XDOR format is generated using MATLAB; S13: Using RoadRunner modeling software, we fully added additional lanes, defined road surface types, set road boundaries, and drew lane lines. Next, we used Unreal Engine to draw the terrain and other landmarks. After packaging, we loaded this custom map into the autonomous driving simulation software CARLA, which served as a test scenario for deep reinforcement learning-based embodied intelligence solutions for hybrid vehicles. Furthermore, we loaded an 80% realistic slope onto the path trajectory of the Curved Road, the official MALAB map, and used this as a training scenario.

3. The deep reinforcement learning-based embodied intelligence method according to claim 1, characterized in that: In step S2, a real-time driving environment BEV perception solution is constructed, which specifically includes the following steps: S21: In CARLA, not only the custom map model from the real world is loaded, but also the camera installation position is arranged according to the sensor acquisition device specifications of the nuScenes dataset and the internal and external parameters of the monocular RGB camera in the dataset. S22: After completing the modeling and loading of the scene, vehicle, and sensors, the BEV Fusion algorithm is used to segment and detect key traffic elements in the current driving scenario. The BEV Fusion algorithm maps data from different sensors to a bird's-eye view. The processed data is then subjected to feature extraction and segmentation by a deep learning model to separate key traffic elements from the background. Ultimately, the generated perception results serve as the basic input features for the subsequent deep reinforcement learning-based control strategy model.

4. The deep reinforcement learning-based embodied intelligence method according to claim 1, characterized in that: In step S3, training the "Alpha HEV Embodied Intelligence Model" specifically includes the following steps: S31: After the BEV Fusion algorithm completes environmental perception, it generates a four-dimensional spatial tensor, where each dimension represents the number of batch samples, the number of feature layers, the length of a feature layer, and the width of a feature layer. Using deep reinforcement learning algorithms as development tools, intelligent hybrid vehicles as research targets, and vehicle-wide energy-saving driving as learning tasks, the "Alpha HEV Embodied Intelligence Model" is trained, integrating an experienced AI driver with a professional AI engineer. This model includes four core tasks: Adaptive Cruise Control (ACC), Lane Keeping Assist (LKA), Energy Management System (EMS), and Transmission Shift Strategy (TSS). S32: A deep reinforcement learning-based adaptive cruise control training concept uses a hybrid action strategy combined with a PID-based controller. Speed ​​deviation is weighted to enable the deep reinforcement learning agent to maintain a stable following speed. If speed loss occurs, the PID-based controller quickly stabilizes the actual speed when the deviation exceeds 1 m / s. in, R ACC is a bonus for adaptive cruise control, A ACC is the action output by reinforcement learning, A Hybrid It is the hybrid action that actually controls the vehicle, VehSpd is the vehicle speed, TgtSpd is the target speed; S33: The deep reinforcement learning-based lane keeping assist training concept uses a desired steering angle based on angular deviation and lateral deviation, where the output action directly controls the steering wheel angle. Angular deviation refers to the difference between the vehicle's heading and the direction of the road ahead over a short distance, while lateral deviation is converted to an angle value based on the same longitudinal distance and lateral offset distance. By combining these two correction angles, the goal of simultaneously reducing angular deviation and lateral deviation is achieved. Furthermore, in addition to adhering to the reference angle, an additional optimization term for lateral deviation is added, aiming to stay close to the center of the lane while maintaining the correct driving direction. in, R LKA is the reward for lane keeping assist, A LKA is the steering action actually output by reinforcement learning, A PreLka It is a steering action predicted based on the characteristics of the road ahead. LatOff is the lateral offset; S34: The deep reinforcement learning-based energy management strategy training concept is focused on ensuring that the engine torque is maintained within the efficient fuel economy zone by controlling the throttle opening and that the operating point reaches a pre-calibrated optimal operating line. Simultaneously, the energy management strategy must maintain the battery state of charge within a stable and reliable range while actively preventing overcharging and over-discharging. Therefore, under the influence of rewards, when hybrid power distribution is required, the desired engine operating area will be concentrated on the efficient side of the optimal operating curve. in, R EMS is the reward value of the energy management strategy, BSFC is the fuel consumption rate, T eng is the engine torque, T maxeng and T mineng are the maximum and minimum engine torques, T OOL It is the calibrated optimal operating line of the engine. is a minimum constant that prevents division by zero; S35: The concept of training the transmission shift strategy based on deep reinforcement learning is very intuitive and involves continuous adjustment of the gear ratio to keep the power source operating at 2500 rpm; in, R TSS is the reward for the transmission shift strategy, MotSpd is the real-time motor speed, MotSpd target is the target motor speed.

5. The deep reinforcement learning embodied intelligence method according to claim 1, characterized in that: In step S4, the "Alpha HEV Embodied Intelligence Model" is verified, which specifically includes the following steps: S41: After offline training, the "Alpha HEV embodied intelligence model" is obtained, which is the optimal control policy fitted by the neural network parameters of the actor network. All policy models are directly loaded into a real-world test scenario. Under the condition of a small learning rate, the deep reinforcement learning agent is always in update mode. S42: For offline training environments, a guardian network is created and trained for unfamiliar environment recognition and identification. In a new test environment, the guardian network's ability to discriminate real-time state tensors within the scope of deviation and historical sample space, as well as its membership and reliability assessment results, is tested. Once the assessment result falls below the default reliability threshold, the energy-saving driving strategy of the deep reinforcement learning-based intelligent hybrid vehicle will gradually become ineffective, and will be replaced by a "bottom-line" control strategy with safety control. In addition, for new and unfamiliar state characteristics, imitation learning is first used to fit reliable control commands as quickly as possible, followed by cloud-based reconstruction of unfamiliar scenes to enhance training for newly added scenarios.

Images