Intelligent driving scene adaptive teaching method and system based on reinforcement learning

Abstract

The invention provides an intelligent driving scene adaptive teaching method and system based on reinforcement learning, and relates to the technical field of reinforcement learning, and the method comprises the steps: training a plurality of scene recognition models based on historical data, carrying out the scene classification of the historical data, and generating a training sample set with labels; and training by using a reinforcement learning algorithm to obtain an intelligent driving decision model for different scene types. In the actual driving process, the current driving scene is recognized according to the information such as the vehicle position, the vehicle speed and the acceleration which are collected in real time, and the corresponding decision model is selected to output the standard driving behavior sequence. And comparing the actual behavior of the driver with the standard behavior, calculating a deviation value, determining a teaching feedback strategy according to the deviation value, outputting driving guidance information in real time, and using the feedback of the driver to update the decision model. Personalized teaching can be provided according to different driving scenes, the driving skill of a driver is improved, and the driving safety is enhanced.

Description

Technical Field

[0001] The present invention relates to reinforcement learning technology, and in particular to an intelligent driving scenario adaptive teaching method and system based on reinforcement learning. Background Art

[0002] Intelligent driving technology has developed rapidly in recent years, aiming to improve driving safety and efficiency. To achieve this goal, many studies have focused on developing autonomous driving systems that can imitate or even surpass the driving capabilities of human drivers in various scenarios. Among them, reinforcement learning methods have attracted widespread attention due to their ability to optimize decision-making strategies from trial and error learning, and have been applied to the training of intelligent driving decision models. At the same time, considering the complexity and diversity of different driving scenarios, such as highways, urban roads, rural roads, etc., as well as driving under different weather conditions, how to enable intelligent driving systems to adapt to different scenarios and make corresponding decisions has become an important research direction. At present, some studies have tried to combine scene recognition with reinforcement learning to improve the adaptability of intelligent driving systems in different scenarios.

[0003] However, the existing intelligent driving scenario adaptive teaching methods based on reinforcement learning still have some defects and shortcomings: First, traditional scene classification methods usually rely on predefined rules or simple feature extraction, which makes it difficult to accurately identify complex and changing driving scenarios. Second, existing reinforcement learning models are often trained for specific scenarios, lack generalization capabilities, and are difficult to adapt to new or unseen scenarios. Finally, the feedback method of driving guidance information is usually relatively simple, lacking personalized teaching strategies for different drivers, making it difficult to effectively improve the learning efficiency of drivers. Summary of the invention

[0004] The embodiments of the present invention provide a smart driving scenario adaptive teaching method and system based on reinforcement learning, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention, Provides an adaptive teaching method for intelligent driving scenarios based on reinforcement learning, including: Collecting intelligent driving history data, the intelligent driving history data including driver operation data, vehicle status data, and environmental perception data; training multiple scene recognition models according to the intelligent driving history data, each of the scene recognition models corresponding to a preset scene type; performing scene classification on the intelligent driving history data based on each of the scene recognition models to generate a training sample set with scene labels, wherein the scene recognition model is constructed based on a bidirectional long short-term memory network; inputting the training sample set with scene labels into a reinforcement learning algorithm for training to obtain intelligent driving decision models for different scene types; Collecting intelligent driving data in real time through vehicle-mounted sensors, the intelligent driving data including vehicle position information, vehicle speed information, acceleration information, steering wheel angle information, brake pedal position information, and accelerator pedal position information; inputting the intelligent driving data into the scene recognition model, and outputting the scene type of the current driving scene; and selecting a corresponding model from the pre-trained intelligent driving decision models as the current decision model according to the scene type; The intelligent driving data is input into the current decision model to obtain a standard driving behavior sequence; the actual driving behavior sequence of the driver is compared with the standard driving behavior sequence to calculate a driving behavior deviation value; a teaching feedback strategy is determined according to the driving behavior deviation value, and the teaching feedback strategy includes at least one of voice prompts, visual prompts, and tactile feedback; driving guidance information is output in real time according to the teaching feedback strategy, and the driver's feedback result on the driving guidance information is used to update the training parameters of the current decision model.

[0006] Classifying the intelligent driving history data based on each of the scene recognition models to generate a training sample set with scene labels includes: Determine the time window length according to the ratio of the absolute speed of the vehicle to the preset reference speed, and set the time window length with a minimum threshold and a maximum threshold; perform sliding segmentation on the normalized feature vector corresponding to the intelligent driving history data based on the time window length, and calculate the differential feature and statistical feature of the normalized feature vector in each time window, wherein the statistical feature includes a mean feature, a standard deviation feature, a skewness feature, and a kurtosis feature; and combine the differential feature and the statistical feature to form an enhanced feature vector; Inputting the enhanced feature vector into a scene recognition model constructed based on a bidirectional long short-term memory network to obtain a forward hidden layer state and a reverse hidden layer state respectively; fusing the forward hidden layer state and the reverse hidden layer state to obtain a fused hidden layer state; Perform multi-head self-attention calculation on the fused hidden layer state to obtain attention features; combine the fused hidden layer state with the attention features and input them into a classifier to obtain a scene type probability distribution; when the maximum value of the scene type probability distribution is greater than a preset confidence threshold, take the corresponding scene type as the classification result; when the maximum value of the scene type probability distribution is less than a preset confidence threshold, mark the scene type as an unknown type.

[0007] Before inputting the training sample set with scene labels into a reinforcement learning algorithm for training to obtain intelligent driving decision models for different scene types, the method further includes constructing reinforcement learning training samples: Constructing a vehicle state feature vector, an environment state feature vector, and a driving state feature vector based on the intelligent driving history data; fusing the vehicle state feature vector, the environment state feature vector, the driving state feature vector, and the scene type information to obtain a state space representation; Determine the steering action constraint range according to the current vehicle speed and road type, and determine the acceleration action constraint range according to the current vehicle speed and traffic flow information; construct a steering control action space based on the steering action constraint range, and the steering control action space includes a target steering angle, a steering angular rate, and a steering execution duration; construct a longitudinal control action space based on the acceleration action constraint range, and the longitudinal control action space includes an acceleration instruction and a braking instruction; combine the steering control action space with the longitudinal control action space to form a continuous action space; Calculating the minimum obstacle avoidance distance to obtain a collision risk assessment value, calculating the steering change and the acceleration change to obtain a trajectory stability assessment value, calculating the acceleration and the steering angle rate to obtain a ride comfort assessment value, and calculating the speed deviation and the heading angle error to obtain a driving efficiency assessment value; calculating the scene adaptive weight coefficient according to the state space representation; The collision risk assessment value, the trajectory stability assessment value, the ride comfort assessment value, and the driving efficiency assessment value are weighted with the corresponding scene adaptive weight coefficients to obtain a comprehensive reward value; the state space representation, continuous action space, comprehensive reward value, scene type information, and scene adaptive weight coefficient are combined into training samples, and the training samples are classified and stored according to the scene type.

[0008] The training sample set with scene labels is input into the reinforcement learning algorithm for training, and the intelligent driving decision-making models for different scene types are obtained, including: Obtain a scene type code and a scene attribute vector, input the scene type code and the scene attribute vector into a multilayer perceptron for feature extraction to obtain scene features; fuse the scene features with the state space representation to obtain a scene condition embedding vector; construct an action mean network and an action variance network based on the scene condition embedding vector, the action mean network outputs an action mean, and the action variance network outputs an action variance after softplus transformation; construct a Gaussian strategy distribution based on the action mean and the action variance; Input the scenario condition embedding vector into the value network to obtain a state value estimate; calculate the temporal difference error based on the state value estimate; calculate the priority sampling probability based on the temporal difference error and the advantage function; sample training data from the experience pool based on the priority sampling probability; calculate the importance weight ratio of the current strategy distribution to the historical strategy distribution; truncate the advantage function according to the importance weight ratio to obtain the strategy objective function; Calculate the mean square error between the state value estimate and the target value to obtain the value function loss; calculate the information entropy of the policy distribution to obtain the policy entropy loss; perform weighted combination of the policy objective function, value function loss, and policy entropy loss to obtain the overall loss function; calculate the scenario difficulty index, and construct a scenario sampling probability distribution based on the scenario difficulty index; perform multi-scenario cross-training according to the scenario sampling probability distribution; and iteratively optimize the policy network parameters and the value network parameters using an adaptive learning rate, which is dynamically adjusted according to the training rounds.

[0009] Inputting the scene condition embedding vector into the value network to obtain a state value estimate; calculating a temporal difference error based on the state value estimate; and calculating a priority sampling probability based on the temporal difference error and an advantage function include: Construct a dual value network, input the scenario condition embedding vector into the first value network and the second value network respectively, and obtain the first state value estimation value and the second state value estimation value; select the minimum value of the first state value estimation value and the second state value estimation value as the state value estimation result; calculate the discount sum of the n-step cumulative reward and the target state value to obtain the n-step return value; perform weighted combination of the return values ​​of different step lengths to obtain the λ return value; Calculate the difference between the discounted sum of the immediate reward and the target state value and the current state value to obtain a time series difference error; calculate the mean and standard deviation of the time series difference error, and use the mean and standard deviation to normalize the time series difference error to obtain a normalized time series difference error; calculate a generalized advantage estimate based on the normalized time series difference error, and use the weighted sum of the generalized advantage estimate and the time series difference error as a basic priority; perform momentum update on the basic priority and the historical priority to obtain an updated priority; A sampling probability distribution is calculated according to the updated priority; an importance weight is calculated based on the sampling probability distribution; the updated priority is divided by a temperature parameter and indexed to obtain an indexed priority; and the indexed priority is normalized to obtain a final sampling probability.

[0010] Inputting the intelligent driving data into the current decision model to obtain a standard driving behavior sequence; comparing the driver's actual driving behavior sequence with the standard driving behavior sequence to calculate a driving behavior deviation value; and determining a teaching feedback strategy according to the driving behavior deviation value includes: The Euclidean distance between the actual position of the driver and the standard position is calculated to obtain the position deviation value, the absolute difference between the actual heading angle of the driver and the standard heading angle is calculated to obtain the heading deviation value, the Euclidean distance between the actual speed of the driver and the standard speed, and the Euclidean distance between the actual acceleration of the driver and the standard acceleration are calculated respectively, and the speed Euclidean distance and the acceleration Euclidean distance are weighted to obtain the motion state deviation value; the absolute difference between the actual steering angle of the driver and the standard steering angle is calculated to obtain the steering deviation value, and the absolute difference between the actual longitudinal control amount of the driver and the standard longitudinal control amount is calculated to obtain the longitudinal control deviation value; the position deviation value, heading deviation value, motion state deviation value, steering deviation value, and longitudinal control deviation value are weighted to obtain the comprehensive behavior deviation value; Calculate the mean and standard deviation of the comprehensive behavior deviation value; construct a risk assessment interval based on the mean and standard deviation, determine the moment when the comprehensive behavior deviation value is higher than the sum of the mean and standard deviation as high risk, determine the moment when the comprehensive behavior deviation value is between the mean minus the standard deviation and the mean plus the standard deviation as medium risk, and determine the moment when the comprehensive behavior deviation value is lower than the mean minus the standard deviation as low risk; identify the moment when the comprehensive behavior deviation value exceeds a preset proportion of the maximum deviation value as a critical moment; calculate the contribution of the position deviation value, heading deviation value, motion state deviation value, steering deviation value, and longitudinal control deviation value to the comprehensive behavior deviation value, and select the deviation type with the largest contribution as the main deviation cause at the current moment; Generate driving guidance suggestions based on the risk level, critical moments, and major deviation causes; determine feedback timing based on the comprehensive behavioral deviation value and its rate of change, determine feedback intensity based on the risk level, mean, and standard deviation; and output control of the driving guidance suggestions based on the feedback timing and feedback intensity.

[0011] A second aspect of an embodiment of the present invention provides an intelligent driving scenario adaptive teaching system based on reinforcement learning, including: The first unit is used to collect intelligent driving history data, wherein the intelligent driving history data includes driver operation data, vehicle status data, and environmental perception data; train multiple scene recognition models according to the intelligent driving history data, each of which corresponds to a preset scene type; classify the intelligent driving history data based on each scene recognition model to generate a training sample set with scene labels, wherein the scene recognition model is constructed based on a bidirectional long short-term memory network; input the training sample set with scene labels into a reinforcement learning algorithm for training to obtain an intelligent driving decision model for different scene types; The second unit is used to collect intelligent driving data in real time through vehicle-mounted sensors, wherein the intelligent driving data includes vehicle position information, vehicle speed information, acceleration information, steering wheel angle information, brake pedal position information, and accelerator pedal position information; input the intelligent driving data into the scene recognition model, and output the scene type of the current driving scene; and select a corresponding model from the pre-trained intelligent driving decision model as the current decision model according to the scene type; The third unit is used to input the intelligent driving data into the current decision model to obtain a standard driving behavior sequence; compare the driver's actual driving behavior sequence with the standard driving behavior sequence to calculate a driving behavior deviation value; determine a teaching feedback strategy according to the driving behavior deviation value, wherein the teaching feedback strategy includes at least one of voice prompts, visual prompts, and tactile feedback; output driving guidance information in real time according to the teaching feedback strategy, and use the driver's feedback result on the driving guidance information to update the training parameters of the current decision model.

[0012] A third aspect of the embodiments of the present invention An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0013] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0014] The beneficial effects of this application are as follows: 1. Improve the efficiency and pertinence of intelligent driving training: Through scenario-based reinforcement learning methods, special intelligent driving decision-making models are trained for different driving scenarios, avoiding the problems of low training efficiency and poor generalization ability of traditional methods in complex scenarios, making training more efficient and pertinent.

[0015] 2. Improve the driver's driving skills: By comparing the driver's actual driving behavior with the standard driving behavior and providing real-time, personalized driving guidance information, it helps the driver learn and master safer driving skills, effectively improving the driver's driving skills in different scenarios.

[0016] 3. Achieve adaptive learning and continuous optimization: The system can update the parameters of the decision model in real time based on the driver's feedback on the guidance information, achieve adaptive learning and continuous optimization, and thus continuously improve the performance of the intelligent driving system and the driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of an adaptive teaching method for intelligent driving scenarios based on reinforcement learning according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an intelligent driving scenario adaptive teaching system based on reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0020] Figure 1 : is a flow chart of an intelligent driving scenario adaptive teaching method based on reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes: S101. Collecting intelligent driving history data, the intelligent driving history data including driver operation data, vehicle status data, and environmental perception data; training multiple scene recognition models based on the intelligent driving history data, each of the scene recognition models corresponding to a preset scene type; performing scene classification on the intelligent driving history data based on each of the scene recognition models to generate a training sample set with scene labels, wherein the scene recognition model is constructed based on a bidirectional long short-term memory network; inputting the training sample set with scene labels into a reinforcement learning algorithm for training to obtain intelligent driving decision models for different scene types; S102. Collecting intelligent driving data in real time through vehicle-mounted sensors, the intelligent driving data including vehicle position information, vehicle speed information, acceleration information, steering wheel angle information, brake pedal position information, and accelerator pedal position information; inputting the intelligent driving data into the scene recognition model, and outputting the scene type of the current driving scene; and selecting a corresponding model from the pre-trained intelligent driving decision model as the current decision model according to the scene type; S103. Input the intelligent driving data into the current decision model to obtain a standard driving behavior sequence; compare the driver's actual driving behavior sequence with the standard driving behavior sequence to calculate a driving behavior deviation value; determine a teaching feedback strategy based on the driving behavior deviation value, wherein the teaching feedback strategy includes at least one of voice prompts, visual prompts, and tactile feedback; output driving guidance information in real time based on the teaching feedback strategy, and use the driver's feedback results on the driving guidance information to update the training parameters of the current decision model.

[0021] In an optional implementation, performing scene classification on the intelligent driving history data based on each of the scene recognition models to generate a training sample set with scene labels includes: Determine the time window length according to the ratio of the absolute speed of the vehicle to the preset reference speed, and set the time window length with a minimum threshold and a maximum threshold; perform sliding segmentation on the normalized feature vector corresponding to the intelligent driving history data based on the time window length, and calculate the differential feature and statistical feature of the normalized feature vector in each time window, wherein the statistical feature includes a mean feature, a standard deviation feature, a skewness feature, and a kurtosis feature; and combine the differential feature and the statistical feature to form an enhanced feature vector; Inputting the enhanced feature vector into a scene recognition model constructed based on a bidirectional long short-term memory network to obtain a forward hidden layer state and a reverse hidden layer state respectively; fusing the forward hidden layer state and the reverse hidden layer state to obtain a fused hidden layer state; Perform multi-head self-attention calculation on the fused hidden layer state to obtain attention features; combine the fused hidden layer state with the attention features and input them into a classifier to obtain a scene type probability distribution; when the maximum value of the scene type probability distribution is greater than a preset confidence threshold, the corresponding scene type is used as a classification result, and when the maximum value of the scene type probability distribution is less than a preset confidence threshold, the scene type is marked as an unknown type. An intelligent driving scene recognition method is used to perform scene classification on intelligent driving historical data and generate a training sample set with scene labels.

[0022] First, determine the length of the time window. Set a reference speed value in advance. Obtain the absolute speed of the vehicle at each moment. Calculate the ratio of the vehicle's absolute speed to the preset reference speed. Determine the length of the time window based on the ratio. For example, set the reference speed to 60 km / h. When the vehicle speed is 120 km / h, the speed ratio is 2, and the time window length is set to 2 seconds. When the vehicle speed is 30 km / h, the speed ratio is 0.5, and the time window length is set to 0.5 seconds. In order to avoid the time window being too long or too short, set the minimum threshold and maximum threshold of the time window length, for example, the minimum threshold is 0.1 seconds and the maximum threshold is 5 seconds. If the calculated time window length is less than the minimum threshold, the minimum threshold is used as the time window length; if the calculated time window length is greater than the maximum threshold, the maximum threshold is used as the time window length.

[0023] Next, the intelligent driving historical data is segmented. The intelligent driving historical data contains data collected by multiple sensors, such as vehicle speed, acceleration, steering wheel angle, GPS location, etc. First, these data are normalized to convert data of different dimensions into the same numerical range. Then, according to the time window length calculated previously, the normalized feature vector is slidingly segmented. For example, assuming that the time window length is 1 second and the sampling frequency is 10 Hz, each time window contains 10 data points. Starting from the first data point, take 10 consecutive data points as a time window, and then starting from the second data point, take 10 consecutive data points as the next time window, and so on.

[0024] Then, the features within each time window are calculated. For the normalized feature vector within each time window, the differential features and statistical features are calculated. The differential features refer to the difference between adjacent data points. The statistical features include the mean feature, the standard deviation feature, the skewness feature, and the kurtosis feature. The mean feature describes the average level of the data, the standard deviation feature describes the degree of dispersion of the data, the skewness feature describes the symmetry of the data distribution, and the kurtosis feature describes the sharpness of the data distribution. For example, for a time window containing 10 data points, 9 differential features and 4 statistical features can be calculated.

[0025] After that, the differential features and statistical features are combined to form an enhanced feature vector. The calculated differential features and statistical features are concatenated together to form a new feature vector, called an enhanced feature vector. For example, assuming there are 9 differential features and 4 statistical features, the enhanced feature vector contains 13 features.

[0026] Next, the enhanced feature vector is input into the scene recognition model. The scene recognition model adopts a bidirectional long short-term memory network (BiLSTM) structure. The enhanced feature vector is input into the BiLSTM network to obtain the forward hidden layer state and the reverse hidden layer state respectively. Then the forward hidden layer state and the reverse hidden layer state are fused, for example, the states at corresponding moments are concatenated or added to obtain the fused hidden layer state.

[0027] After that, multi-head self-attention is calculated on the fused hidden state. Multi-head self-attention is calculated on the fused hidden state to obtain the attention feature. The multi-head self-attention mechanism can capture the correlation between different time windows. Then, the fused hidden state is combined with the attention feature. The fused hidden state is combined with the attention feature, for example, concatenated, and the combined feature is input into the classifier.

[0028] Next, a classifier is used to classify the scene type. The classifier can be, for example, a softmax classifier. The classifier outputs a probability distribution of the scene type.

[0029] Finally, determine the scene type. When the maximum value of the scene type probability distribution is greater than the preset confidence threshold, the corresponding scene type is used as the classification result. For example, suppose there are three scene types: highways, urban roads, and rural roads. If the probability of highways is 0.8, the probability of urban roads is 0.1, the probability of rural roads is 0.1, and the confidence threshold is set to 0.7, the scene is identified as a highway. When the maximum value of the scene type probability distribution is less than the preset confidence threshold, the scene type is marked as an unknown type. For example, if the probability of highways is 0.5, the probability of urban roads is 0.3, the probability of rural roads is 0.2, and the confidence threshold is set to 0.7, the scene is marked as an unknown type.

[0030] Beneficial effects: 1. Improve scene recognition accuracy: By combining differential features and statistical features, and using a bidirectional long short-term memory network and a multi-head self-attention mechanism, scene features can be better captured and the accuracy of scene recognition can be improved.

[0031] 2. Adapt to different vehicle speeds: By dynamically adjusting the time window length according to the vehicle speed, it can better adapt to scene recognition at different vehicle speeds and avoid information loss or redundancy caused by too long or too short a time window.

[0032] 3. Improve model robustness: By setting the confidence threshold, you can avoid the model giving incorrect classification results when the confidence level is low, thereby improving the robustness of the model.

[0033] In an optional implementation, before inputting the training sample set with scene labels into a reinforcement learning algorithm for training to obtain intelligent driving decision models for different scene types, the method further includes constructing reinforcement learning training samples: Constructing a vehicle state feature vector, an environment state feature vector, and a driving state feature vector based on the intelligent driving history data; fusing the vehicle state feature vector, the environment state feature vector, the driving state feature vector, and the scene type information to obtain a state space representation; Determine the steering action constraint range according to the current vehicle speed and road type, and determine the acceleration action constraint range according to the current vehicle speed and traffic flow information; construct a steering control action space based on the steering action constraint range, and the steering control action space includes a target steering angle, a steering angular rate, and a steering execution duration; construct a longitudinal control action space based on the acceleration action constraint range, and the longitudinal control action space includes an acceleration instruction and a braking instruction; combine the steering control action space with the longitudinal control action space to form a continuous action space; Calculating the minimum obstacle avoidance distance to obtain a collision risk assessment value, calculating the steering change and the acceleration change to obtain a trajectory stability assessment value, calculating the acceleration and the steering angle rate to obtain a ride comfort assessment value, and calculating the speed deviation and the heading angle error to obtain a driving efficiency assessment value; calculating the scene adaptive weight coefficient according to the state space representation; The collision risk assessment value, the trajectory stability assessment value, the ride comfort assessment value, and the driving efficiency assessment value are weighted with the corresponding scene adaptive weight coefficients to obtain a comprehensive reward value; the state space representation, continuous action space, comprehensive reward value, scene type information, and scene adaptive weight coefficient are combined into training samples, and the training samples are classified and stored according to the scene type.

[0034] The intelligent driving decision model training method aims to improve the decision-making ability of the intelligent driving system in different scenarios. The method is based on the reinforcement learning algorithm and trains the model by constructing a high-quality, scenario-labeled training sample set.

[0035] First, collect a large amount of intelligent driving history data. This data includes the vehicle's own state information, the surrounding environment's state information, and the driver's operation information. Vehicle state information includes: vehicle speed, acceleration, steering wheel angle, yaw rate, etc. Environmental state information includes: road type, lane line information, traffic light status, location and speed of surrounding vehicles, etc. Driver operation information includes: throttle opening, brake pedal strength, steering wheel rotation angle, etc.

[0036] Next, the collected historical data is processed to construct a feature vector. The vehicle state information is converted into a vehicle state feature vector, for example, the values ​​of vehicle speed, acceleration, steering wheel angle, yaw rate, etc. are combined into a vector. Similarly, the environmental state information is converted into an environmental state feature vector, for example, the road type is represented by a unique hot encoding, the lane line information is represented by curve parameters, the traffic light state is represented by three states of red, green and yellow, and the position and speed of the surrounding vehicles are represented by the coordinates and speed relative to the vehicle. The driver operation information is converted into a driving state feature vector, for example, the values ​​of throttle opening, brake pedal force, steering wheel rotation angle, etc. are combined into a vector. Then, the vehicle state feature vector, the environmental state feature vector, the driving state feature vector and the scene type information (for example, highway, urban road, rural road, etc.) are fused and processed to finally obtain a state space representation. For example, these feature vectors can be spliced ​​together to form a larger vector as a state space representation.

[0037] In order for the reinforcement learning algorithm to effectively explore the action space, it is necessary to determine the action constraint range based on the vehicle's current state and environmental information. For example, the steering action constraint range is determined based on the vehicle's current speed and road type: on highways, the maximum steering angle limit is small; when driving at low speeds and on roads with large curves, a larger steering angle is allowed. The acceleration action constraint range is determined based on the vehicle's current speed and traffic flow information: on congested roads, the maximum acceleration and maximum deceleration limits are large; on roads with less traffic, larger acceleration and deceleration are allowed.

[0038] Based on the determined action constraint range, an action space is constructed. The steering control action space includes the target steering angle, steering angular rate, and steering execution time. For example, the target steering angle ranges from -30 degrees to 30 degrees, the steering angular rate ranges from -10 degrees per second to 10 degrees per second, and the steering execution time ranges from 0.1 seconds to 1 second. The longitudinal control action space includes acceleration instructions and braking instructions. For example, the acceleration instruction ranges from -5 meters per second squared to 5 meters per second squared, and the braking instruction ranges from 0 to 1. The steering control action space is combined with the longitudinal control action space to form a continuous action space.

[0039] In order to guide the reinforcement learning algorithm to learn a better driving strategy, a reasonable reward function needs to be designed. First, calculate various evaluation indicators. For example, the collision risk assessment value is obtained by calculating the minimum distance between the vehicle and the surrounding obstacles. The smaller the distance, the higher the collision risk; the trajectory stability assessment value is obtained by calculating the steering change and the acceleration change. The larger the change, the more unstable the trajectory; the ride comfort assessment value is obtained by calculating the acceleration and steering angular rate. The larger the value, the worse the ride comfort; the driving efficiency assessment value is obtained by calculating the speed deviation and the heading angle error. The larger the deviation, the lower the driving efficiency. Then, the scene adaptive weight coefficient is calculated according to the state space representation. For example, on the highway, the weight coefficient of driving efficiency is larger; on urban roads, the weight coefficients of collision risk and ride comfort are larger. Finally, the collision risk assessment value, trajectory stability assessment value, ride comfort assessment value, and driving efficiency assessment value are weighted with the corresponding scene adaptive weight coefficients to obtain a comprehensive reward value.

[0040] The state space representation, continuous action space, comprehensive reward value, scene type information, and scene adaptive weight coefficient are combined into training samples. For example, a training sample can be represented as: [state space representation, continuous action space, comprehensive reward value, scene type information, scene adaptive weight coefficient]. Then, the training samples are classified and stored according to the scene type. For example, the training samples of the highway scene are stored in one file, and the training samples of the urban road scene are stored in another file.

[0041] Finally, the training sample set with scene labels is input into the reinforcement learning algorithm for training to obtain an intelligent driving decision-making model for different scene types.

[0042] For example, in an urban road scenario, the vehicle is traveling at a speed of 30 kilometers per hour and a pedestrian appears in front. The state space at this time is expressed as [vehicle speed: 30 kilometers per hour, pedestrian distance: 10 meters, road type: urban road], and the action space is [target steering angle: -10 degrees, steering angle rate: -5 degrees per second, steering execution time: 0.5 seconds, acceleration command: -2 meters per second squared, braking command: 0.8], the comprehensive reward value is -5, the scene type information is "urban road", and the scene adaptive weight coefficient is [collision risk: 0.8, trajectory stability: 0.1, ride comfort: 0.05, driving efficiency: 0.05].

[0043] The beneficial effects of this method can be summarized in the following three aspects: 1. Improve the scenario adaptability of the model: By constructing a scene-labeled training sample set and a scene-adaptive weight coefficient, the model can learn driving strategies for different scenarios, thereby improving the adaptability of the model in different scenarios.

[0044] 2. Improve the decision-making ability of the model: By using reinforcement learning algorithms, the model can learn better driving strategies, thereby improving the decision-making ability of the model.

[0045] 3. Improve training efficiency: By constructing a high-quality training sample set, the training efficiency of the reinforcement learning algorithm can be improved, thereby shortening the training time of the model.

[0046] In an optional implementation, the training sample set with scene labels is input into a reinforcement learning algorithm for training, and the intelligent driving decision model for different scene types is obtained, including: Obtain a scene type code and a scene attribute vector, input the scene type code and the scene attribute vector into a multilayer perceptron for feature extraction to obtain scene features; fuse the scene features with the state space representation to obtain a scene condition embedding vector; construct an action mean network and an action variance network based on the scene condition embedding vector, the action mean network outputs an action mean, and the action variance network outputs an action variance after softplus transformation; construct a Gaussian strategy distribution based on the action mean and the action variance; Input the scenario condition embedding vector into the value network to obtain a state value estimate; calculate the temporal difference error based on the state value estimate; calculate the priority sampling probability based on the temporal difference error and the advantage function; sample training data from the experience pool based on the priority sampling probability; calculate the importance weight ratio of the current strategy distribution to the historical strategy distribution; truncate the advantage function according to the importance weight ratio to obtain the strategy objective function; Calculate the mean square error between the state value estimate and the target value to obtain the value function loss; calculate the information entropy of the policy distribution to obtain the policy entropy loss; perform weighted combination of the policy objective function, value function loss, and policy entropy loss to obtain the overall loss function; calculate the scenario difficulty index, and construct a scenario sampling probability distribution based on the scenario difficulty index; perform multi-scenario cross-training according to the scenario sampling probability distribution; and iteratively optimize the policy network parameters and the value network parameters using an adaptive learning rate, which is dynamically adjusted according to the training rounds.

[0047] A scene-aware reinforcement learning intelligent driving decision-making method aims to improve the decision-making ability and generalization performance of intelligent driving models in different scenarios. The core of this method is to integrate scene information into the reinforcement learning framework, guide the learning of the policy network and value network through the scene condition embedding vector, and conduct multi-scenario cross-training in combination with scene difficulty.

[0048] First, obtain the scene type code and scene attribute vector. The scene type code is the result of discretizing the scene, such as highway, urban road, rural road, etc. The scene attribute vector describes the specific characteristics of the scene, such as traffic flow, road width, weather conditions, etc. Assuming that the scene type code is "highway" (represented by the number 1), the scene attribute vector is [traffic flow: high, road width: wide, weather conditions: clear], which can be quantized as [10, 5, 1].

[0049] Then, the scene type code and scene attribute vector are input into the multilayer perceptron for feature extraction to obtain scene features. The multilayer perceptron consists of multiple fully connected layers, which can map the input scene information to a high-dimensional feature space. For example, the scene type code "1" and the scene attribute vector [10, 5, 1] ​​are input into a perceptron containing two fully connected layers. After being processed by a nonlinear activation function, a scene feature vector with a dimension of 64 can be obtained.

[0050] Next, the scene features are fused with the state space representation to obtain the scene condition embedding vector. The state space representation describes the current state of the intelligent driving vehicle, such as speed, position, acceleration, etc. Assuming the state space representation is [speed: 60km / h, position: x=10, y=20, acceleration: 0], it can be quantized to [60, 10, 20, 0]. This vector is concatenated with the 64-dimensional scene feature vector obtained above to obtain a higher-dimensional scene condition embedding vector.

[0051] Based on the scene condition embedding vector, the action mean network and the action variance network are constructed. The action mean network outputs the action mean, and the action variance network outputs the action variance after softplus transformation. Both networks take the scene condition embedding vector as input. For example, by inputting the scene condition embedding vector into the action mean network, an action mean vector can be obtained, such as [steering angle: 0.1, acceleration: 0.5]; by inputting the scene condition embedding vector into the action variance network, after softplus transformation, an action variance vector can be obtained, such as [steering angle variance: 0.01, acceleration variance: 0.05].

[0052] A Gaussian policy distribution is constructed based on the action mean and action variance. This distribution is used to generate the actions of intelligent driving vehicles, such as steering angle and acceleration. Using the action mean and action variance obtained above, a Gaussian policy distribution can be constructed.

[0053] At the same time, the scene condition embedding vector is input into the value network to obtain the state value estimate. The value network is used to evaluate the value of the current state, such as the possible rewards in the future. For example, by inputting the scene condition embedding vector into the value network, a state value estimate can be obtained, such as 10.

[0054] The temporal difference error is calculated based on the state value estimate. The temporal difference error is used to measure the pros and cons of the current strategy. The priority sampling probability is calculated based on the temporal difference error and the advantage function. The priority sampling probability is used to sample training data from the experience pool. The experience pool stores the historical experience data of the intelligent driving vehicle, such as state, action, reward, etc.

[0055] Sample training data from the experience pool based on the priority sampling probability. Calculate the importance weight ratio of the current policy distribution to the historical policy distribution. Truncate the advantage function according to the importance weight ratio to obtain the policy objective function. Calculate the mean square error between the state value estimate and the target value to obtain the value function loss. Calculate the information entropy of the policy distribution to obtain the policy entropy loss.

[0056] The strategy objective function, value function loss, and strategy entropy loss are weighted together to obtain the overall loss function. Calculate the scene difficulty index. The scene difficulty index can be calculated based on the scene attribute vector. For example, a scene difficulty value can be calculated based on factors such as traffic flow and road width.

[0057] Construct a scenario sampling probability distribution based on the scenario difficulty index. The scenario sampling probability distribution is used to select training scenarios in multi-scenario cross-training. Perform multi-scenario cross-training based on the scenario sampling probability distribution. Use an adaptive learning rate to iteratively optimize the policy network parameters and value network parameters. The adaptive learning rate is dynamically adjusted according to the training round.

[0058] The beneficial effects of this method are mainly reflected in the following three aspects: First, improve the safety of intelligent driving decisions. By incorporating scenario information into the reinforcement learning framework, the model can better understand the risk factors in different scenarios and make safer decisions.

[0059] Second, improve the efficiency of intelligent driving decision-making. Through technologies such as priority sampling and adaptive learning rate, the model training process can be accelerated and decision-making efficiency can be improved.

[0060] Third, improve the generalization performance of the intelligent driving model. Through multi-scenario cross-training and the guidance of scenario difficulty indicators, the model can better adapt to different scenarios and improve generalization performance.

[0061] In an optional implementation, the scene condition embedding vector is input into a value network to obtain a state value estimate; a temporal difference error is calculated based on the state value estimate; and a priority sampling probability is calculated based on the temporal difference error and an advantage function, including: Construct a dual value network, input the scenario condition embedding vector into the first value network and the second value network respectively, and obtain the first state value estimation value and the second state value estimation value; select the minimum value of the first state value estimation value and the second state value estimation value as the state value estimation result; calculate the discount sum of the n-step cumulative reward and the target state value to obtain the n-step return value; perform weighted combination of the return values ​​of different step lengths to obtain the λ return value; Calculate the difference between the discounted sum of the immediate reward and the target state value and the current state value to obtain a time series difference error; calculate the mean and standard deviation of the time series difference error, and use the mean and standard deviation to normalize the time series difference error to obtain a normalized time series difference error; calculate a generalized advantage estimate based on the normalized time series difference error, and use the weighted sum of the generalized advantage estimate and the time series difference error as a basic priority; perform momentum update on the basic priority and the historical priority to obtain an updated priority; A sampling probability distribution is calculated according to the updated priority; an importance weight is calculated based on the sampling probability distribution; the updated priority is divided by a temperature parameter and indexed to obtain an indexed priority; and the indexed priority is normalized to obtain a final sampling probability.

[0062] A reinforcement learning priority sampling method based on value network and advantage function improves sampling efficiency and algorithm stability through more accurate state value estimation and more stable advantage function calculation.

[0063] First, construct scene features. Convert the state information of the environment in which the reinforcement learning agent is located into a vector representation, which is called a scene condition embedding vector. For example, in a simple maze game, information such as the agent's position, target position, and obstacle position can all be encoded into the scene condition embedding vector. Assuming the maze size is 5x5, the agent is located at (2,3), and the target is located at (4,5), a 25-dimensional vector can be constructed, where the values ​​of the positions (2,3) and (4,5) are 1, the obstacle position is -1, and the remaining positions are 0.

[0064] Next, perform state value estimation. Input the scene condition embedding vector into two independent value networks to obtain two state value estimates. Select the smaller of the two estimates as the final state value estimation result. This dual value network structure can effectively reduce the risk of overestimation and improve the stability of learning. Assuming that the state value estimate output by the first value network is 0.8 and the state value estimate output by the second value network is 0.7, the final state value estimate result is 0.7.

[0065] Then, calculate the multi-step reward value. Calculate the cumulative reward for n steps, for example, calculate the sum of the rewards for the next 3 steps. Discount the target state value (that is, the state value after n steps), and the discount factor is usually set to a value close to 1 such as 0.99. Add the n-step cumulative reward and the discounted target state value to get the n-step reward value. In order to more fully utilize the information of different step lengths, the reward values ​​of different step lengths are weighted and combined to obtain the λ reward value, where λ is a parameter between 0 and 1 that controls the weights of the reward values ​​of different step lengths. Assuming that the 1-step reward value is 0.5, the 2-step reward value is 0.8, the 3-step reward value is 1.0, and the λ value is 0.8, then the λ reward value can be calculated .

[0066] After that, calculate the TD error. Add the immediate reward and the discounted target state value, and then subtract the current state value to get the TD error. In order to improve the stability of learning, the TD error is normalized. Calculate the mean and standard deviation of the TD errors of all samples, then subtract the mean from the TD error of each sample and divide it by the standard deviation to get the normalized TD error. Assuming that the TD error of a sample is 0.2, the mean of the TD errors of all samples is 0.1, and the standard deviation is 0.05, then the normalized TD error is (0.2- 0.1) / 0.05 = 2.

[0067] Next, calculate the generalized advantage estimate. Based on the normalized time difference error, calculate the generalized advantage estimate. The generalized advantage estimate and the time difference error are weighted to obtain the basic priority. In order to make the learning process smoother, the basic priority and the historical priority are momentum updated to obtain the updated priority. Assuming that the basic priority of a sample is 0.9, the historical priority is 0.8, and the momentum parameter is 0.9, the updated priority is .

[0068] Finally, calculate the sampling probability. Divide the updated priority by a temperature parameter and exponentially obtain the exponential priority. Normalize the exponential priority to obtain the final sampling probability. Assuming that the updated priority of a sample is 0.9 and the temperature parameter is 1, the exponential priority is exp(0.9 / 1) = 2.46. Assuming that the sum of the exponential priorities of all samples is 10, the final sampling probability of the sample is 2.46 / 10 = 0.246.

[0069] Beneficial effects: 1. Improve sampling efficiency: Through priority sampling, more computing resources are concentrated on more valuable experience samples, thereby improving sampling efficiency and accelerating learning.

[0070] 2. Enhanced algorithm stability: The normalization of the dual value network and temporal difference errors can effectively reduce the risk and variance of overestimation, thereby enhancing the stability of the algorithm.

[0071] 3. Improve learning effect: Through more accurate state value estimation and more stable advantage function calculation, the agent can be more effectively guided to learn the optimal strategy, thereby improving the learning effect.

[0072] In an optional implementation, the intelligent driving data is input into the current decision model to obtain a standard driving behavior sequence; the actual driving behavior sequence of the driver is compared with the standard driving behavior sequence to calculate a driving behavior deviation value; and determining a teaching feedback strategy according to the driving behavior deviation value includes: The Euclidean distance between the actual position of the driver and the standard position is calculated to obtain the position deviation value, the absolute difference between the actual heading angle of the driver and the standard heading angle is calculated to obtain the heading deviation value, the Euclidean distance between the actual speed of the driver and the standard speed, and the Euclidean distance between the actual acceleration of the driver and the standard acceleration are calculated respectively, and the speed Euclidean distance and the acceleration Euclidean distance are weighted to obtain the motion state deviation value; the absolute difference between the actual steering angle of the driver and the standard steering angle is calculated to obtain the steering deviation value, and the absolute difference between the actual longitudinal control amount of the driver and the standard longitudinal control amount is calculated to obtain the longitudinal control deviation value; the position deviation value, heading deviation value, motion state deviation value, steering deviation value, and longitudinal control deviation value are weighted to obtain the comprehensive behavior deviation value; Calculate the mean and standard deviation of the comprehensive behavior deviation value; construct a risk assessment interval based on the mean and standard deviation, determine the moment when the comprehensive behavior deviation value is higher than the sum of the mean and standard deviation as high risk, determine the moment when the comprehensive behavior deviation value is between the mean minus the standard deviation and the mean plus the standard deviation as medium risk, and determine the moment when the comprehensive behavior deviation value is lower than the mean minus the standard deviation as low risk; identify the moment when the comprehensive behavior deviation value exceeds a preset proportion of the maximum deviation value as a critical moment; calculate the contribution of the position deviation value, heading deviation value, motion state deviation value, steering deviation value, and longitudinal control deviation value to the comprehensive behavior deviation value, and select the deviation type with the largest contribution as the main deviation cause at the current moment; Generate driving guidance suggestions based on the risk level, critical moments, and major deviation causes; determine feedback timing based on the comprehensive behavioral deviation value and its rate of change, determine feedback intensity based on the risk level, mean, and standard deviation; and output control of the driving guidance suggestions based on the feedback timing and feedback intensity.

[0073] In order to improve the driver's driving skills, this solution provides an intelligent driving teaching feedback method and system, which can provide personalized guidance suggestions based on the differences between the driver's actual operation and standard driving behavior.

[0074] Data collection and preprocessing: First, the driver's real-time driving data is collected through on-board sensors (such as GPS, camera, radar, IMU, etc.), including vehicle position, heading angle, speed, acceleration, steering angle and longitudinal control amount (such as the opening of the accelerator and brake pedals). At the same time, using high-precision maps and preset driving scenarios, the corresponding standard driving behavior data is obtained, including standard position, standard heading angle, standard speed, standard acceleration, standard steering angle and standard longitudinal control amount. The collected data is preprocessed, such as removing noise, smoothing data, coordinate system conversion, etc., to ensure the accuracy and reliability of the data.

[0075] For example, in a simulated city road driving scenario, the vehicle is set to turn right at an intersection. The system will pre-record standard driving behavior data, such as the vehicle's position, speed, steering angle, etc. at different times. At the same time, the system will collect the student's driving data in real time.

[0076] Driving behavior deviation value calculation: The collected driving data is input into a pre-trained decision model to obtain a standard driving behavior sequence. The decision model can be a rule-based model, a machine learning model, or a deep learning model.

[0077] Compare the driver's actual driving behavior sequence with the standard driving behavior sequence to calculate the driving behavior deviation value. The specific calculation method is as follows: The straight-line distance between the actual position of the driver and the standard position is calculated to obtain the position deviation value. The absolute value of the difference between the actual heading angle of the driver and the standard heading angle is calculated to obtain the heading deviation value.

[0078] The linear distances between the driver's actual speed and the standard speed, and the driver's actual acceleration and the standard acceleration are calculated respectively, and the linear distance of the speed and the linear distance of the acceleration are weighted and summed to obtain the motion state deviation value. The weight value can be adjusted according to the actual situation.

[0079] Calculate the absolute value of the difference between the driver's actual steering angle and the standard steering angle to obtain the steering deviation value. Calculate the absolute value of the difference between the driver's actual longitudinal control amount and the standard longitudinal control amount to obtain the longitudinal control deviation value. Perform weighted summation on the position deviation value, heading deviation value, motion state deviation value, steering deviation value, and longitudinal control deviation value to obtain the comprehensive behavior deviation value. The weight value can be adjusted according to the actual situation. For example, in the right turn scenario at an intersection, if the trainee's actual position deviates from the standard position by 2 meters, the position deviation value is 2.

[0080] Risk Assessment and Identification of Critical Moments: Calculate the mean and standard deviation of the comprehensive behavior deviation value. Construct a risk assessment interval based on the mean and standard deviation. Determine the moment when the comprehensive behavior deviation value is higher than the sum of the mean and standard deviation as high risk, the moment when the comprehensive behavior deviation value is between the mean minus the standard deviation and the mean plus the standard deviation as medium risk, and the moment when the comprehensive behavior deviation value is lower than the mean minus the standard deviation as low risk.

[0081] The moment when the comprehensive behavior deviation value exceeds a preset proportion (e.g., 80%) of the maximum deviation value is identified as a critical moment.

[0082] Calculate the contribution of the position deviation value, heading deviation value, motion state deviation value, steering deviation value and longitudinal control deviation value to the comprehensive behavior deviation value, and select the deviation type with the largest contribution as the main deviation cause at the current moment.

[0083] For example, if the student's comprehensive behavioral deviation value suddenly increases and exceeds the preset threshold when approaching an intersection, the system will identify this moment as a critical moment and analyze that the main cause of the deviation is the excessive steering deviation value.

[0084] Driving guidance suggestion generation and output control: Generate driving guidance suggestions based on risk level, critical moments and main deviation causes. For example, if the system determines that the current risk level is high and the main deviation cause is excessive steering deviation, the system will prompt the trainee to "pay attention to the excessive steering angle, please correct it in time."

[0085] The timing of feedback is determined based on the comprehensive behavioral deviation value and its rate of change, and the intensity of feedback is determined based on the risk level, average value, and standard deviation. For example, if the comprehensive behavioral deviation value continues to increase, the system will give feedback more frequently; if the risk level is high, the system will give stronger feedback.

[0086] The output of driving guidance suggestions is controlled based on the feedback timing and feedback intensity. For example, the system can provide guidance suggestions to the trainee through voice prompts, image displays or tactile feedback.

[0087] Beneficial effects: Improve driving safety: Through real-time monitoring and analysis of driver behavior, potential risks can be discovered in a timely manner, and corresponding guidance suggestions can be provided to help drivers avoid dangerous operations, thereby improving driving safety. Personalized teaching: According to the driver's actual driving behavior and deviations, personalized guidance suggestions are provided to make teaching more targeted and effectively improve the driver's driving skills. Optimize teaching efficiency: By automatically analyzing driving data and generating feedback, the instructor's workload can be reduced and teaching efficiency can be improved.

[0088] Figure 2 Schematic diagram of the structure of the intelligent driving scenario adaptive teaching system based on reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to collect intelligent driving history data, wherein the intelligent driving history data includes driver operation data, vehicle status data, and environmental perception data; train multiple scene recognition models according to the intelligent driving history data, each of which corresponds to a preset scene type; classify the intelligent driving history data based on each scene recognition model to generate a training sample set with scene labels, wherein the scene recognition model is constructed based on a bidirectional long short-term memory network; input the training sample set with scene labels into a reinforcement learning algorithm for training to obtain an intelligent driving decision model for different scene types; The second unit is used to collect intelligent driving data in real time through vehicle-mounted sensors, wherein the intelligent driving data includes vehicle position information, vehicle speed information, acceleration information, steering wheel angle information, brake pedal position information, and accelerator pedal position information; input the intelligent driving data into the scene recognition model, and output the scene type of the current driving scene; and select a corresponding model from the pre-trained intelligent driving decision model as the current decision model according to the scene type; The third unit is used to input the intelligent driving data into the current decision model to obtain a standard driving behavior sequence; compare the driver's actual driving behavior sequence with the standard driving behavior sequence to calculate a driving behavior deviation value; determine a teaching feedback strategy according to the driving behavior deviation value, wherein the teaching feedback strategy includes at least one of voice prompts, visual prompts, and tactile feedback; output driving guidance information in real time according to the teaching feedback strategy, and use the driver's feedback result on the driving guidance information to update the training parameters of the current decision model.

[0089] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0090] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0091] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The intelligent driving scenario adaptive teaching method based on reinforcement learning is characterized by: include: Collecting intelligent driving historical data, wherein the intelligent driving historical data includes driver operation data, vehicle status data, and environmental perception data; Training a plurality of scene recognition models according to the intelligent driving historical data, each of the scene recognition models corresponding to a preset scene type; Based on each of the scene recognition models, the intelligent driving historical data is subjected to scene classification to generate a training sample set with scene labels, wherein the scene recognition model is constructed based on a bidirectional long short-term memory network; the training sample set with scene labels is input into a reinforcement learning algorithm for training to obtain an intelligent driving decision model for different scene types; Collecting intelligent driving data in real time through vehicle-mounted sensors, the intelligent driving data including vehicle position information, vehicle speed information, acceleration information, steering wheel angle information, brake pedal position information, and accelerator pedal position information; inputting the intelligent driving data into the scene recognition model, and outputting the scene type of the current driving scene; and selecting a corresponding model from the pre-trained intelligent driving decision models as the current decision model according to the scene type; The intelligent driving data is input into the current decision model to obtain a standard driving behavior sequence; the actual driving behavior sequence of the driver is compared with the standard driving behavior sequence to calculate a driving behavior deviation value; a teaching feedback strategy is determined according to the driving behavior deviation value, and the teaching feedback strategy includes at least one of voice prompts, visual prompts, and tactile feedback; driving guidance information is output in real time according to the teaching feedback strategy, and the driver's feedback result on the driving guidance information is used to update the training parameters of the current decision model.

2. The method according to claim 1, characterized in that Classifying the intelligent driving history data based on each of the scene recognition models to generate a training sample set with scene labels includes: Determine the time window length according to the ratio of the absolute speed of the vehicle to the preset reference speed, and set the time window length with a minimum threshold and a maximum threshold; perform sliding segmentation on the normalized feature vector corresponding to the intelligent driving history data based on the time window length, and calculate the differential feature and statistical feature of the normalized feature vector in each time window, wherein the statistical feature includes a mean feature, a standard deviation feature, a skewness feature, and a kurtosis feature; and combine the differential feature and the statistical feature to form an enhanced feature vector; Inputting the enhanced feature vector into a scene recognition model constructed based on a bidirectional long short-term memory network to obtain a forward hidden layer state and a reverse hidden layer state respectively; fusing the forward hidden layer state and the reverse hidden layer state to obtain a fused hidden layer state; Perform multi-head self-attention calculation on the fused hidden layer state to obtain attention features; combine the fused hidden layer state with the attention features and input them into a classifier to obtain a scene type probability distribution; when the maximum value of the scene type probability distribution is greater than a preset confidence threshold, take the corresponding scene type as the classification result; when the maximum value of the scene type probability distribution is less than a preset confidence threshold, mark the scene type as an unknown type.

3. The method according to claim 1, characterized in that: Before inputting the training sample set with scene labels into a reinforcement learning algorithm for training to obtain intelligent driving decision models for different scene types, the method further includes constructing reinforcement learning training samples: Constructing a vehicle state feature vector, an environment state feature vector, and a driving state feature vector based on the intelligent driving history data; fusing the vehicle state feature vector, the environment state feature vector, the driving state feature vector, and the scene type information to obtain a state space representation; Determine the steering action constraint range according to the current vehicle speed and road type, and determine the acceleration action constraint range according to the current vehicle speed and traffic flow information; construct a steering control action space based on the steering action constraint range, and the steering control action space includes a target steering angle, a steering angular rate, and a steering execution duration; construct a longitudinal control action space based on the acceleration action constraint range, and the longitudinal control action space includes an acceleration instruction and a braking instruction; combine the steering control action space with the longitudinal control action space to form a continuous action space; The minimum obstacle avoidance distance is calculated to obtain the collision risk assessment value, the steering change and acceleration change are calculated to obtain the trajectory stability assessment value, the acceleration and steering angle rate are calculated to obtain the ride comfort assessment value, and the speed deviation and heading angle error are calculated to obtain the driving efficiency assessment value; Calculating a scene adaptation weight coefficient according to the state space representation; The collision risk assessment value, the trajectory stability assessment value, the ride comfort assessment value, and the driving efficiency assessment value are weighted with the corresponding scene adaptive weight coefficients to obtain a comprehensive reward value; the state space representation, continuous action space, comprehensive reward value, scene type information, and scene adaptive weight coefficient are combined into training samples, and the training samples are classified and stored according to the scene type.

4. The method according to claim 1, characterized in that The training sample set with scene labels is input into the reinforcement learning algorithm for training, and the intelligent driving decision-making models for different scene types are obtained, including: Obtain a scene type code and a scene attribute vector, input the scene type code and the scene attribute vector into a multilayer perceptron for feature extraction to obtain scene features; fuse the scene features with the state space representation to obtain a scene condition embedding vector; construct an action mean network and an action variance network based on the scene condition embedding vector, the action mean network outputs an action mean, and the action variance network outputs an action variance after softplus transformation; construct a Gaussian strategy distribution based on the action mean and the action variance; Input the scenario condition embedding vector into the value network to obtain a state value estimate; calculate the temporal difference error based on the state value estimate; calculate the priority sampling probability based on the temporal difference error and the advantage function; sample training data from the experience pool based on the priority sampling probability; calculate the importance weight ratio of the current strategy distribution to the historical strategy distribution; truncate the advantage function according to the importance weight ratio to obtain the strategy objective function; Calculate the mean square error between the state value estimate and the target value to obtain the value function loss; calculate the information entropy of the policy distribution to obtain the policy entropy loss; perform weighted combination of the policy objective function, value function loss, and policy entropy loss to obtain the overall loss function; calculate the scenario difficulty index, and construct a scenario sampling probability distribution based on the scenario difficulty index; perform multi-scenario cross-training according to the scenario sampling probability distribution; and iteratively optimize the policy network parameters and the value network parameters using an adaptive learning rate, which is dynamically adjusted according to the training rounds.

5. The method according to claim 4, characterized in that Inputting the scene condition embedding vector into the value network to obtain a state value estimate; calculating a temporal difference error based on the state value estimate; Calculating the priority sampling probability according to the timing difference error and the advantage function includes: Construct a dual value network, input the scenario condition embedding vector into the first value network and the second value network respectively, and obtain the first state value estimation value and the second state value estimation value; select the minimum value of the first state value estimation value and the second state value estimation value as the state value estimation result; calculate the discount sum of the n-step cumulative reward and the target state value to obtain the n-step return value; perform weighted combination of the return values ​​of different step lengths to obtain the λ return value; Calculate the difference between the discounted sum of the immediate reward and the target state value and the current state value to obtain a time series difference error; calculate the mean and standard deviation of the time series difference error, and use the mean and standard deviation to normalize the time series difference error to obtain a normalized time series difference error; calculate a generalized advantage estimate based on the normalized time series difference error, and use the weighted sum of the generalized advantage estimate and the time series difference error as a basic priority; perform momentum update on the basic priority and the historical priority to obtain an updated priority; A sampling probability distribution is calculated according to the updated priority; an importance weight is calculated based on the sampling probability distribution; the updated priority is divided by a temperature parameter and indexed to obtain an indexed priority; and the indexed priority is normalized to obtain a final sampling probability.

6. The method according to claim 1, characterized in that Inputting the intelligent driving data into the current decision model to obtain a standard driving behavior sequence; Comparing the driver's actual driving behavior sequence with the standard driving behavior sequence, and calculating a driving behavior deviation value; Determining the teaching feedback strategy according to the driving behavior deviation value includes: The Euclidean distance between the actual position of the driver and the standard position is calculated to obtain the position deviation value, the absolute difference between the actual heading angle of the driver and the standard heading angle is calculated to obtain the heading deviation value, the Euclidean distance between the actual speed of the driver and the standard speed, and the Euclidean distance between the actual acceleration of the driver and the standard acceleration are calculated respectively, and the speed Euclidean distance and the acceleration Euclidean distance are weighted to obtain the motion state deviation value; the absolute difference between the actual steering angle of the driver and the standard steering angle is calculated to obtain the steering deviation value, and the absolute difference between the actual longitudinal control amount of the driver and the standard longitudinal control amount is calculated to obtain the longitudinal control deviation value; the position deviation value, heading deviation value, motion state deviation value, steering deviation value, and longitudinal control deviation value are weighted to obtain the comprehensive behavior deviation value; Calculate the mean and standard deviation of the comprehensive behavior deviation value; construct a risk assessment interval based on the mean and standard deviation, determine the moment when the comprehensive behavior deviation value is higher than the sum of the mean and standard deviation as high risk, determine the moment when the comprehensive behavior deviation value is between the mean minus the standard deviation and the mean plus the standard deviation as medium risk, and determine the moment when the comprehensive behavior deviation value is lower than the mean minus the standard deviation as low risk; identify the moment when the comprehensive behavior deviation value exceeds a preset proportion of the maximum deviation value as a critical moment; calculate the contribution of the position deviation value, heading deviation value, motion state deviation value, steering deviation value, and longitudinal control deviation value to the comprehensive behavior deviation value, and select the deviation type with the largest contribution as the main deviation cause at the current moment; Generate driving guidance suggestions based on risk level, critical moments, and major deviation causes; determine feedback timing based on the comprehensive behavioral deviation value and its rate of change, determine feedback intensity based on the risk level, mean, and standard deviation; and output control of the driving guidance suggestions based on the feedback timing and feedback intensity.

7. An intelligent driving scenario adaptive teaching system based on reinforcement learning, used to implement the method as described in any one of claims 1 to 6, characterized in that: include: The first unit is used to collect intelligent driving historical data, wherein the intelligent driving historical data includes driver operation data, vehicle status data, and environment perception data; Training a plurality of scene recognition models according to the intelligent driving historical data, each of the scene recognition models corresponding to a preset scene type; Based on each of the scene recognition models, the intelligent driving historical data is subjected to scene classification to generate a training sample set with scene labels, wherein the scene recognition model is constructed based on a bidirectional long short-term memory network; the training sample set with scene labels is input into a reinforcement learning algorithm for training to obtain an intelligent driving decision model for different scene types; The second unit is used to collect intelligent driving data in real time through vehicle-mounted sensors, wherein the intelligent driving data includes vehicle position information, vehicle speed information, acceleration information, steering wheel angle information, brake pedal position information, and accelerator pedal position information; input the intelligent driving data into the scene recognition model, and output the scene type of the current driving scene; and select a corresponding model from the pre-trained intelligent driving decision model as the current decision model according to the scene type; The third unit is used to input the intelligent driving data into the current decision model to obtain a standard driving behavior sequence; compare the driver's actual driving behavior sequence with the standard driving behavior sequence to calculate a driving behavior deviation value; determine a teaching feedback strategy according to the driving behavior deviation value, wherein the teaching feedback strategy includes at least one of voice prompts, visual prompts, and tactile feedback; output driving guidance information in real time according to the teaching feedback strategy, and use the driver's feedback result on the driving guidance information to update the training parameters of the current decision model.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images