Driver Misoperation Recognition and Protection Method Based on Online Learning and Back-end Fusion

Abstract

The present invention belongs to the technical field of artificial intelligence, and discloses a method for identifying and protecting driver's misoperations based on online learning and backend fusion. The method includes: during the driving process of the vehicle, acquiring the driver's operation data, vehicle state data, and video stream collected by the vehicle's camera at the current moment; inputting the driver's operation data, vehicle state data, and video stream into an intelligent safety feedback network for online feedback learning; outputting the driving information of the vehicle at the next moment through the intelligent safety feedback network, and controlling the vehicle based on the driving information. The intelligent safety feedback network performs online feedback learning according to the online feedback learning strategy. In the present invention, the intelligent safety feedback network combines a neural network with a feedback learning network. The intelligent safety feedback network has the ability of online learning, improving the safety of vehicle driving. As the number of times the driver drives increases, the number of times the intelligent safety feedback network performs online learning also increases, and the prediction result is more accurate.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to a method for identifying and protecting driver's misoperations based on online learning and backend fusion. Background Art

[0002] Vehicle driving safety has always been the focus of attention of people and society. In recent years, due to the rise of generative large models, vehicle driving safety technology has reached a new climax. Currently, technologies such as environmental perception and decision-making technology, vehicle networking technology, intelligent driving function development, and driver behavior monitoring technology are mainly applied. The generative large model is the culmination of various artificial intelligence means mentioned above. It is precisely because of the addition of the generative large model that the intelligence of vehicle safety has been improved to a higher level.

[0003] However, the results generated by the generative large model in related technologies have deviation, and the output results of the same problem may have errors or even incorrect results, and have a certain degree of randomness. The output results under the same conditions may not be consistent. Moreover, since the generative large model is a pre-trained model, it has problems such as a large data scale, a deep network structure, and an unexplainable principle. Therefore, when applying the generative large model to vehicle intelligent safety, there will be certain potential safety hazards for the vehicle. Summary of the Invention

[0004] The purpose of the present invention is to combine a neural network with a feedback learning network in an intelligent safety feedback network. The entire intelligent safety feedback network has the ability of online learning, realizes the application of a small model network, avoids the error of the results of the generative large model, and improves the safety of vehicle driving. As the number of times the driver drives increases, the number of times of online learning of the intelligent safety feedback network also increases, and the prediction results are more accurate.

[0005] In a first aspect, an embodiment of the present invention provides a method for identifying and protecting driver's misoperations based on online learning and backend fusion, and the method includes:

[0006] During the vehicle driving process, obtain the driver operation data, vehicle state data, and video stream collected by the vehicle's camera at the current moment;

[0007] Input the driver operation data, the vehicle state data, and the video stream into an intelligent safety feedback network for online feedback learning;

[0008] Output the driving information of the vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information;

[0009] Wherein, the online feedback learning process of the intelligent safety feedback network includes:

[0010] Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream;

[0011] Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature;

[0012] Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment;

[0013] The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

[0014] Optionally, the intelligent safety feedback network includes a feature extraction network, a prediction network and a feedback learning network;

[0015] The generating of the online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment comprises:

[0016] The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between the pedal stepping feature prediction value and a pedal stepping feature true value at a next moment, and an error between the vehicle motion feature prediction value and a vehicle motion feature true value at a next moment;

[0017] The feedback learning network determines a second online feedback learning strategy of the prediction network based on an error between the predicted value of the driving information and a true value of the driving information at a next moment;

[0018] The feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy.

[0019] Optionally, the feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork, and the feedback learning network includes a first feedback learning subnetwork and a second feedback learning subnetwork;

[0020] The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between the pedal stepping feature prediction value and a pedal stepping feature true value at the next moment, and an error between the vehicle motion feature prediction value and a vehicle motion feature true value at the next moment, including:

[0021] The first feedback learning sub-network generates a first online feedback learning sub-strategy for a first feature extraction sub-network based on an error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, wherein the first feature extraction sub-network is used to extract the pedal stepping feature and the driving feature;

[0022] The second feedback learning sub-network generates a second online feedback learning sub-strategy for a second feature extraction sub-network based on the error between the predicted value of the vehicle motion feature and the actual value of the vehicle motion feature at the next moment, and the second feature extraction sub-network is used to extract vehicle motion features and vehicle driving environment features.

[0023] Optionally, the prediction network includes a first prediction sub-network and a second prediction sub-network, and the feedback learning network also includes a third feedback learning sub-network and a fourth feedback learning sub-network;

[0024] The feedback learning network determines a second online feedback learning strategy of the prediction network based on an error between the predicted value of the driving information and the actual value of the driving information at the next moment, including:

[0025] The third feedback learning subnetwork determines a third online feedback learning sub-strategy for the first prediction subnetwork and the first feature extraction subnetwork based on an error between a driving information prediction value predicted by the first prediction subnetwork and a true driving information value at the next moment, wherein the first prediction subnetwork predicts a driving information prediction value at the next moment based on the driving features at the current moment;

[0026] The fourth feedback learning subnetwork determines a fourth online feedback learning sub-strategy for the second prediction subnetwork and the second feature extraction subnetwork based on the error between the driving information prediction value predicted by the second prediction subnetwork and the actual value of the driving information at the next moment, wherein the second prediction subnetwork predicts the driving information prediction value at the next moment based on the vehicle driving environment characteristics at the current moment.

[0027] Optionally, the feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy, including:

[0028] The first feature extraction sub-network adjusts the model parameters based on the first online feedback learning sub-strategy and the third online feedback learning sub-strategy;

[0029] The second feature extraction sub-network adjusts the model parameters based on the second online feedback learning sub-strategy and the fourth online feedback learning sub-strategy;

[0030] The first prediction sub-network adjusts the model parameters based on the third online feedback learning sub-strategy;

[0031] The second prediction sub-network adjusts the model parameters based on the fourth online feedback learning sub-strategy.

[0032] Optionally, the controlling the vehicle based on the driving information includes:

[0033] Obtaining driver state information from a driver monitoring system;

[0034] Inputting the driver state information and the driving information output from the intelligent safety feedback network into a misoperation detection network;

[0035] Determining whether the operation of the vehicle driving in the current driving environment is a misoperation through the misoperation detection network and the driver monitoring system;

[0036] If it is determined that the operation of the vehicle driving in the current driving environment is a misoperation, triggering a misoperation protection system to execute a misoperation protection strategy, and controlling the vehicle through the operation protection strategy.

[0037] Optionally, the triggering the misoperation protection system to execute the misoperation protection strategy and controlling the vehicle through the operation protection strategy includes:

[0038] Determining an acceleration range and a speed range of the vehicle based on the current driving environment and the driving mode of the driver;

[0039] Controlling the acceleration of the vehicle within the acceleration range and controlling the speed of the vehicle within the speed range.

[0040] In a second aspect, an embodiment of the present invention provides a driver misoperation recognition and protection device based on online learning and backend fusion. The device includes:

[0041] A data acquisition module, configured to acquire driver operation data, vehicle state data, and a video stream collected by a camera of the vehicle at the current moment during the driving process of the vehicle;

[0042] A data input module, configured to input the driver operation data, the vehicle state data, and the video stream into an intelligent safety feedback network for online feedback learning;

[0043] A driving information prediction module, configured to output driving information of a vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information;

[0044] An online learning feedback module, specifically configured to:

[0045] Extract driving features and pedal depression features at the current moment based on the driver operation data and vehicle state data, and extract vehicle driving environment features and vehicle motion features at the current moment based on the video stream;

[0046] Predict a predicted value of the pedal depression feature at the next moment based on the pedal depression feature at the current moment, predict a predicted value of the vehicle motion feature at the next moment based on the vehicle motion feature at the current moment, and predict a predicted value of the driving information at the next moment based on the driving feature and vehicle driving environment feature at the current moment;

[0047] Generate an online feedback learning strategy based on the error between the predicted value of the pedal depression feature and the true value of the pedal depression feature at the next moment, the error between the predicted value of the vehicle motion feature and the true value of the vehicle motion feature at the next moment, and the error between the predicted value of the driving information and the true value of the driving information at the next moment;

[0048] Perform online feedback learning on the intelligent safety feedback network based on the online feedback learning strategy.

[0049] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0050] At least one processor;

[0051] A memory for storing executable instructions of the at least one processor;

[0052] Wherein, the at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0053] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enabling the electronic device to execute the method described in the first aspect.

[0054] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, where the computer program implements the method described in the first aspect when executed by a processor.

[0055] The technical solution provided by the embodiments of the present invention combines a neural network with a feedback learning network in the intelligent safety feedback network, enabling the entire intelligent safety feedback network to have the ability of online learning, realizing the application of a small model network, avoiding the error of the results of the generative large model, and improving the safety of vehicle driving. As the number of driving times of the driver increases, the number of online learning times of the intelligent safety feedback network also increases. Under the adjustment of the feedback learning network, the parameter weights of the entire intelligent safety feedback network will be more accurate than those adjusted solely relying on a mathematical model, improving the matching degree between the vehicle and the driver and enhancing the safety of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the prediction process and online feedback learning of the intelligent safety feedback network;

[0057] Figure 2 It is a schematic diagram of the overall execution process of the intelligent safety feedback network and the misoperation detection network;

[0058] Figure 3 It is a schematic diagram of the misoperation protection system performing misoperation protection;

[0059] Figure 4 It is a flowchart of a driver misoperation recognition and protection method based on online learning and backend fusion provided by the embodiments of the present invention;

[0060] Figure 5 It is a flowchart of the online feedback learning process of the intelligent safety feedback network provided by the embodiments of the present invention;

[0061] Figure 6 It is a schematic structural diagram of a driver misoperation recognition and protection device based on online learning and backend fusion provided by the embodiments of the present invention;

[0062] Figure 7 It is a schematic structural diagram of an electronic device provided by the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] The present invention will be described in detail below through embodiments.

[0064] Vehicle driving safety has always been a key concern of people and society. In recent years, due to the rise of generative large models, vehicle driving safety technology has reached a new climax. Currently, technologies such as environmental perception and decision-making technology, vehicle networking technology, intelligent driving function development, and driver behavior monitoring technology are mainly applied. The generative large model is the culmination of various artificial intelligence means mentioned above. It is precisely because of the addition of the generative large model that the intelligence of vehicle safety has been elevated to a new level.

[0065] However, the problem that follows is that the results generated by generative large models have a deviation in orientation. The output results for the same problem may have errors, or even incorrect results, and have a certain degree of randomness. The output results under the same conditions may not be consistent. Generative large models have a large data scale and a deep network structure, and their principles are unexplainable. Therefore, there are certain potential hazards when applied to vehicle intelligent safety.

[0066] In order to achieve the safety of intelligent vehicle control, the risks brought by generative large models should be avoided, and a small model network should be established. In addition, at present, generative large models are a type of pre-trained model, which cannot achieve online learning or synchronize the information of the driver. While this solution is based on the development of artificial intelligence technology, an online feedback learning network is established. The small model network formed by the combination of the two can real-time monitor driving elements such as the driver's driving actions, changes in the driving environment, traffic signals, and weather. Under the intelligent safety level that the pre-trained model can reach, it can also continuously learn and adjust the parameters of the model, making the prediction accuracy of the model relatively high.

[0067] Specifically, this solution is based on the existing neural network technology, combined with the feedback learning network. On the original pre-trained model, as the vehicle usage rate of the driver increases, the feedback learning network continuously corrects the neural network, forming a small model network, enabling the neural network to have the function of online learning, which can greatly avoid the error of the output results of generative large models and improve the intelligent safety performance of the vehicle.

[0068] To describe the solution clearly, first, the overall technical solution provided by the embodiments of the present invention will be elaborated in detail. The technical solution of the present invention mainly consists of three parts, including an intelligent safety feedback network, a misoperation detection network, and a misoperation protection system.

[0069] First, the intelligent safety feedback network will be elaborated below. As Figure 1 shown, it is a schematic diagram of the prediction process and online feedback learning of the intelligent safety feedback network. As Figure 1 can be seen, the intelligent safety feedback network has two inputs, two feature extraction networks (convolutional neural network one and convolutional neural network two respectively), and two prediction networks (Markov chain network one and Markov chain network two respectively), and is jointly regulated by four feedback learning networks for the parameters of the two feature extraction networks and the parameters of the two prediction networks.

[0070] The function of the intelligent safety feedback network is to predict driving information such as the driver's operations, driving modes, and various elements during vehicle driving, and through the feedback network learning, update the parameters of the two feature extraction networks and the parameters of the two prediction networks online, so that as the driver's usage amount increases, the prediction value becomes more accurate.

[0071] There are two types of input signals for the intelligent safety feedback network. The first input signal is the driver's driving operation and the vehicle's motion status. The second input signal is the video signal (also called video stream) collected by the vehicle's camera.

[0072] In the first part, the first input signal will be explained.

[0073] (1) The driver's operation data can be monitored through operation monitoring, and the driver's operation data may include pedal driving and pedal braking data. The vehicle status data may also be obtained, and the vehicle status data may include vehicle speed, vehicle acceleration, vehicle automatic cruise status, vehicle yaw status, etc.

[0074] (2) The data obtained through (1) is input into a time series encoder. The time series encoder is used to associate the series of input data. The associated data is then input into a convolutional neural network 1. The convolutional neural network 1 compresses the data capacity and extracts data features.

[0075] (3) Convolutional neural network: The extracted features are input into the time series decoder, and the driving features are output through the time series decoder ( Figure 1 Driving scheme shown) and pedal depression characteristics ( Figure 1 For any current moment, the pedal pedaling characteristics at the next moment can be predicted based on the pedal pedaling characteristics at the current moment.

[0076] (4) Input the driving characteristics into the prediction network 1 to predict the driver's operation, the vehicle's current operating mode, the vehicle's environment and other driving information. The prediction network 1 is formed based on the Markov chain mathematical model.

[0077] (5) All prediction results in prediction network 1 are input into feedback learning network 2, and are also input into the network integrating Kalman filter algorithm as one of the inputs of the network.

[0078] (6) The pedal stepping feature at the next moment predicted in step (3) is input into the feedback learning network 1 as one of the inputs of the feedback learning network 1. The other input of the feedback learning network 1 is the true value of the pedal stepping feature at the next moment.

[0079] In the second part, the second input signal and the intelligent safety feedback network jointly regulated by the two input signals will be described.

[0080] (7) The video signal is used as the input signal and input into the convolutional neural network 2. By compressing the video information, the feature information of the current moment is extracted into the feature space.

[0081] (8) Extract obstacle information (corresponding to the target in Figure 1 ), traffic light information (corresponding to the traffic signal in Figure 1 ), lane line information (corresponding to the lane line in Figure 1 ) etc. from the feature space at the current moment, and input the obstacle information, traffic light information, and lane line information into the second prediction network to predict the driver's operations (corresponding to the operation prediction in Figure 1 ), the current operating mode of the vehicle (corresponding to the driving mode prediction in Figure 1 ), the environment where the vehicle is located (corresponding to the vehicle driving environment prediction in Figure 1 ) and other driving information. The second prediction network is also formed based on the Markov chain mathematical model.

[0082] (9) Input all the prediction results output by the second prediction network into the system of the integrated Kalman filter algorithm network as one of the inputs of this network.

[0083] (10) Input all the prediction results output by the second prediction network into the feedback learning network three.

[0084] (11) Extract the vehicle motion features at the previous moment (corresponding to the previous moment motion signal in Figure 1 ) from the feature space to the motion feature space, and predict the vehicle motion features at the next moment based on the vehicle motion features at the previous moment, that is, predict the vehicle motion features at the current moment, and input the predicted vehicle motion features into the feedback learning network four (not shown in Figure 1 ).

[0085] (12) Input the vehicle motion features at the current moment (corresponding to the current moment motion signal in Figure 1 ) into the feedback learning network four, that is, input the real vehicle motion features at the current moment into the feedback learning network four. The function of the feedback learning network four can be understood as: for any moment, adjust the parameters of the second convolutional neural network based on the error between the real value and the predicted value of the vehicle motion features at that moment.

[0086] (13) Calibrate the second convolutional neural network by the feedback network four to form an instant learning network, and train the weight values of the second convolutional neural network related to the motion features, that is, adjust the parameters of the second convolutional neural network by the feedback network four.

[0087] (14) Evaluate the accuracy of the driving information predicted by the integrated Kalman filter algorithm network for Markov chain network 1 and Markov chain network 2, that is, compare the driving information predicted by Markov chain network 1 with the true driving information, and compare the driving information predicted by Markov chain network 2 with the true driving information, and input the comparison results into feedback learning network 2 and feedback learning network 3 respectively.

[0088] (15) Feedback learning network 2 adjusts the weight parameters inside Markov chain network 1 according to the input of the Kalman filter algorithm integration system and the input of prediction network 1, and completes one learning cycle of Markov chain network 1.

[0089] (16) Input the output signal of feedback learning network 2 into feedback learning network 1, and combine the predicted pedal stepping characteristics to adjust the parameter weights of predicting pedal actions in convolutional neural network 1, and complete one learning cycle of convolutional neural network 1.

[0090] (17) Feedback learning network 3 adjusts the weight parameters in Markov chain network 2 according to the output result of Markov chain network 2 and the output result of the integrated Kalman algorithm network, and completes one learning cycle of Markov chain network 2. At the same time, the output results of this feedback network 3 and the output results of feedback learning network 4 can jointly adjust the network weight parameters for extracting spatial features in convolutional neural network 2, and complete one learning cycle of spatial feature extraction.

[0091] It can be seen that for the technical solution provided by the embodiment of the present invention, the intelligent safety feedback network combines the neural network and the feedback learning network, enabling the entire intelligent safety feedback network to have the ability of online learning, realizing the application of small model networks, avoiding the error of the results of generative large models, and improving the safety of vehicle driving. As the number of driving times of the driver increases, the number of online learning times of the intelligent safety feedback network also increases. Under the adjustment of the feedback learning network, the parameter weights of the entire intelligent safety feedback network will be more accurate than those adjusted solely by mathematical models, improving the adaptation degree between the vehicle and the driver and enhancing the safety of the vehicle.

[0092] Next, the misoperation detection network will be elaborated.

[0093] Figure 2 It is a schematic diagram of the overall execution process of the intelligent safety feedback network and the misoperation detection network. As Figure 2 shown, driving plan prediction, integrated Kalman filter network (corresponding to Figure 1 the integrated Kalman algorithm network in Figure 1The driving information output by the intelligent safety feedback network described in Figure 2 That is, the input of the misoperation detection network (corresponding to the misoperation detection of

[0094] The following will elaborate on Figure 2 in detail.

[0095] (1) Input pedal drive, pedal brake, vehicle speed, acceleration, gear shifting, etc. into the intelligent safety feedback network, and the intelligent safety feedback network predicts the driving plan, that is, predicts the current driving mode of the driver. This driving mode can be aggressive or conservative, etc., and serves as one input to the misoperation detection network.

[0096] (2) Take the operating design domain of the driving plan control output by the driving plan prediction as the input signal of the integrated Kalman filter algorithm network. Among them, the operating design domain of the driving plan control can include urban, suburban, highway, mountain road, rural area, parking lot, etc.

[0097] (3) Input the video signal into the intelligent safety feedback network, extract the driving information features in the video stream, and obtain the determination results of the driving environment as rural, suburban, urban section, highway, parking lot, etc.

[0098] (4) Take the determination result as one signal of the integrated Kalman filter algorithm network.

[0099] (5) Extract the obstacle information features of the video signal through the intelligent safety feedback network to obtain the recognized obstacle signal, traffic signals (speed limit signs, traffic lights, landmark buildings, etc.), and the line, that is, the lane line (the obstacle is on the lane line, near the lane line, and the obstacle moves closer to the lane line). And take the obstacle signal, traffic signal, and line information as one input to the misoperation detection network.

[0100] (6) Extract the road features, weather environment features, road condition features, etc. in the video signal through the intelligent safety feedback network as one input to the misoperation detection network.

[0101] (7) Take the output signal of the integrated Kalman filter network (that is, the integrated Kalman algorithm network shown in Figure 1 ) as one input to the misoperation detection network.

[0102] (8) Input the signals of the driver detection system (including the driver's expression and heart rate, etc.) into the misoperation detection network.

[0103] (9) The misoperation detection network and the driver detection system determine whether the operation of the driver at the current moment in the current driving environment is a misoperation.

[0104] (10) Trigger the misoperation protection system. That is, by controlling operations such as pedal drive, pedal braking, acceleration, speed, and gear shifting, the purpose of misoperation protection is achieved, that is, the misoperation is corrected.

[0105] The misoperation protection system will be elaborated in detail below.

[0106] As Figure 3 shown, it is a schematic diagram of the misoperation protection system performing misoperation protection.

[0107] The misoperation protection system designed in the present invention mainly functions to improve the vehicle intelligent safety mechanism. It belongs to the vehicle control part and will control the vehicle speed within a set speed threshold. The driver cannot exceed this range by accelerating or decelerating through the drive pedal, and moreover, it will control the vehicle acceleration according to the actual situation. Among them, Figure 3 VCU in it refers to the vehicle control unit, MCU refers to the motor controller, EPS refers to the electric power steering system, and ABS refers to the anti-lock braking system.

[0108] (1) When the vehicle is driving, if the misoperation protection system determines that it is in an urban environment or a parking lot environment and there is a speed limit signal, short-term acceleration is allowed, and the vehicle acceleration should decrease slowly subsequently.

[0109] (2) When the misoperation protection system determines that there is an obstacle ahead and it is a static obstacle, the vehicle speed is restricted within a certain range.

[0110] (3) When the misoperation protection system determines that it enters a speed limit area or is about to enter the speed limit area, the acceleration is slowly restricted to a certain specific value in advance.

[0111] (4) When the misoperation protection system determines that there is a dynamic obstacle ahead, the vehicle system speed range and acceleration range are dynamically adjusted according to the speed and acceleration of the dynamic obstacle.

[0112] (5) When the misoperation protection system determines that it is in environments such as cities, villages, suburbs, parking lots, etc., the vehicle speed upper limit and acceleration upper limit are correspondingly restricted.

[0113] (6) When the misoperation protection system determines that the lane line is about to become narrower or wider, the vehicle speed and the upper limit of vehicle acceleration are correspondingly increased or decreased dynamically according to the predicted change amplitude.

[0114] (7) When the misoperation protection system determines that the driver's operation behavior belongs to the aggressive type, the relationship between the pedal change amplitude and the vehicle acceleration is correspondingly restricted.

[0115] (8) When the misoperation protection system determines that the driver's operation behavior is conservative, the relationship between the change range of the gain pedal and the vehicle acceleration.

[0116] (9) When the misoperation protection system determines that the weather has a great influence on the road friction coefficient and the driver's vision, the vehicle speed is restricted according to the degree of influence.

[0117] It should be noted that the above (1) to (9) only introduce the misoperation protection process of the misoperation protection system in the form of examples. In actual applications, there are many other situations, which will not be listed one by one here.

[0118] After elaborating on the intelligent safety feedback network, the misoperation detection network, and the misoperation protection system, the following will elaborate in detail on a driver misoperation recognition and protection method based on online learning and backend fusion provided by an embodiment of the present invention, as Figure 4 shown, this method may include the following steps:

[0119] S410, during the vehicle driving process, obtain the driver operation data, vehicle state data, and video stream collected by the vehicle's camera at the current moment.

[0120] Among them, the driver operation data may include pedal drive data, pedal brake data, etc. The vehicle state data includes vehicle speed, vehicle acceleration, vehicle automatic cruise state data, and vehicle yaw state data. The video stream may be the video of multiple consecutive moments including the current moment collected by the vehicle's camera.

[0121] S420, input the driver operation data, vehicle state data, and video stream into the intelligent safety feedback network for online feedback learning.

[0122] S430, output the driving information of the vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information.

[0123] Specifically, after obtaining the driver operation data, vehicle state data, and video stream, the driver operation data, vehicle state data, and video stream can be input into the intelligent safety feedback network for online feedback learning. The intelligent safety feedback network outputs the driving information of the vehicle at the next moment, and performs intelligent control on the vehicle according to the driving information.

[0124] The driving information at the next moment may include the driver's driving operation, driving mode, and vehicle driving environment, among which the driving operation may include a driving pedal or a brake pedal; the driving mode may include smoothness or aggressiveness, etc., and the vehicle driving environment may include cities, suburbs, highways, mountain roads, villages, parking lots, etc., and may also include traffic signals, obstacles, lane lines, road conditions, traffic conditions, and weather conditions, etc. It is understandable that the driving information may also include other information, which will not be listed here one by one.

[0125] Among them, the online feedback learning process of the intelligent safety feedback network, such as Figure 5 As shown, the following steps may be included:

[0126] S510, based on the driver operation data and vehicle state data at the current moment, extract the driving characteristics and pedal stepping characteristics at the current moment, and extract the vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream.

[0127] In practical applications, the intelligent safety feedback network may include two feature extraction networks. The current driver operation data and vehicle status data are input into one of the feature extraction networks to extract the current driving characteristics and pedal stepping characteristics. The video stream is input into another feature extraction network to extract the current vehicle driving environment characteristics and vehicle motion characteristics.

[0128] S520, predicting the pedal depression characteristic prediction value at the next moment based on the pedal depression characteristic at the current moment, predicting the vehicle motion characteristic prediction value at the next moment based on the vehicle motion characteristic at the current moment, and predicting the driving information prediction value at the next moment based on the driving characteristics at the current moment and the vehicle driving environment characteristics.

[0129] S530, generates an online feedback learning strategy based on the error between the pedal depression feature prediction value and the actual value of the pedal depression feature at the next moment, the error between the vehicle motion feature prediction value and the actual value of the vehicle motion feature at the next moment, and the error between the driving information prediction value and the actual value of the driving information at the next moment.

[0130] In one embodiment, the intelligent safety feedback network includes a feature extraction network, a prediction network, and a feedback learning network.

[0131] At this time, based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment, generating an online feedback learning strategy may include the following steps:

[0132] The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between a pedal stepping feature prediction value and a pedal stepping feature true value at a next moment, and an error between a vehicle motion feature prediction value and a vehicle motion feature true value at a next moment;

[0133] The feedback learning network determines a second online feedback learning strategy of the prediction network based on the error between the predicted value of the driving information and the true value of the driving information at the next moment;

[0134] The feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy.

[0135] S540: Perform online feedback learning on the intelligent safety feedback network based on the online feedback learning strategy.

[0136] Specifically, the intelligent safety feedback network includes a feature extraction network and a prediction network. Based on the error between the pedal stepping feature prediction value and the actual value of the pedal stepping feature at the next moment, and the error between the vehicle motion feature prediction value and the actual value of the vehicle motion feature at the next moment, the first online feedback learning strategy determined can realize online feedback learning of the feature extraction network, that is, adjust the model parameters of the feature extraction network. Based on the error between the driving information prediction value and the actual value of the driving information at the next moment, the second online feedback learning strategy determined can realize online feedback learning of the prediction network, that is, adjust the model parameters of the prediction network, so that the features extracted by the feature extraction network are more accurate, and the driving information predicted by the prediction network is more accurate.

[0137] It can be seen that the technical solution provided by the embodiment of the present invention is that the intelligent safety feedback network combines the neural network with the feedback learning network, so that the entire intelligent safety feedback network has the ability of online learning, realizes the use of small model networks, avoids the error of the generative large model results, and improves the safety of vehicle driving. As the number of times the driver drives increases, the number of times the intelligent safety feedback network learns online also increases. Under the regulation of the feedback learning network, the parameter weights of the entire intelligent safety feedback network will be more accurate than those adjusted by relying solely on mathematical models, which improves the degree of adaptation between the vehicle and the driver and improves the safety of the vehicle.

[0138] As an implementation method of the embodiment of the present invention, the feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork, and the feedback learning network includes a first feedback learning subnetwork and a second feedback learning subnetwork. The first feature extraction subnetwork corresponds to Figure 1 The convolutional neural network in the first and second feature extraction sub-networks correspond to Figure 1 Convolutional neural network 2 in the first feedback learning sub-network corresponds toFigure 1 The feedback learning network 1 in the second feedback learning sub-network corresponds to Figure 1 Feedback learning network in IV.

[0139] At this time, the feedback learning network determines the first online feedback learning strategy of the feature extraction network based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, and the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, which may include the following steps:

[0140] The first feedback learning sub-network generates a first online feedback learning sub-strategy for the first feature extraction sub-network based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment.

[0141] Among them, the first feature extraction sub-network is used to extract pedal stepping features and driving features;

[0142] The second feedback learning sub-network generates a second online feedback learning sub-strategy for the second feature extraction sub-network based on the error between the predicted value of the vehicle motion feature and the true value of the vehicle motion feature at the next moment.

[0143] Among them, the second feature extraction sub-network is used to extract vehicle motion features and vehicle driving environment features.

[0144] It should be noted that in Figure 1 This part has been explained in the embodiment and will not be repeated here.

[0145] As an implementation method of an embodiment of the present invention, the prediction network includes a first prediction sub-network and a second prediction sub-network, and the feedback learning network also includes a third feedback learning sub-network and a fourth feedback learning sub-network. Figure 1 The Markov chain network 1 in the second prediction subnetwork corresponds to Figure 1 The Markov chain network 2 in the third feedback learning sub-network corresponds to Figure 1 The feedback learning network 2 in the fourth feedback learning sub-network corresponds to Figure 2 Feedback learning network in III.

[0146] At this time, the feedback learning network determines the second online feedback learning strategy of the prediction network based on the error between the predicted value of the driving information and the true value of the driving information at the next moment, which may include the following steps, namely step 1 and step 2:

[0147] Step 1: The third feedback learning subnetwork determines a third online feedback learning sub-strategy for the first prediction subnetwork and the first feature extraction subnetwork based on the error between the driving information prediction value predicted by the first prediction subnetwork and the actual value of the driving information at the next moment.

[0148] Among them, the first prediction sub-network predicts the predicted value of driving information at the next moment based on the driving characteristics at the current moment;

[0149] Step 2, the fourth feedback learning sub-network determines the fourth online feedback learning sub-strategy for the second prediction sub-network and the second feature extraction sub-network according to the error between the predicted value of the driving information predicted by the second prediction sub-network and the true value of the driving information at the next moment.

[0150] Among them, the second prediction sub-network predicts the predicted value of driving information at the next moment based on the vehicle driving environment characteristics at the current moment.

[0151] As an implementation manner of the embodiment of the present invention, the feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy, which may include the following steps:

[0152] The first step, the first feature extraction sub-network adjusts the model parameters based on the first online feedback learning sub-strategy and the third online feedback learning sub-strategy.

[0153] The second step, the second feature extraction sub-network adjusts the model parameters based on the second online feedback learning sub-strategy and the fourth online feedback learning sub-strategy.

[0154] The third step, the first prediction sub-network adjusts the model parameters based on the third online feedback learning sub-strategy.

[0155] The fourth step, the second prediction sub-network adjusts the model parameters based on the fourth online feedback learning sub-strategy.

[0156] Specifically, the first prediction sub-network predicts driving information based on the driving characteristics extracted by the first feature extraction sub-network. Therefore, the first feature extraction sub-network can adjust the model parameters through the first online feedback learning sub-strategy and the third online feedback learning sub-strategy. Similarly, the second prediction sub-network predicts driving information based on the vehicle driving environment characteristics extracted by the first feature extraction sub-network. Therefore, the second feature extraction sub-network can adjust the model parameters through the second online feedback learning sub-strategy and the fourth online feedback learning sub-strategy.

[0157] Based on the above embodiment, as an implementation manner of the embodiment of the present invention, the method may further include the following steps:

[0158] The first step, obtain the driver state information from the driver monitoring system.

[0159] Among them, the driver monitoring system is Figure 2In the DMS of the illustrated embodiment, the driver status information may include information such as the driver's expression and heart rate.

[0160] In the second step, the driver status information and the driving information output from the intelligent safety feedback network are input into the misoperation detection network.

[0161] In the third step, it is determined whether the vehicle operation in the current driving environment is a misoperation through the misoperation detection network and the driver monitoring system.

[0162] In the fourth step, if it is determined that the vehicle operation in the current driving environment is a misoperation, the misoperation protection system is triggered to execute the misoperation protection strategy, and the vehicle is controlled through the operation protection strategy.

[0163] Specifically, after obtaining the driver status information, the driver status information and the driving information can be input into the misoperation detection network. Through the misoperation detection network and the driver monitoring system, it is determined whether there is a misoperation. If there is a misoperation, it indicates that there is a safety hazard in the vehicle currently. Then, the misoperation protection system is triggered to execute the misoperation protection strategy, and the vehicle is controlled through the operation protection strategy.

[0164] As an implementation manner of the embodiment of the present invention, triggering the misoperation protection system to execute the misoperation protection strategy and controlling the vehicle through the operation protection strategy may include the following two steps:

[0165] Step 1, based on the current driving environment and the driver's driving mode, determine the acceleration range and speed range of the vehicle.

[0166] Step 2, control the acceleration of the vehicle within the acceleration range and control the speed of the vehicle within the speed range.

[0167] Specifically, the misoperation protection system can control the speed and acceleration of the vehicle according to the current driving environment and the driver's driving mode. Among them, the acceleration range and speed range can be determined according to the actual situation and are not specifically limited here.

[0168] And how to control the speed and acceleration of the vehicle has been Figure 3 detailedly exemplified in (1) to (9) of the illustrated embodiment and will not be elaborated here.

[0169] The embodiment of the present invention also provides a driver misoperation recognition and protection device 60 based on online learning and backend fusion, as Figure 6 shown, the device includes:

[0170] The data acquisition module 610 is used to acquire the driver's operation data, vehicle status data, and the video stream collected by the camera of the vehicle at the current moment during the vehicle's driving process;

[0171] A data input module 620, for inputting the driver operation data, the vehicle status data, and the video stream into an intelligent safety feedback network for online feedback learning;

[0172] A driving information prediction module 630, configured to output the driving information of the vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information;

[0173] The online learning feedback module 640 is specifically used for:

[0174] Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream;

[0175] Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature;

[0176] Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment;

[0177] The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

[0178] In a third aspect, an embodiment of the present invention provides an electronic device 700, such as Figure 7 As shown, including:

[0179] at least one processor 701;

[0180] a memory 702 for storing the at least one processor executable instruction;

[0181] The at least one processor is configured to execute the instructions to implement the method as described in the first aspect.

[0182] Fourthly, an embodiment of the present invention provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect.

[0183] Fifthly, an embodiment of the present invention provides a computer program product, including a computer program which, when executed by a processor, implements the method described in the first aspect.

[0184] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A driver misoperation identification and protection method based on online learning and back-end fusion, characterized in that: The method comprises: During the driving process of the vehicle, the driver's operation data, vehicle status data, and the video stream collected by the camera of the vehicle are obtained at the current moment; Inputting the driver operation data, the vehicle status data, and the video stream into an intelligent safety feedback network for online feedback learning; Outputting driving information of the vehicle at the next moment through the intelligent safety feedback network, and controlling the vehicle based on the driving information; The online feedback learning process of the intelligent safety feedback network includes: Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream; Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature; Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment; The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

2. The method according to claim 1, characterized in that The intelligent safety feedback network includes a feature extraction network, a prediction network and a feedback learning network; The generating of the online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment comprises: The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between the pedal stepping feature prediction value and a pedal stepping feature true value at a next moment, and an error between the vehicle motion feature prediction value and a vehicle motion feature true value at a next moment; The feedback learning network determines a second online feedback learning strategy of the prediction network based on an error between the predicted value of the driving information and a true value of the driving information at a next moment; The feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy.

3. The method according to claim 2, characterized in that The feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork, and the feedback learning network includes a first feedback learning subnetwork and a second feedback learning subnetwork; The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between the pedal stepping feature prediction value and a pedal stepping feature true value at the next moment, and an error between the vehicle motion feature prediction value and a vehicle motion feature true value at the next moment, including: The first feedback learning sub-network generates a first online feedback learning sub-strategy for a first feature extraction sub-network based on an error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, wherein the first feature extraction sub-network is used to extract the pedal stepping feature and the driving feature; The second feedback learning sub-network generates a second online feedback learning sub-strategy for a second feature extraction sub-network based on the error between the predicted value of the vehicle motion feature and the actual value of the vehicle motion feature at the next moment, and the second feature extraction sub-network is used to extract vehicle motion features and vehicle driving environment features.

4. The method according to claim 3, characterized in that The prediction network includes a first prediction sub-network and a second prediction sub-network, and the feedback learning network also includes a third feedback learning sub-network and a fourth feedback learning sub-network; The feedback learning network determines a second online feedback learning strategy of the prediction network based on an error between the predicted value of the driving information and the actual value of the driving information at the next moment, including: The third feedback learning subnetwork determines a third online feedback learning sub-strategy for the first prediction subnetwork and the first feature extraction subnetwork based on an error between a driving information prediction value predicted by the first prediction subnetwork and a true driving information value at the next moment, wherein the first prediction subnetwork predicts a driving information prediction value at the next moment based on the driving features at the current moment; The fourth feedback learning subnetwork determines a fourth online feedback learning sub-strategy for the second prediction subnetwork and the second feature extraction subnetwork based on the error between the driving information prediction value predicted by the second prediction subnetwork and the actual value of the driving information at the next moment, wherein the second prediction subnetwork predicts the driving information prediction value at the next moment based on the vehicle driving environment characteristics at the current moment.

5. The method according to claim 4, characterized in that The feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy, including: The first feature extraction sub-network adjusts model parameters based on the first online feedback learning sub-strategy and the third online feedback learning sub-strategy; The second feature extraction sub-network adjusts model parameters based on the second online feedback learning sub-strategy and the fourth online feedback learning sub-strategy; The first prediction sub-network adjusts model parameters based on the third online feedback learning sub-strategy; The second prediction sub-network adjusts model parameters based on the fourth online feedback learning sub-strategy.

6. The method according to any one of claims 1 to 5, characterized in that: The controlling the vehicle based on the driving information includes: obtaining driver status information from a driver monitoring system; Inputting the driver status information and the driving information output from the intelligent safety feedback network into a misoperation detection network; Determining whether the operation of the vehicle driving in the current driving environment is an erroneous operation by the erroneous operation detection network and the driver monitoring system; If it is determined that the operation of the vehicle driving in the current driving environment is an erroneous operation, the erroneous operation protection system is triggered to execute the erroneous operation protection strategy, and the vehicle is controlled by the operation protection strategy.

7. The method according to claim 6, characterized in that The triggering of the misoperation protection system to execute the misoperation protection strategy and controlling the vehicle through the operation protection strategy includes: Determine the acceleration range and speed range of the vehicle based on the current driving environment and the driver's driving mode; The acceleration of the vehicle is controlled within the acceleration range, and the speed of the vehicle is controlled within the speed range.

8. A driver misoperation identification and protection device based on online learning and back-end fusion, characterized in that: The device comprises: A data acquisition module is used to acquire the driver's operation data, vehicle status data, and the video stream collected by the camera of the vehicle at the current moment during the vehicle's driving process; A data input module, used for inputting the driver operation data, the vehicle status data, and the video stream into an intelligent safety feedback network for online feedback learning; A driving information prediction module, used to output the driving information of the vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information; Online learning feedback module, specifically for: Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream; Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature; Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment; The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 7.

Images