Driver misoperation identification and protection method based on online learning and back-end fusion

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a driver misoperation identification and protection method based on online learning and back-end fusion, and the method comprises the steps: obtaining driver operation data and vehicle state data at a current moment, and a video stream collected by a camera of a vehicle in a vehicle driving process; inputting the driver operation data, the vehicle state data and the video stream into an intelligent safety feedback network for online feedback learning; and outputting the driving information of the vehicle at the next moment through the intelligent safety feedback network, and controlling the vehicle based on the driving information. And the intelligent security feedback network performs online feedback learning according to the online feedback learning strategy. According to the intelligent safety feedback network, the neural network and the feedback learning network are combined, the intelligent safety feedback network has the online learning capability, and the safety of vehicle driving is improved. Along with the increase of the number of driving times of the driver, the number of online learning times of the intelligent safety feedback network is increased, and the prediction result is more accurate.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a driver misoperation recognition and protection method based on online learning and back-end fusion. Background Art

[0002] Vehicle driving safety has always been the focus of people and society. In recent years, due to the rise of generative big models, vehicle driving safety technology has reached a new climax. Currently, the main applications include environmental perception and decision-making technology, vehicle networking technology, intelligent driving function development, and driver behavior monitoring technology. Generative big models are the culmination of the above various artificial intelligence methods. It is precisely because of the addition of generative big models that the intelligence of vehicle safety has been raised to a higher level.

[0003] However, the results generated by the generative big models in related technologies are biased. The output results of the same problem may have errors or even wrong results, and have a certain degree of randomness. The output results under the same conditions may be inconsistent. And because the generative big model is a pre-trained model, it has problems such as large data scale, deep network structure and unexplainable principles. Therefore, if the generative big model is applied to vehicle intelligent safety, there will be certain safety hazards in the vehicle. Summary of the invention

[0004] The purpose of the present invention is to combine the neural network with the feedback learning network in the intelligent safety feedback network. The entire intelligent safety feedback network has the ability of online learning, realizes the use of small model networks, avoids the error of the generative large model results, and improves the safety of vehicle driving. As the number of driving times of the driver increases, the number of online learning times of the intelligent safety feedback network also increases, and the prediction results are more accurate.

[0005] In a first aspect, an embodiment of the present invention provides a method for identifying and protecting driver misoperation based on online learning and back-end fusion, the method comprising: During the driving process of the vehicle, the driver's operation data, vehicle status data, and the video stream collected by the camera of the vehicle are obtained at the current moment; Inputting the driver operation data, the vehicle status data, and the video stream into an intelligent safety feedback network for online feedback learning; Outputting driving information of the vehicle at the next moment through the intelligent safety feedback network, and controlling the vehicle based on the driving information; The online feedback learning process of the intelligent safety feedback network includes: Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream; Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature; Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment; The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

[0006] Optionally, the intelligent safety feedback network includes a feature extraction network, a prediction network and a feedback learning network; The generating of the online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment comprises: The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between the pedal stepping feature prediction value and a pedal stepping feature true value at a next moment, and an error between the vehicle motion feature prediction value and a vehicle motion feature true value at a next moment; The feedback learning network determines a second online feedback learning strategy of the prediction network based on an error between the predicted value of the driving information and a true value of the driving information at a next moment; The feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy.

[0007] Optionally, the feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork, and the feedback learning network includes a first feedback learning subnetwork and a second feedback learning subnetwork; The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between the pedal stepping feature prediction value and a pedal stepping feature true value at the next moment, and an error between the vehicle motion feature prediction value and a vehicle motion feature true value at the next moment, including: The first feedback learning sub-network generates a first online feedback learning sub-strategy for a first feature extraction sub-network based on an error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, wherein the first feature extraction sub-network is used to extract the pedal stepping feature and the driving feature; The second feedback learning sub-network generates a second online feedback learning sub-strategy for a second feature extraction sub-network based on the error between the predicted value of the vehicle motion feature and the actual value of the vehicle motion feature at the next moment, and the second feature extraction sub-network is used to extract vehicle motion features and vehicle driving environment features.

[0008] Optionally, the prediction network includes a first prediction sub-network and a second prediction sub-network, and the feedback learning network also includes a third feedback learning sub-network and a fourth feedback learning sub-network; The feedback learning network determines a second online feedback learning strategy of the prediction network based on an error between the predicted value of the driving information and the actual value of the driving information at the next moment, including: The third feedback learning subnetwork determines a third online feedback learning sub-strategy for the first prediction subnetwork and the first feature extraction subnetwork based on an error between a driving information prediction value predicted by the first prediction subnetwork and a true driving information value at the next moment, wherein the first prediction subnetwork predicts a driving information prediction value at the next moment based on the driving features at the current moment; The fourth feedback learning subnetwork determines a fourth online feedback learning sub-strategy for the second prediction subnetwork and the second feature extraction subnetwork based on the error between the driving information prediction value predicted by the second prediction subnetwork and the actual value of the driving information at the next moment, wherein the second prediction subnetwork predicts the driving information prediction value at the next moment based on the vehicle driving environment characteristics at the current moment.

[0009] Optionally, the feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy, including: The first feature extraction sub-network adjusts model parameters based on the first online feedback learning sub-strategy and the third online feedback learning sub-strategy; The second feature extraction sub-network adjusts model parameters based on the second online feedback learning sub-strategy and the fourth online feedback learning sub-strategy; The first prediction sub-network adjusts model parameters based on the third online feedback learning sub-strategy; The second prediction sub-network adjusts model parameters based on the fourth online feedback learning sub-strategy.

[0010] Optionally, the controlling the vehicle based on the driving information includes: obtaining driver status information from a driver monitoring system; Inputting the driver status information and the driving information output from the intelligent safety feedback network into a misoperation detection network; Determining whether the operation of the vehicle driving in the current driving environment is an erroneous operation by the erroneous operation detection network and the driver monitoring system; If it is determined that the operation of the vehicle driving in the current driving environment is an erroneous operation, the erroneous operation protection system is triggered to execute the erroneous operation protection strategy, and the vehicle is controlled by the operation protection strategy.

[0011] Optionally, the triggering of the misoperation protection system to execute the misoperation protection strategy and controlling the vehicle through the misoperation protection strategy includes: Determine the acceleration range and speed range of the vehicle based on the current driving environment and the driver's driving mode; The acceleration of the vehicle is controlled to be within the acceleration range, and the speed of the vehicle is controlled to be within the speed range.

[0012] In a second aspect, an embodiment of the present invention provides a driver misoperation identification and protection device based on online learning and back-end fusion, the device comprising: A data acquisition module is used to acquire the driver's operation data, vehicle status data, and the video stream collected by the camera of the vehicle at the current moment during the vehicle's driving process; A data input module, used for inputting the driver operation data, the vehicle status data, and the video stream into an intelligent safety feedback network for online feedback learning; A driving information prediction module, used to output the driving information of the vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information; Online learning feedback module, specifically for: Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream; Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature; Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment; The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect.

[0015] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0016] The technical solution provided by the embodiment of the present invention is that the intelligent safety feedback network combines the neural network with the feedback learning network, so that the entire intelligent safety feedback network has the ability of online learning, realizes the use of small model networks, avoids the error of the generative large model results, and improves the safety of vehicle driving. As the number of times the driver drives increases, the number of online learning of the intelligent safety feedback network also increases. Under the regulation of the feedback learning network, the parameter weights of the entire intelligent safety feedback network will be more accurate than those adjusted by relying solely on mathematical models, which improves the degree of adaptation between the vehicle and the driver and improves the safety of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the prediction process and online feedback learning of the intelligent safety feedback network; Figure 2 It is a schematic diagram of the overall execution process of the intelligent safety feedback network and the misoperation detection network; Figure 3A schematic diagram of a misoperation protection system performing misoperation protection; Figure 4 A flowchart of a driver misoperation identification and protection method based on online learning and back-end fusion provided in an embodiment of the present invention; Figure 5 A flow chart of an online feedback learning process of an intelligent safety feedback network provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a driver misoperation recognition and protection device based on online learning and back-end fusion provided by an embodiment of the present invention; Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be described in detail below through examples.

[0019] Vehicle driving safety has always been the focus of people and society. In recent years, due to the rise of generative big models, vehicle driving safety technology has reached a new climax. Currently, the main applications include environmental perception and decision-making technology, vehicle networking technology, intelligent driving function development and driver behavior monitoring technology. Generative big models are the culmination of the above various artificial intelligence methods. It is precisely because of the addition of generative big models that the intelligence of vehicle safety has been raised to a higher level.

[0020] However, the problem that follows is that the results generated by the generative large model are biased. The output results of the same problem may have errors or even wrong results, and have a certain degree of randomness. The output results under the same conditions may be inconsistent. The data scale of the generative large model is large, the network structure is deep, and its principle is unexplainable. Therefore, there are certain hidden dangers in its application in vehicle intelligent safety.

[0021] In order to achieve the safety of intelligent vehicle control, the risks brought by the generative large model should be avoided and a small model network should be established. In addition, the current generative large model is a pre-trained model, which cannot achieve online learning and cannot synchronize the driver's information. This solution is based on the development of artificial intelligence technology and establishes an online feedback learning network. The small model network formed by the combination of the two can monitor the driver's driving actions, driving environment changes, traffic signals, weather and other driving factors in real time. Under the intelligent safety level that the pre-trained model can achieve, it can also continuously learn and adjust the model parameters, so that the model's prediction accuracy is higher.

[0022] Specifically, this solution is based on the existing neural network technology and combined with a feedback learning network. On the original pre-trained model, as the driver's vehicle usage rate increases, the feedback learning network continuously corrects the neural network to form a small model network, so that the neural network has online learning capabilities, which can greatly avoid the error of the output results of the generative large model and improve the vehicle's intelligent safety performance.

[0023] In order to describe the solution clearly, the overall technical solution provided by the embodiment of the present invention is first described in detail. The technical solution of the present invention mainly consists of three parts, including an intelligent safety feedback network, a misoperation detection network and a misoperation protection system.

[0024] The following first describes the intelligent safety feedback network. Figure 1 As shown in Figure 1, it is a schematic diagram of the prediction process and online feedback learning of the intelligent safety feedback network. Figure 1 It can be seen that the intelligent safety feedback network has two inputs, two feature extraction networks (convolutional neural network 1 and convolutional neural network 2), and two prediction networks (Markov chain network 1 and Markov chain network 2), and four feedback learning networks jointly adjust the parameters of the two feature extraction networks and the parameters of the two prediction networks.

[0025] The function of the intelligent safety feedback network is to predict the driver's operation, driving mode, various factors in the vehicle's driving process and other driving information, and to update the parameters of the two feature extraction networks and the parameters of the two prediction networks online through feedback network learning, so that the prediction value becomes more accurate as the driver's usage increases.

[0026] There are two types of input signals for the intelligent safety feedback network. The first input signal is the driver's driving operation and the vehicle's motion status. The second input signal is the video signal (also called video stream) collected by the vehicle's camera.

[0027] In the first part, the first input signal will be explained.

[0028] (1) The driver's operation data can be monitored through operation monitoring, and the driver's operation data may include pedal driving and pedal braking data. The vehicle status data may also be obtained, and the vehicle status data may include vehicle speed, vehicle acceleration, vehicle automatic cruise status, vehicle yaw status, etc.

[0029] (2) The data obtained through (1) is input into a time series encoder. The time series encoder is used to associate the series of input data. The associated data is then input into a convolutional neural network 1. The convolutional neural network 1 compresses the data capacity and extracts data features.

[0030] (3) Convolutional neural network: The extracted features are input into the time series decoder, and the driving features are output through the time series decoder ( Figure 1 Driving scheme shown) and pedal depression characteristics ( Figure 1 For any current moment, the pedal pedaling characteristics at the next moment can be predicted based on the pedal pedaling characteristics at the current moment.

[0031] (4) Input the driving characteristics into the prediction network 1 to predict the driver's operation, the current operating mode of the vehicle, the environment the vehicle is in, and other driving information. The prediction network 1 is formed based on the Markov chain mathematical model.

[0032] (5) All prediction results in prediction network 1 are input into feedback learning network 2, and are also input into the network integrating Kalman filter algorithm as one of the inputs of the network.

[0033] (6) The pedal stepping feature at the next moment predicted in step (3) is input into the feedback learning network 1 as one of the inputs of the feedback learning network 1. The other input of the feedback learning network 1 is the true value of the pedal stepping feature at the next moment.

[0034] In the second part, the second input signal and the intelligent safety feedback network jointly regulated by the two input signals will be described.

[0035] (7) The video signal is used as the input signal and input into the convolutional neural network 2. By compressing the video information, the feature information of the current moment is extracted into the feature space.

[0036] (8) Extract obstacle information from the feature space at the current moment (corresponding to Figure 1 ), traffic light information (corresponding to Figure 1 Traffic signals in) and lane information (corresponding to Figure 1 The obstacle information, traffic light information and lane line information are input into the prediction network 2 to predict the driver's operation (corresponding to the lane line in the Figure 1 Operation prediction in ), the vehicle’s current operating mode (corresponding to Figure 1 driving mode prediction), the vehicle’s environment (corresponding to Figure 1 The prediction network 2 is also formed based on the Markov chain mathematical model.

[0037] (9) All prediction results output by the prediction network 2 are input into the system of the integrated Kalman filter algorithm network as one of the inputs of the network.

[0038] (10) All prediction results output by the prediction network 2 are input into the feedback learning network 3.

[0039] (11) Extract the vehicle motion features at the previous moment from the feature space (corresponding to Figure 1 The vehicle motion features of the previous moment are predicted based on the vehicle motion features of the previous moment, that is, the vehicle motion features of the current moment are predicted, and the predicted vehicle motion features are input into the feedback learning network 4 (in Figure 1 not shown).

[0040] (12) The vehicle motion characteristics at the current moment (corresponding to Figure 1 The current moment motion signal in the convolutional neural network is input into the feedback learning network 4, that is, the real vehicle motion feature at the current moment is input into the feedback learning network 4. The role of the feedback learning network 4 can be understood as: for any moment, based on the error between the real value of the vehicle motion feature at that moment and the predicted value of the vehicle motion feature, the parameters of the convolutional neural network 2 are adjusted.

[0041] (13) The feedback network 4 is used to calibrate the convolutional neural network 2 to form a real-time learning network, and the weight values ​​of the motion features in the convolutional neural network 2 are trained, that is, the feedback network 4 is used to adjust the parameters of the convolutional neural network 2.

[0042] (14) The integrated Kalman filter algorithm network judges the accuracy of the driving information predicted by the Markov chain network 1 and the accuracy of the driving information predicted by the Markov chain network 2, that is, the driving information predicted by the Markov chain network 1 is compared with the actual driving information, and the driving information predicted by the Markov chain network 2 is compared with the actual driving information, and the comparison results are input into the feedback learning network 2 and the feedback learning network 3, respectively.

[0043] (15) Feedback learning network 2 adjusts the weight parameters inside Markov chain network 1 according to the input of the Kalman filter algorithm integrated system and the input of prediction network 1, thus completing a learning cycle of Markov chain network 1.

[0044] (16) The output signal of the feedback learning network 2 is input into the feedback learning network 1. Combined with the predicted pedal stepping characteristics, the parameter weight of the predicted pedal action in the convolutional neural network 1 is adjusted to complete a learning cycle of the convolutional neural network 1.

[0045] (17) Feedback learning network 3 adjusts the weight parameters in Markov chain network 2 according to the output results of Markov chain network 2 and the output results of the integrated Kalman algorithm network, completing a learning cycle of Markov chain network 2. At the same time, the output results of feedback network 3 and the output results of feedback learning network 4 can jointly adjust the network weight parameters for extracting spatial features in convolutional neural network 2, completing a learning cycle for spatial feature extraction.

[0046] It can be seen that the technical solution provided by the embodiment of the present invention is that the intelligent safety feedback network combines the neural network with the feedback learning network, so that the entire intelligent safety feedback network has the ability of online learning, realizes the use of small model networks, avoids the error of the generative large model results, and improves the safety of vehicle driving. As the number of times the driver drives increases, the number of times the intelligent safety feedback network learns online also increases. Under the regulation of the feedback learning network, the parameter weights of the entire intelligent safety feedback network will be more accurate than those adjusted by relying solely on mathematical models, which improves the degree of adaptation between the vehicle and the driver and improves the safety of the vehicle.

[0047] Next, the misoperation detection network will be explained.

[0048] Figure 2 This is a schematic diagram of the overall execution process of the intelligent safety feedback network and the misoperation detection network. Figure 2 As shown, driving plan prediction, integrated Kalman filter network (corresponding to Figure 1 The integrated Kalman algorithm network in the Figure 1 The driving information output by the intelligent safety feedback network described in the above. That is, the misoperation detection network (corresponding to Figure 2 The input of the intelligent safety feedback network (misoperation detection) includes the driving information output by the intelligent safety feedback network, and also includes the data input by the driver monitoring system DMS, wherein the data input by the DMS includes the driver's expression and heart rate, etc. Of course, this is only explained in the form of an example and is not specifically limited here.

[0049] Below Figure 2 Elaborate in detail.

[0050] (1) Pedal drive, pedal braking, vehicle speed, acceleration, and gear shifting are input into the intelligent safety feedback network, which then predicts the driving plan, i.e., the driver’s current driving mode, which can be aggressive or conservative, as an input to the misoperation detection network.

[0051] (2) The operating design domain of the driving plan control output by the driving plan prediction is used as the input signal of the integrated Kalman filter algorithm network, where the operating design domain of the driving plan control may include cities, suburbs, highways, mountain roads, rural areas, and parking lots.

[0052] (3) The video signal is input into the intelligent safety feedback network, the driving information features in the video stream are extracted, and the driving environment is determined to be rural, suburban, urban road, highway, parking lot, etc.

[0053] (4) The judgment result is used as a signal of the integrated Kalman filter algorithm network.

[0054] (5) Through the intelligent safety feedback network, the obstacle information features of the video signal are extracted to obtain the obstacle signal, traffic signal (speed limit sign, traffic light, landmark building, etc.), line or lane line (obstacle on the lane line, obstacle near the lane line, and obstacle moving close to the lane line). The obstacle signal, traffic signal and line information are used as one input of the misoperation detection network.

[0055] (6) The road features, weather environment features, road condition features, etc. in the video signal are extracted through the intelligent safety feedback network as an input to the misoperation detection network.

[0056] (7) The integrated Kalman filter network (i.e. Figure 1 The output signal of the integrated Kalman algorithm network shown in FIG. 1 is used as an input of the misoperation detection network.

[0057] (8) The signal from the driver detection system (including the driver’s facial expression and heart rate, etc.) is input into the misoperation detection network.

[0058] (9) The error operation detection network and the driver detection system determine whether the operation performed by the driver at the current moment in the current driving environment is an error operation.

[0059] (10) Triggering the misoperation protection system. That is, by controlling the pedal drive, pedal brake, acceleration, speed, gear shifting and other operations to achieve the purpose of misoperation protection, that is, to correct the misoperation.

[0060] The misoperation protection system will be described in detail below.

[0061] like Figure 3 As shown, it is a schematic diagram of the misoperation protection system performing misoperation protection.

[0062] The present invention designs a misoperation protection system, the main function of which is to improve the vehicle intelligent safety mechanism. It belongs to the vehicle control part and will control the vehicle speed within a set speed threshold. The driver cannot exceed this range by accelerating or decelerating through the driving pedal, and will control the vehicle acceleration according to the actual situation. Figure 3 The VCU here refers to the vehicle control unit, MCU refers to the motor controller, EPS refers to the electric power steering system, and ABS refers to the anti-lock braking system.

[0063] (1) When the vehicle is driving and the misoperation protection system determines that the vehicle is in an urban environment or a parking lot environment and there is a speed limit signal, acceleration is allowed for a short period of time and the subsequent vehicle acceleration should slowly decrease.

[0064] (2) When the misoperation protection system determines that there is an obstacle ahead and it is a static obstacle, the vehicle speed will be limited to a certain range.

[0065] (3) When the misoperation protection system determines that the vehicle has entered or is about to enter a speed limit area, the acceleration is slowly limited to a specific value in advance.

[0066] (4) When the misoperation protection system determines that there is a dynamic obstacle ahead, it dynamically adjusts the vehicle system speed range and acceleration range according to the speed and acceleration of the dynamic obstacle.

[0067] (5) When the misoperation protection system determines that the vehicle is in an urban, rural, suburban, or parking lot environment, it will limit the vehicle's speed and acceleration limits accordingly.

[0068] (6) When the misoperation protection system determines that the lane line is about to narrow or widen, it dynamically increases or decreases the upper limit of the vehicle speed and vehicle acceleration according to the predicted magnitude of the change.

[0069] (7) When the misoperation protection system determines that the driver's operation behavior is aggressive, it limits the relationship between the pedal change range and the vehicle acceleration.

[0070] (8) When the misoperation protection system determines that the driver's operation behavior is conservative, the relationship between the corresponding gain pedal change amplitude and vehicle acceleration.

[0071] (9) When the misoperation protection system determines that the weather has a significant impact on the friction coefficient of the road and a significant impact on the driver's field of vision, the vehicle speed is limited according to the degree of impact.

[0072] It should be explained that the above (1) to (9) are only examples of the incorrect operation protection process of the incorrect operation protection system. In actual applications, there are many other situations, which will not be listed here one by one.

[0073] After describing the intelligent safety feedback network, the misoperation detection network, and the misoperation protection system, a driver misoperation identification and protection method based on online learning and back-end fusion provided by an embodiment of the present invention will be described in detail below. Figure 4 As shown, the method may include the following steps: S410, during the driving of the vehicle, obtaining the driver's operation data, vehicle status data, and the video stream collected by the vehicle's camera at the current moment.

[0074] The driver operation data may include pedal driving data and pedal braking data, etc. The vehicle status data includes vehicle speed, vehicle acceleration, vehicle automatic cruise status data, and vehicle yaw status data. The video stream may be a video of multiple consecutive moments including the current moment captured by the vehicle's camera.

[0075] S420, inputting the driver operation data, vehicle status data, and video stream into an intelligent safety feedback network for online feedback learning.

[0076] S430: Output the driving information of the vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information.

[0077] Specifically, after obtaining the driver operation data, vehicle status data, and video stream, the driver operation data, vehicle status data, and video stream can be input into the intelligent safety feedback network of online feedback learning. The intelligent safety feedback network outputs the driving information of the vehicle at the next moment and intelligently controls the vehicle based on the driving information.

[0078] The driving information at the next moment may include the driver's driving operation, driving mode, and vehicle driving environment, among which the driving operation may include a driving pedal or a brake pedal; the driving mode may include smoothness or aggressiveness, etc., and the vehicle driving environment may include cities, suburbs, highways, mountain roads, villages, parking lots, etc., and may also include traffic signals, obstacles, lane lines, road conditions, traffic conditions, and weather conditions, etc. It is understandable that the driving information may also include other information, which will not be listed here one by one.

[0079] Among them, the online feedback learning process of the intelligent safety feedback network, such as Figure 5 As shown, the following steps may be included: S510, based on the driver operation data and vehicle state data at the current moment, extract the driving characteristics and pedal stepping characteristics at the current moment, and extract the vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream.

[0080] In practical applications, the intelligent safety feedback network may include two feature extraction networks. The current driver operation data and vehicle status data are input into one of the feature extraction networks to extract the current driving characteristics and pedal stepping characteristics. The video stream is input into another feature extraction network to extract the current vehicle driving environment characteristics and vehicle motion characteristics.

[0081] S520, predicting the pedal depression characteristic prediction value at the next moment based on the pedal depression characteristic at the current moment, predicting the vehicle motion characteristic prediction value at the next moment based on the vehicle motion characteristic at the current moment, and predicting the driving information prediction value at the next moment based on the driving characteristics at the current moment and the vehicle driving environment characteristics.

[0082] S530, generates an online feedback learning strategy based on the error between the pedal depression feature prediction value and the actual value of the pedal depression feature at the next moment, the error between the vehicle motion feature prediction value and the actual value of the vehicle motion feature at the next moment, and the error between the driving information prediction value and the actual value of the driving information at the next moment.

[0083] In one embodiment, the intelligent safety feedback network includes a feature extraction network, a prediction network, and a feedback learning network.

[0084] At this time, based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment, generating an online feedback learning strategy may include the following steps: The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between a pedal stepping feature prediction value and a pedal stepping feature true value at a next moment, and an error between a vehicle motion feature prediction value and a vehicle motion feature true value at a next moment; The feedback learning network determines a second online feedback learning strategy of the prediction network based on the error between the predicted value of the driving information and the true value of the driving information at the next moment; The feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy.

[0085] S540: Perform online feedback learning on the intelligent safety feedback network based on the online feedback learning strategy.

[0086] Specifically, the intelligent safety feedback network includes a feature extraction network and a prediction network. Based on the error between the pedal stepping feature prediction value and the actual value of the pedal stepping feature at the next moment, and the error between the vehicle motion feature prediction value and the actual value of the vehicle motion feature at the next moment, the first online feedback learning strategy determined can realize online feedback learning of the feature extraction network, that is, adjust the model parameters of the feature extraction network. Based on the error between the driving information prediction value and the actual value of the driving information at the next moment, the second online feedback learning strategy determined can realize online feedback learning of the prediction network, that is, adjust the model parameters of the prediction network, so that the features extracted by the feature extraction network are more accurate, and the driving information predicted by the prediction network is more accurate.

[0087] It can be seen that the technical solution provided by the embodiment of the present invention is that the intelligent safety feedback network combines the neural network with the feedback learning network, so that the entire intelligent safety feedback network has the ability of online learning, realizes the use of small model networks, avoids the error of the generative large model results, and improves the safety of vehicle driving. As the number of times the driver drives increases, the number of times the intelligent safety feedback network learns online also increases. Under the regulation of the feedback learning network, the parameter weights of the entire intelligent safety feedback network will be more accurate than those adjusted by relying solely on mathematical models, which improves the degree of adaptation between the vehicle and the driver and improves the safety of the vehicle.

[0088] As an implementation method of the embodiment of the present invention, the feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork, and the feedback learning network includes a first feedback learning subnetwork and a second feedback learning subnetwork. The first feature extraction subnetwork corresponds to Figure 1 The convolutional neural network in the first and second feature extraction sub-networks correspond to Figure 1 Convolutional neural network 2 in the first feedback learning sub-network corresponds to Figure 1 The feedback learning network 1 in the second feedback learning sub-network corresponds to Figure 1 Feedback learning network in IV.

[0089] At this time, the feedback learning network determines the first online feedback learning strategy of the feature extraction network based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, and the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, which may include the following steps: The first feedback learning sub-network generates a first online feedback learning sub-strategy for the first feature extraction sub-network based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment.

[0090] Among them, the first feature extraction sub-network is used to extract pedal stepping features and driving features; The second feedback learning sub-network generates a second online feedback learning sub-strategy for the second feature extraction sub-network based on the error between the predicted value of the vehicle motion feature and the true value of the vehicle motion feature at the next moment.

[0091] Among them, the second feature extraction sub-network is used to extract vehicle motion features and vehicle driving environment features.

[0092] It should be noted that in Figure 1 This part has been explained in the embodiment and will not be repeated here.

[0093] As an implementation method of an embodiment of the present invention, the prediction network includes a first prediction sub-network and a second prediction sub-network, and the feedback learning network also includes a third feedback learning sub-network and a fourth feedback learning sub-network. Figure 1 The Markov chain network 1 in the second prediction subnetwork corresponds to Figure 1 The Markov chain network 2 in the third feedback learning sub-network corresponds to Figure 1 The feedback learning network 2 in the fourth feedback learning sub-network corresponds to Figure 2 Feedback learning network in III.

[0094] At this time, the feedback learning network determines the second online feedback learning strategy of the prediction network based on the error between the predicted value of the driving information and the true value of the driving information at the next moment, which may include the following steps, namely step 1 and step 2: Step 1: The third feedback learning subnetwork determines a third online feedback learning sub-strategy for the first prediction subnetwork and the first feature extraction subnetwork based on the error between the driving information prediction value predicted by the first prediction subnetwork and the actual value of the driving information at the next moment.

[0095] The first prediction subnetwork predicts the driving information prediction value at the next moment based on the driving characteristics at the current moment; Step 2: The fourth feedback learning subnetwork determines the fourth online feedback learning sub-strategy for the second prediction subnetwork and the second feature extraction subnetwork by calculating the error between the driving information prediction value obtained by the second prediction subnetwork and the actual value of the driving information at the next moment.

[0096] Among them, the second prediction subnetwork predicts the driving information prediction value at the next moment based on the vehicle driving environment characteristics at the current moment.

[0097] As an implementation of an embodiment of the present invention, the feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy, which may include the following steps: In the first step, the first feature extraction sub-network adjusts model parameters based on the first online feedback learning sub-strategy and the third online feedback learning sub-strategy.

[0098] In the second step, the second feature extraction sub-network adjusts model parameters based on the second online feedback learning sub-strategy and the fourth online feedback learning sub-strategy.

[0099] In the third step, the first prediction sub-network adjusts the model parameters based on the third online feedback learning sub-strategy.

[0100] In the fourth step, the second prediction sub-network adjusts the model parameters based on the fourth online feedback learning sub-strategy.

[0101] Specifically, the first prediction subnetwork predicts driving information based on the driving features extracted by the first feature extraction subnetwork, so the first feature extraction subnetwork can adjust model parameters through the first online feedback learning substrategy and the third online feedback learning substrategy. Similarly, the second prediction subnetwork predicts driving information based on the vehicle driving environment features extracted by the first feature extraction subnetwork, so the second feature extraction subnetwork can adjust model parameters through the second online feedback learning substrategy and the fourth online feedback learning substrategy.

[0102] On the basis of the above embodiment, as an implementation manner of the embodiment of the present invention, the method may further include the following steps: The first step is to obtain driver status information from the driver monitoring system.

[0103] The driver monitoring system is Figure 2 In the DMS of the illustrated embodiment, the driver status information may include information such as the driver's facial expression and heart rate.

[0104] In the second step, the driver status information and the driving information output from the intelligent safety feedback network are input into the misoperation detection network.

[0105] The third step is to determine whether the vehicle's operation in the current driving environment is an erroneous operation through the erroneous operation detection network and the driver monitoring system.

[0106] The fourth step is, if it is determined that the operation of the vehicle driving in the current driving environment is an erroneous operation, the erroneous operation protection system is triggered to execute the erroneous operation protection strategy, and the vehicle is controlled through the operation protection strategy.

[0107] Specifically, after obtaining the driver status information, the driver status information and driving information can be input into the misoperation detection network. The misoperation detection network and the driver monitoring system can be used to determine whether there is any misoperation. If there is any misoperation, it means that there is a safety hazard in the vehicle. Then, the misoperation protection system is triggered to execute the misoperation protection strategy, and the vehicle is controlled through the operation protection strategy.

[0108] As an implementation of the embodiment of the present invention, triggering the misoperation protection system to execute the misoperation protection strategy and controlling the vehicle through the operation protection strategy may include the following two steps: Step 1: Determine the acceleration range and speed range of the vehicle based on the current driving environment and the driver's driving mode.

[0109] Step 2: Control the acceleration of the vehicle within an acceleration range, and control the speed of the vehicle within a speed range.

[0110] Specifically, the triggering of the misoperation protection system can control the speed and acceleration of the vehicle according to the current driving environment and the driver's driving mode. The acceleration range and speed range can be determined according to actual conditions and are not specifically limited here.

[0111] In addition, how to control the speed and acceleration of the vehicle has been Figure 3 (1) to (9) in the illustrated embodiment are described in detail with examples and will not be elaborated here.

[0112] The embodiment of the present invention also provides a driver misoperation recognition and protection device 60 based on online learning and back-end fusion, such as Figure 6 As shown, the device comprises: The data acquisition module 610 is used to acquire the driver's operation data, vehicle status data, and the video stream collected by the camera of the vehicle at the current moment during the vehicle's driving process; A data input module 620, for inputting the driver operation data, the vehicle status data, and the video stream into an intelligent safety feedback network for online feedback learning; A driving information prediction module 630, configured to output the driving information of the vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information; The online learning feedback module 640 is specifically used for: Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream; Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature; Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment; The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

[0113] In a third aspect, an embodiment of the present invention provides an electronic device 700, such as Figure 7 As shown, including: at least one processor 701; a memory 702 for storing the at least one processor executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in the first aspect.

[0114] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect.

[0115] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0116] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A driver misoperation identification and protection method based on online learning and back-end fusion, characterized in that: The method comprises: During the driving process of the vehicle, the driver's operation data, vehicle status data, and the video stream collected by the camera of the vehicle are obtained at the current moment; Inputting the driver operation data, the vehicle status data, and the video stream into an intelligent safety feedback network for online feedback learning; Outputting driving information of the vehicle at the next moment through the intelligent safety feedback network, and controlling the vehicle based on the driving information; The online feedback learning process of the intelligent safety feedback network includes: Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream; Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature; Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment; The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

2. The method according to claim 1, characterized in that: The intelligent safety feedback network includes a feature extraction network, a prediction network and a feedback learning network; The generating of the online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment comprises: The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between the pedal stepping feature prediction value and a pedal stepping feature true value at a next moment, and an error between the vehicle motion feature prediction value and a vehicle motion feature true value at a next moment; The feedback learning network determines a second online feedback learning strategy of the prediction network based on an error between the predicted value of the driving information and a true value of the driving information at a next moment; The feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy.

3. The method according to claim 2, characterized in that The feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork, and the feedback learning network includes a first feedback learning subnetwork and a second feedback learning subnetwork; The feedback learning network determines a first online feedback learning strategy of the feature extraction network based on an error between the pedal stepping feature prediction value and a pedal stepping feature true value at the next moment, and an error between the vehicle motion feature prediction value and a vehicle motion feature true value at the next moment, including: The first feedback learning sub-network generates a first online feedback learning sub-strategy for a first feature extraction sub-network based on an error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, wherein the first feature extraction sub-network is used to extract the pedal stepping feature and the driving feature; The second feedback learning sub-network generates a second online feedback learning sub-strategy for a second feature extraction sub-network based on the error between the predicted value of the vehicle motion feature and the actual value of the vehicle motion feature at the next moment, and the second feature extraction sub-network is used to extract vehicle motion features and vehicle driving environment features.

4. The method according to claim 3, characterized in that The prediction network includes a first prediction sub-network and a second prediction sub-network, and the feedback learning network also includes a third feedback learning sub-network and a fourth feedback learning sub-network; The feedback learning network determines a second online feedback learning strategy of the prediction network based on an error between the predicted value of the driving information and the actual value of the driving information at the next moment, including: The third feedback learning subnetwork determines a third online feedback learning sub-strategy for the first prediction subnetwork and the first feature extraction subnetwork based on an error between a driving information prediction value predicted by the first prediction subnetwork and a true driving information value at the next moment, wherein the first prediction subnetwork predicts a driving information prediction value at the next moment based on the driving features at the current moment; The fourth feedback learning subnetwork determines a fourth online feedback learning sub-strategy for the second prediction subnetwork and the second feature extraction subnetwork based on the error between the driving information prediction value predicted by the second prediction subnetwork and the actual value of the driving information at the next moment, wherein the second prediction subnetwork predicts the driving information prediction value at the next moment based on the vehicle driving environment characteristics at the current moment.

5. The method according to claim 4, characterized in that The feature extraction network performs online feedback learning based on the first online feedback learning strategy and the second online feedback learning strategy, and the prediction network performs online feedback learning based on the second online feedback learning strategy, including: The first feature extraction sub-network adjusts model parameters based on the first online feedback learning sub-strategy and the third online feedback learning sub-strategy; The second feature extraction sub-network adjusts model parameters based on the second online feedback learning sub-strategy and the fourth online feedback learning sub-strategy; The first prediction sub-network adjusts model parameters based on the third online feedback learning sub-strategy; The second prediction sub-network adjusts model parameters based on the fourth online feedback learning sub-strategy.

6. The method according to any one of claims 1 to 5, characterized in that: The controlling the vehicle based on the driving information includes: obtaining driver status information from a driver monitoring system; Inputting the driver status information and the driving information output from the intelligent safety feedback network into a misoperation detection network; Determining whether the operation of the vehicle driving in the current driving environment is an erroneous operation by the erroneous operation detection network and the driver monitoring system; If it is determined that the operation of the vehicle driving in the current driving environment is an erroneous operation, the erroneous operation protection system is triggered to execute the erroneous operation protection strategy, and the vehicle is controlled by the operation protection strategy.

7. The method according to claim 6, characterized in that The triggering of the misoperation protection system to execute the misoperation protection strategy and controlling the vehicle through the operation protection strategy includes: Determine the acceleration range and speed range of the vehicle based on the current driving environment and the driver's driving mode; The acceleration of the vehicle is controlled within the acceleration range, and the speed of the vehicle is controlled within the speed range.

8. A driver misoperation identification and protection device based on online learning and back-end fusion, characterized in that: The device comprises: A data acquisition module is used to acquire the driver's operation data, vehicle status data, and the video stream collected by the camera of the vehicle at the current moment during the vehicle's driving process; A data input module, used for inputting the driver operation data, the vehicle status data, and the video stream into an intelligent safety feedback network for online feedback learning; A driving information prediction module, used to output the driving information of the vehicle at the next moment through the intelligent safety feedback network, and control the vehicle based on the driving information; Online learning feedback module, specifically for: Extracting driving characteristics and pedal stepping characteristics at the current moment based on the driver operation data and vehicle state data, and extracting vehicle driving environment characteristics and vehicle motion characteristics at the current moment based on the video stream; Predicting a predicted value of a pedal stepping feature at a next moment based on a pedal stepping feature at a current moment, predicting a predicted value of a vehicle motion feature at a next moment based on a vehicle motion feature at a current moment, and predicting a predicted value of driving information at a next moment based on a driving feature at a current moment and a vehicle driving environment feature; Generate an online feedback learning strategy based on the error between the pedal stepping feature prediction value and the pedal stepping feature true value at the next moment, the error between the vehicle motion feature prediction value and the vehicle motion feature true value at the next moment, and the error between the driving information prediction value and the driving information true value at the next moment; The intelligent safety feedback network is subjected to online feedback learning based on the online feedback learning strategy.

9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 7.

Images