Abnormal driving behavior recognition method and device based on real-time data online training

Abstract

The invention relates to the technical field of abnormal driving behavior detection, in particular to an abnormal driving behavior recognition method and device based on real-time data online training, and the method comprises the steps: obtaining initial multi-modal data; training a pre-constructed deep learning model to obtain a current normal driving behavior model, an abnormal judgment threshold and a driver basic personalized file; acquiring real-time multi-modal data, and performing abnormal behavior monitoring on the real-time multi-modal data by using the current normal driving behavior model to obtain an abnormal score; under the condition that the exception score does not exceed an exception judgment threshold value, continuously obtaining multi-moment multi-modal data based on an active learning strategy, performing online incremental training on the model by utilizing the two data to obtain an updated model and an exception judgment threshold value, and synchronously updating the personalized file; otherwise, the abnormal driving behavior is judged, and an alarm signal is triggered. Therefore, the problems that an existing anomaly recognition method lacks individual adaptability and is difficult to deal with dynamic changes and the like are solved.

Description

Technical Field

[0001] This invention relates to the field of abnormal driving behavior detection technology, and in particular to an abnormal driving behavior recognition method and apparatus based on real-time data online training. Background Technology

[0002] Road traffic accidents have become a major threat to residents' lives. Studies have found that while accidents caused by poor driving skills, lack of experience, vehicle malfunctions, or environmental factors are sometimes unavoidable, many more accidents are caused by the driver's intentional or unintentional actions, such as speeding, tailgating, not wearing a seatbelt, and fatigued or inattentive driving. These accidents can be prevented with timely warnings. Therefore, there is an urgent need to develop a method that can accurately identify abnormal driving behavior, provide timely warnings, and offer drivers accurate, professional, and personalized driving guidance to ensure road traffic safety.

[0003] Most existing abnormal driving behavior detection methods use offline general models, which cannot be customized or updated, resulting in low detection accuracy. For example, South China University of Technology has disclosed a method and system for identifying and warning of abnormal driving behavior on highways. This method collects vehicle trajectory information and lane boundary line information to calculate vehicle deviation distance and trajectory degree, and calculates an abnormal driving judgment threshold based on the current driving state. The threshold is then used to determine whether the vehicle is driving abnormally. Huaneng Shaanxi Jingbian Power Co., Ltd. has disclosed a deep learning-based method for identifying abnormal driving behavior. The main steps of this method include: acquiring historical driving data and station environment data to construct an offline training dataset; extracting first-type temporal features representing abnormal driving behavior and second-type temporal features representing abnormal scenarios; establishing an abnormal driving recognition model based on neural network learning; and inputting real-time collected data into the model to output the abnormal driving behavior recognition result. Chongqing University of Posts and Telecommunications has disclosed a multimodal method and system for judging abnormal driving behavior, which simultaneously monitors driver state, vehicle state, and environmental state, and integrates multimodal data to construct a multi-index dynamic threshold judgment and recognition algorithm based on logistic regression, thereby realizing the judgment of abnormal driving behavior.

[0004] However, the limitations of the above identification methods are: (1) The model is static and lacks individual adaptability: Most existing methods rely on the limited historical data collected in the offline stage to train static models. Such models are essentially an "average" expression of group behavior, and it is difficult to model the individualized habits (such as lane changing style, following distance preference) and physiological differences of different drivers. When applied to specific drivers, a large number of false alarms are easily generated due to individual differences, which leads to a decrease in the credibility of the warning system and even being disabled by users. (2) The environmental generalization ability is weak and it is difficult to cope with dynamic changes: Driving behavior is highly coupled with environmental factors such as road type, traffic density, and weather conditions. The performance of existing static models will significantly degrade in new scenarios not covered by training data (such as special road conditions never encountered before, extreme weather). More importantly, it is impossible to use the real-time data stream continuously generated by the vehicle in actual driving to self-adjust and optimize, and it is impossible to adapt to the slow evolution of driver behavior over time (such as the improvement of driving skills or changes in habits). (3) The deployment is rigid and cannot realize flexible services centered on drivers: In order to achieve high-precision recognition, the existing solutions are often bound to specific vehicles and cannot be used with the driver to different vehicles. When the driver changes vehicles, the system needs to be reconfigured, resulting in low equipment reuse rate, high overall investment cost, and lack of flexible adaptation capability across vehicles and users. Summary of the Invention

[0005] This invention provides a method and apparatus for identifying abnormal driving behavior based on real-time data online training, in order to solve the problems of existing abnormal identification methods and models being static and fixed, lacking individual adaptability, having weak environmental generalization ability, being unable to cope with dynamic changes, having rigid deployment, and being unable to achieve flexible services centered on the driver.

[0006] A first aspect of this invention provides a method for identifying abnormal driving behavior based on real-time data online training, comprising the following steps: Acquire initial multimodal data of the driver operating the current vehicle; The pre-built deep learning model is trained using the initial multimodal data to construct a model of current normal driving behavior; Based on the current normal driving behavior model, historical behavior patterns, personalized thresholds or feature preferences are calculated to construct a basic personalized profile of the driver and anomaly judgment thresholds; Acquire real-time multimodal data of the driver driving the current vehicle, and use the current normal driving behavior model to monitor abnormal behavior in the real-time multimodal data to obtain an anomaly score; The abnormal score is compared with the abnormal judgment threshold. If the abnormal score does not exceed the abnormal judgment threshold, multi-moment multimodal data is continuously acquired based on an active learning strategy. The real-time multimodal data and the multi-moment multimodal data are used to perform online incremental training on the current normal driving behavior model to obtain an updated normal driving behavior model. The driver's basic personalized profile and the abnormal judgment threshold are updated simultaneously. Otherwise, it is determined to be abnormal driving behavior and an alarm signal is triggered.

[0007] Optionally, training a pre-built deep learning model using the initial multimodal data to construct a current normal driving behavior model includes: The initial multimodal data is preprocessed to obtain an abnormal driving behavior feature set; The deep learning model is trained using the abnormal driving behavior feature set to construct the current normal driving behavior model.

[0008] Optionally, if the anomaly score does not exceed the anomaly judgment threshold, the driver continuously acquires multi-moment multimodal data based on an active learning strategy to perform online incremental training on the current normal driving behavior model using the real-time multimodal data and the multi-moment multimodal data, thereby obtaining an updated normal driving behavior model and anomaly judgment threshold, and simultaneously updating the driver's basic personalized profile, including: If the anomaly score does not exceed the anomaly judgment threshold, the real-time multimodal data is saved as anomaly-free data. Based on an active learning strategy, the driver continuously acquires real-time multimodal data of driving the current vehicle. When the real-time multimodal data meets the preset training conditions, the current normal driving behavior model is incrementally trained online using the real-time multimodal data and the abnormal data based on the catastrophic forgetting incremental learning algorithm to obtain the updated normal driving behavior model. The updated normal driving behavior model is used to update the driver's basic personalized profile to obtain the updated driver's basic personalized profile and the updated anomaly judgment threshold.

[0009] Optionally, the anomaly detection threshold is dynamically adjusted based on the updated driver basic personalized profile.

[0010] Optionally, it also includes: When the driver drives a new vehicle, the driver's basic personal profile is retrieved; By identifying the parameters of the new vehicle and fine-tuning the short-term data, the driver's basic personalized profile is adapted to the new vehicle.

[0011] A second aspect of the present invention provides an abnormal driving behavior recognition device based on real-time data online training, comprising: The acquisition module is used to acquire the initial multimodal data of the driver driving the current vehicle; The training module is used to train a pre-built deep learning model using the initial multimodal data to construct a model of current normal driving behavior. A profile building module is used to calculate historical behavior patterns, personalized thresholds, or feature preferences based on the current normal driving behavior model, so as to build a basic personalized profile of the driver and anomaly judgment thresholds. An anomaly detection module is used to acquire real-time multimodal data of the driver driving the current vehicle, and to use the current normal driving behavior model to detect abnormal behavior in the real-time multimodal data in order to obtain an anomaly score. The training and alarm module is used to compare the anomaly score with the anomaly judgment threshold. If the anomaly score does not exceed the anomaly judgment threshold, it continuously acquires multi-moment multimodal data based on an active learning strategy. It then uses the real-time multimodal data and the multi-moment multimodal data to perform online incremental training on the current normal driving behavior model to obtain an updated normal driving behavior model. Simultaneously, it updates the driver's basic personalized profile and the anomaly judgment threshold. Otherwise, it is determined to be abnormal driving behavior, and an alarm signal is triggered.

[0012] Optionally, the training module includes: A preprocessing unit is used to preprocess the initial multimodal data to obtain an abnormal driving behavior feature set; An initial training unit is used to train the deep learning model using the abnormal driving behavior feature set to construct the current normal driving behavior model.

[0013] Optionally, the training and alarm module includes: The storage unit is used to save the real-time multimodal data as data without anomalies when the anomaly score does not exceed the anomaly judgment threshold. The continuous acquisition unit is used to continuously acquire real-time multimodal data of the driver driving the current vehicle based on an active learning strategy; The training unit is used to perform online incremental training on the current normal driving behavior model based on the real-time multimodal data and the abnormal data, using an incremental learning algorithm that prevents catastrophic forgetting, when the real-time multimodal data meets the preset training conditions, so as to obtain the updated normal driving behavior model. The update unit is used to update the driver's basic personalized profile using the updated normal driving behavior model, so as to obtain the updated driver's basic personalized profile and the updated anomaly judgment threshold.

[0014] Optionally, the anomaly detection threshold is dynamically adjusted based on the updated driver basic personalized profile.

[0015] Optionally, it also includes: The retrieval module is used to retrieve the driver's basic personal profile when the driver drives a new vehicle; An adaptation module is used to adapt the driver's basic personalized profile to the new vehicle by identifying the parameters of the new vehicle and making short-term data fine-tuning.

[0016] The abnormal driving behavior recognition method and device based on real-time data online training proposed in this invention can directly build a normal model of the driver on a portable edge computing device, and use real-time generated driving data for continuous and efficient online monitoring and learning updates, realizing dynamic evolution of model performance and deep adaptation to individual drivers, and supporting rapid scenario migration across vehicles, effectively improving the accuracy, robustness and long-term user experience and practical value of abnormal driving behavior recognition.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating an abnormal driving behavior recognition method based on real-time data online training according to an embodiment of the present invention; Figure 2 This is an overall execution framework diagram of an abnormal driving behavior recognition method based on real-time data online training according to an embodiment of the present invention; Figure 3 This is an execution framework diagram for multimodal abnormal driving behavior recognition provided according to an embodiment of the present invention; Figure 4This is an execution framework diagram of a personalized adaptation mechanism provided according to an embodiment of the present invention; Figure 5 This is a block diagram illustrating an abnormal driving behavior recognition device based on real-time data online training, according to an embodiment of the present invention.

[0019] Explanation of reference numerals in the attached figures: 50 - Abnormal driving behavior recognition device based on real-time data online training; 501 - Initial acquisition module; 502 - Initial training module; 503 - File construction module; 504 - Training module; 505 - File update module; 506 - Abnormal judgment module. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0021] The following description, with reference to the accompanying drawings, describes an embodiment of the abnormal driving behavior recognition method and apparatus based on real-time data online training.

[0022] Figure 1 This is a flowchart illustrating an abnormal driving behavior recognition method based on real-time data online training, as provided in an embodiment of the present invention.

[0023] like Figure 1 As shown, this abnormal driving behavior recognition method based on real-time data online training includes the following steps: In step S101, the initial multimodal data of the driver driving the current vehicle is obtained.

[0024] In actual implementation, embodiments of the present invention utilize multimodal sensors, including but not limited to a miniature camera facing the driver and a nine-axis inertial measurement unit, and are equipped with a standard vehicle diagnostic interface to draw power and read vehicle bus data. They also use low-invasive methods such as smart wearable watches to collect driver electrocardiogram signal data, use built-in sensors to collect vehicle three-axis acceleration data, and use a camera placed in front of the driver to continuously collect driver facial data 24 hours a day.

[0025] In step S102, the pre-built deep learning model is trained using the initial multimodal data to construct a model of current normal driving behavior.

[0026] In some embodiments, a pre-built deep learning model is trained using initial multimodal data to construct a model of current normal driving behavior, including: The initial multimodal data is preprocessed to obtain a feature set of abnormal driving behaviors; A deep learning model based on a set of abnormal driving behavior features is trained to construct a model of current normal driving behavior.

[0027] In actual implementation, the embodiments of the present invention preprocess and perform feature extraction and filtering on the initial multimodal data based on the significant differences of each modality in terms of sampling frequency, noise characteristics, semantic structure, etc., so as to effectively integrate the data of different modalities and form a multimodal abnormal driving behavior feature set.

[0028] Furthermore, a pre-built deep learning network is trained using a set of abnormal driving behavior features to learn stable behavior patterns inherent in different modalities, thereby constructing a personalized model of the driver's current normal driving behavior. The deep learning network includes, but is not limited to, various neural network architectures such as the transformer model and their variants.

[0029] In step S103, the historical behavior patterns, personalized thresholds or feature preferences are calculated using the current normal driving behavior model to construct a basic personalized profile of the driver.

[0030] In actual implementation, historical behavior patterns, personalized thresholds, or feature preferences are calculated using the current normal driving behavior model and recorded in the personalized profile to form a basic personalized profile of the driver. This basic personalized profile of the driver is associated with the driver's unique identity. The profile can be initialized to empty or through a preset short observation and learning period. Initial driving data is used to establish the driver's preliminary behavioral characteristic baseline, providing a benchmark reference for subsequent online personalized adaptation.

[0031] In step S104, real-time multimodal data of the driver driving the current vehicle is acquired, and abnormal behavior is monitored on the real-time multimodal data using the current normal driving behavior model to obtain an abnormal score.

[0032] In actual implementation, such as Figure 2 and 3 As shown, real-time multimodal data of the driver driving the current vehicle is obtained, and multimodal features in the real-time multimodal data are extracted using the current normal driving behavior model. The data is then compared and similarity is calculated in real time with the current normal driving behavior model to obtain an anomaly score.

[0033] Furthermore, in addition to real-time anomaly identification, this embodiment of the invention also performs parallel computation of the confidence index and uncertainty measure (such as entropy, variance, or output of a dedicated uncertainty estimation network) of the model output results. Based on these indicators, an active learning strategy can be implemented: for sample data exhibiting low confidence, high uncertainty, or predicted probabilities close to the classification boundary, manual verification of the true label can be initiated through a simplified human-computer interaction channel (e.g., a short confirmation request pushed by a mobile application bound to the device); for predicted samples with high confidence, self-supervised learning techniques (including but not limited to: methods based on consistency regularization) are used to automatically generate high-quality pseudo-labels. The "data-label" sample pairs constructed in this way will be stored in a fixed-capacity first-in-first-out circular cache queue for subsequent model optimization.

[0034] In step S105, the abnormal score is compared with the abnormal judgment threshold. If the abnormal score does not exceed the abnormal judgment threshold, multi-moment multimodal data is continuously acquired based on the active learning strategy. The current normal driving behavior model is then incrementally trained online using real-time multimodal data and multi-moment multimodal data to obtain an updated normal driving behavior model. The driver's basic personalized profile and abnormal judgment threshold are updated simultaneously. Otherwise, the abnormal driving behavior is determined and an alarm signal is triggered.

[0035] In some embodiments, if the abnormal score does not exceed the abnormality judgment threshold, the real-time multimodal data is saved as abnormal data. Based on an active learning strategy, real-time multimodal data of the driver driving the current vehicle is continuously acquired; When real-time multimodal data meets the preset training conditions, an incremental learning algorithm based on catastrophic forgetting is used to conduct online incremental training on the current normal driving behavior model using real-time multimodal data and abnormal data to obtain an updated normal driving behavior model. The driver's basic personalized profile is updated using the updated normal driving behavior model to obtain the updated driver's basic personalized profile and the updated anomaly judgment threshold.

[0036] In actual implementation, such as Figure 2 and 3As shown, the anomaly score is compared with the anomaly judgment threshold. If the anomaly score does not exceed the anomaly judgment threshold, the real-time multimodal data is saved as anomaly-free data. Based on the active learning strategy, real-time multimodal data of the driver driving the current vehicle is continuously acquired. The real-time multimodal data is preprocessed and processed by feature extraction and filtering to effectively integrate data from different modalities and form online abnormal driving behavior training samples. When the real-time multimodal data reaches the preset training conditions, if a new behavior pattern appears or the preset judgment threshold is reached, an incremental learning algorithm based on catastrophic forgetting prevention is used to input the training samples and anomaly-free data into the current normal driving behavior model for online learning to continuously extract the multimodal features of the current driving behavior and build an updated normal driving behavior model.

[0037] After obtaining the updated normal driving behavior model, new historical behavior patterns, new personalized thresholds, or new feature preferences are calculated using the updated normal driving behavior model and recorded in the personalized profile to dynamically update the driver's basic personalized profile. That is, in this embodiment of the invention, the new behavior patterns and optimized judgment thresholds learned in this training will be synchronously updated to the driver's personalized model profile, thereby achieving continuous evolution of the driver's behavior baseline data and dynamic thresholds.

[0038] In the process of real-time abnormal driving behavior recognition, the front-end multimodal fusion features are used as input, and the behavior baseline in the personalized profile is called simultaneously for joint decision-making and dynamic threshold adjustment to obtain a new preset abnormal judgment threshold, thereby realizing individualized and accurate judgment of abnormal behavior.

[0039] If the abnormal score exceeds the abnormal judgment threshold, it is judged as abnormal driving behavior and a graded warning is triggered. The device sends a graded warning signal to the driver through the local interaction unit integrated with the device (such as, but not limited to, a voice synthesis module, a multi-mode light indicator or a vibration motor) to complete real-time intervention and generate a personalized risk assessment result for the current driver.

[0040] Specifically, in this embodiment of the invention, an active learning strategy is used to continuously monitor information such as the driver's current vehicle's computing load, power status, and vehicle operating status, forming a circular cache queue. When a preset "idle training" condition is detected (such as the vehicle being turned off and stationary and connected to an external power source, and the utilization rate of the main computing unit being lower than a threshold), an online learning task is automatically triggered. Based on an incremental learning algorithm that prevents catastrophic forgetting, a batch of training data is sampled from the circular cache queue, and a lightweight training loop is started to continuously extract multimodal features of the current driving behavior and construct a normal driving behavior model.

[0041] It should be noted that incremental learning algorithms to prevent catastrophic forgetting include, but are not limited to: elastic weight consolidation and its variants, and experience-based replay methods, which incrementally update some parameters of the base model based on gradients obtained from online training using continuously accumulating real-time data, and simultaneously update the driver behavior baseline data in the personalized model profile. Furthermore, when applying gradient updates, this algorithm balances the learning of new knowledge with the retention of previously important knowledge by introducing constraints on historically important parameters of the deep learning model into the loss function. This embodiment of the invention utilizes data sampled from a cache queue to calculate gradients and incrementally update some parameters in the base model responsible for feature fusion and decision-making.

[0042] In some embodiments, it also includes: When a driver drives a new vehicle, the driver's basic personalized profile is retrieved. By identifying parameters of the new vehicle and fine-tuning short-term data, the driver's basic personalized profile is adapted to the new vehicle.

[0043] In practical implementation, the abnormal driving behavior recognition method based on real-time data online training proposed in this invention can be implemented as a plug-and-play, easy-to-install and disassemble independent hardware carrier. When the independent hardware carrier is detected to be installed in a new vehicle, a fast adaptive process will be automatically executed. Through vehicle parameter recognition and short-term data fine-tuning, the driver's personalized profile and the dynamic characteristics of the new vehicle will be rapidly integrated and adapted.

[0044] Specifically, such as Figure 4 As shown, the unique identification information of the new vehicle is read through the vehicle's standard diagnostic interface, and the part of the basic model responsible for front-end feature perception (e.g., the first network layer that processes raw images or signals) is rapidly fine-tuned in a supervised or self-supervised manner to adapt to changes in feature distribution that may be caused by the installation position and viewing angle of the sensors inside the new vehicle and the vehicle's dynamic response characteristics. Subsequently, the new vehicle-specific compensation parameters are intelligently fused with the driver's personalized model profile, enabling the high-precision personalized recognition service to be restored and continued after a very short driving adaptation period, achieving seamless migration of personalized services.

[0045] For example, a plug-and-play, easy-to-install and disassemble standalone hardware carrier can integrate multimodal sensing, edge AI computing, vehicle interface, and communication modules into a portable edge computing device. Its core hardware adopts a modular design, with a high-performance, low-power edge computing unit (such as the NVIDIA Jetson series module) as the main controller. In the data acquisition path, the device uses its integrated wide-angle camera module and nine-axis inertial measurement unit (IMU) to simultaneously acquire driver facial video streams and vehicle three-axis acceleration data. It also uses the communication module to acquire electrocardiogram signals collected by a smart wearable device. The edge computing unit then performs the entire computation process for the aforementioned model. This device achieves a closed loop, integrating data perception, real-time inference, online training, and voice / light warning functions.

[0046] In summary, the abnormal driving behavior recognition method based on real-time data online training proposed in this embodiment of the invention has the following beneficial effects: (1) The present invention proposes an abnormal driving behavior identification method that integrates multimodal data fusion analysis of driver and vehicle status, which effectively utilizes driver's personalized features and multimodal information, and can improve the accuracy of abnormal driving behavior identification; (2) The present invention proposes an abnormal driving behavior recognition method and system based on real-time data online incremental training. It can continuously update and optimize the model according to the driver's real-time driving data, so that the abnormal behavior recognition model is more and more in line with the driver's driving habits and personality characteristics, thus becoming a personal driving assistant. (3) The embodiments of the present invention propose a complete integrated portable device solution, build a framework for effectively utilizing real-time data for online incremental training, provide a clear and explicit guidance scheme for rapid implementation, and help to accurately identify abnormal driving and ensure driving safety; (4) The present invention proposes a method that can widely identify abnormal driving behaviors, not limited to specific abnormal driving behaviors such as fatigued driving, aggressive driving, and drunk driving. It does not require additional abnormal behavior labels. It uses a deep learning model to self-supervisedly mine the implicit distribution in multimodal driving data. Once the real-time data deviates from the personalized normal model, it is judged as abnormal, thus realizing zero-label, high generalization, and low-computing-power online anomaly detection.

[0047] Next, referring to the accompanying drawings, an abnormal driving behavior recognition device based on real-time data online training according to an embodiment of the present invention is described.

[0048] Figure 5 This is a block diagram illustrating an abnormal driving behavior recognition device based on real-time data online training, as provided in an embodiment of the present invention.

[0049] like Figure 5As shown, the abnormal driving behavior recognition device 50 based on real-time data online training includes: an acquisition module 501, a training module 502, a profile construction module 503, an abnormal monitoring module 504, and a training and alarm module 505.

[0050] The system comprises the following modules: Acquisition module 501 acquires initial multimodal data of the driver driving the current vehicle. Training module 502 trains a pre-built deep learning model using the initial multimodal data to construct a current normal driving behavior model. Profile construction module 503 calculates historical behavior patterns, personalized thresholds, or feature preferences based on the current normal driving behavior model to construct a basic personalized profile and anomaly judgment threshold for the driver. Anomaly monitoring module 504 acquires real-time multimodal data of the driver driving the current vehicle and uses the current normal driving behavior model to monitor abnormal behavior in the real-time multimodal data to obtain an anomaly score. Training and alarm module 505 compares the anomaly score with the anomaly judgment threshold. If the anomaly score does not exceed the anomaly judgment threshold, it continuously acquires multimodal data at multiple time points based on an active learning strategy to perform online incremental training of the current normal driving behavior model using real-time and multimodal data to obtain an updated normal driving behavior model, and simultaneously updates the driver's basic personalized profile and anomaly judgment threshold; otherwise, it determines abnormal driving behavior and triggers an alarm signal.

[0051] In some embodiments, the training module 502 includes: The preprocessing unit is used to preprocess the initial multimodal data to obtain a feature set of abnormal driving behavior. The initial training unit is used to train a deep learning model using a set of abnormal driving behavior features to build a model of current normal driving behavior.

[0052] In some embodiments, the optimization module 504 includes: The storage unit is used to save real-time multimodal data as data without anomalies when the anomaly score does not exceed the anomaly judgment threshold. The continuous acquisition unit is used to continuously acquire real-time multimodal data of the driver driving the current vehicle based on an active learning strategy; The training unit is used to perform online incremental training on the current normal driving behavior model based on the incremental learning algorithm that prevents catastrophic forgetting, using real-time multimodal data and abnormal data, when the real-time multimodal data meets the preset training conditions, so as to obtain the updated normal driving behavior model. The update unit is used to update the driver's basic personalized profile using the updated normal driving behavior model, so as to obtain the updated driver's basic personalized profile and the updated anomaly judgment threshold.

[0053] In some embodiments, the anomaly detection threshold is dynamically adjusted based on the updated driver basic personal profile.

[0054] In some embodiments, it also includes: The retrieval module is used to retrieve the driver's basic personalized profile when the driver drives a new vehicle; The adaptation module is used to adapt the driver's basic personalized profile to the new vehicle by recognizing the parameters of the new vehicle and making short-term data fine-tuning.

[0055] It should be noted that the foregoing explanation of the embodiment of the abnormal driving behavior recognition method based on real-time data online training also applies to the abnormal driving behavior recognition device based on real-time data online training in this embodiment, and will not be repeated here.

[0056] The abnormal driving behavior recognition device based on real-time data online training proposed in the embodiments of the present invention has the following beneficial effects: (1) The present invention proposes an abnormal driving behavior identification method that integrates multimodal data fusion analysis of driver and vehicle status, which effectively utilizes driver's personalized features and multimodal information, and can improve the accuracy of abnormal driving behavior identification; (2) The present invention proposes an abnormal driving behavior recognition method and system based on real-time data online incremental training. It can continuously update and optimize the model according to the driver's real-time driving data, so that the abnormal behavior recognition model is more and more in line with the driver's driving habits and personality characteristics, thus becoming a personal driving assistant. (3) The embodiments of the present invention propose a complete integrated portable device solution, build a framework for effectively utilizing real-time data for online incremental training, provide a clear and explicit guidance scheme for rapid implementation, and help to accurately identify abnormal driving and ensure driving safety; (4) The present invention proposes a method that can widely identify abnormal driving behaviors, not limited to specific abnormal driving behaviors such as fatigued driving, aggressive driving, and drunk driving. It does not require additional abnormal behavior labels. It uses a deep learning model to self-supervisedly mine the implicit distribution in multimodal driving data. Once the real-time data deviates from the personalized normal model, it is judged as abnormal, thus realizing zero-label, high generalization, and low-computing-power online anomaly detection.

[0057] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0058] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0059] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

Claims

1. A method for identifying abnormal driving behavior based on real-time data online training, characterized in that, Includes the following steps: Acquire initial multimodal data of the driver operating the current vehicle; The pre-built deep learning model is trained using the initial multimodal data to construct a model of current normal driving behavior; Based on the current normal driving behavior model, historical behavior patterns, personalized thresholds or feature preferences are calculated to construct a basic personalized profile of the driver and anomaly judgment thresholds; Acquire real-time multimodal data of the driver driving the current vehicle, and use the current normal driving behavior model to monitor abnormal behavior in the real-time multimodal data to obtain an anomaly score; The abnormal score is compared with the abnormal judgment threshold. If the abnormal score does not exceed the abnormal judgment threshold, multi-moment multimodal data is continuously acquired based on an active learning strategy. The real-time multimodal data and the multi-moment multimodal data are used to perform online incremental training on the current normal driving behavior model to obtain an updated normal driving behavior model. The driver's basic personalized profile and the abnormal judgment threshold are updated simultaneously. Conversely, if the behavior is not normal, it will be considered abnormal driving behavior and an alarm signal will be triggered.

2. The abnormal driving behavior recognition method based on real-time data online training according to claim 1, characterized in that, The step of training a pre-built deep learning model using the initial multimodal data to construct a current normal driving behavior model includes: The initial multimodal data is preprocessed to obtain an abnormal driving behavior feature set; The deep learning model is trained using the abnormal driving behavior feature set to construct the current normal driving behavior model.

3. The abnormal driving behavior recognition method based on real-time data online training according to claim 1, characterized in that, When the anomaly score does not exceed the anomaly judgment threshold, multi-moment multimodal data is continuously acquired based on an active learning strategy. This data, along with the real-time multimodal data, is used to perform online incremental training on the current normal driving behavior model to obtain an updated normal driving behavior model and anomaly judgment threshold. Simultaneously, the driver's basic personalized profile is updated, including: If the anomaly score does not exceed the anomaly judgment threshold, the real-time multimodal data is saved as anomaly-free data. Based on an active learning strategy, the driver continuously acquires real-time multimodal data of driving the current vehicle. When the real-time multimodal data meets the preset training conditions, the current normal driving behavior model is incrementally trained online using the real-time multimodal data and the abnormal data based on the catastrophic forgetting incremental learning algorithm to obtain the updated normal driving behavior model. The updated normal driving behavior model is used to update the driver's basic personalized profile to obtain the updated driver's basic personalized profile and the updated anomaly judgment threshold.

4. The abnormal driving behavior recognition method based on real-time data online training according to claim 1, characterized in that, The anomaly detection threshold is dynamically adjusted based on the updated driver's basic personalized profile.

5. The abnormal driving behavior recognition method based on real-time data online training according to claim 1, characterized in that, Also includes: When the driver drives a new vehicle, the driver's basic personal profile is retrieved; By identifying the parameters of the new vehicle and fine-tuning the short-term data, the driver's basic personalized profile is adapted to the new vehicle.

6. An abnormal driving behavior recognition device based on real-time data online training, characterized in that, include: The acquisition module is used to acquire the initial multimodal data of the driver driving the current vehicle; The training module is used to train a pre-built deep learning model using the initial multimodal data to construct a model of current normal driving behavior. A profile building module is used to calculate historical behavior patterns, personalized thresholds, or feature preferences based on the current normal driving behavior model, so as to build a basic personalized profile of the driver and anomaly judgment thresholds. An anomaly detection module is used to acquire real-time multimodal data of the driver driving the current vehicle, and to use the current normal driving behavior model to detect abnormal behavior in the real-time multimodal data in order to obtain an anomaly score. The training and alarm module is used to compare the abnormal score with the abnormal judgment threshold. If the abnormal score does not exceed the abnormal judgment threshold, it continuously acquires multi-moment multimodal data based on an active learning strategy. It uses the real-time multimodal data and the multi-moment multimodal data to perform online incremental training on the current normal driving behavior model to obtain an updated normal driving behavior model. It also updates the driver's basic personalized profile and the abnormal judgment threshold simultaneously. Conversely, if the behavior is not normal, it will be considered abnormal driving behavior and an alarm signal will be triggered.

7. The abnormal driving behavior recognition device based on real-time data online training according to claim 6, characterized in that, The training module includes: A preprocessing unit is used to preprocess the initial multimodal data to obtain an abnormal driving behavior feature set; The training unit is used to train the deep learning model using the abnormal driving behavior feature set to construct the current normal driving behavior model.

8. The abnormal driving behavior recognition device based on real-time data online training according to claim 6, characterized in that, The training and alarm module includes: The storage unit is used to save the real-time multimodal data as data without anomalies when the anomaly score does not exceed the anomaly judgment threshold. The continuous acquisition unit is used to continuously acquire real-time multimodal data of the driver driving the current vehicle based on an active learning strategy; The training unit is used to perform online incremental training on the current normal driving behavior model based on the real-time multimodal data and the abnormal data, using an incremental learning algorithm that prevents catastrophic forgetting, when the real-time multimodal data meets the preset training conditions, so as to obtain the updated normal driving behavior model. The update unit is used to update the driver's basic personalized profile using the updated normal driving behavior model, so as to obtain the updated driver's basic personalized profile and the updated anomaly judgment threshold.

9. The abnormal driving behavior recognition device based on real-time data online training according to claim 6, characterized in that, The anomaly detection threshold is dynamically adjusted based on the updated driver's basic personalized profile.

10. The abnormal driving behavior recognition device based on real-time data online training according to claim 6, characterized in that, Also includes: The retrieval module is used to retrieve the driver's basic personal profile when the driver drives a new vehicle; An adaptation module is used to adapt the driver's basic personalized profile to the new vehicle by identifying the parameters of the new vehicle and making short-term data fine-tuning.

Images