A system for in-vehicle passive monitoring of driver behaviors.

Abstract

ABSTRACT TITLE: A SYSTEM FOR IN-VEHICLE PASSIVE MONITORING OF DRIVER BEHAVIORS. A system for monitoring behaviours of driver while driving a vehicle comprising a millimeter wave sensor unit (1.1) for detection of the driver’s activity, an Inertial Measurement Unit (IMU) (1.3) for inferring road induced noises such as road-bumps, potholes and like and a single board computer (1.2) for performing computation on data generated by said mmWave sensor unit (1.2) and IMU (1.3) and thereby generating necessary decisions involving multimodal data and issuing control signals accordingly.

Description

FIELD OF THE INVENTION:The present invention relates to vehicle safety. More specifically, the present invention is directed to provide a Driver Monitoring System (DMS) to monitor a driver's activities involving minimal intrusive footprint and identify a set of activities that could be dangerous while driving a vehicle.BACKGROUND OF THE INVENTION:The primary cause of vehicular accidents worldwide is a mistake on the part of the driver. In this direction, passenger and pedestrian safety has become a serious concern of any modern-day vehicle. The various types of Advanced Driver Assistance Systems (ADAS) are the results of such initiatives. An ADAS can be categorized into different types based on their functional features, which range from just generating a warning for the driver to making automated control decisions of the vehicular components such as the clutch, the steering wheel, etc. Many ADASs have a Driver Monitoring System (DMS) as an integral component. In this regard, some literatures are reviewed as follows.The literature explores the idea of driving behavior monitoring using various types of approaches. Ersal et al.( T. Ersal, H. J. Fuller, O. Tsimhoni, J. L. Stein, and H. K. Fathy, "Model- based analysis and classification of driver distraction under secondary tasks," IEEE transactions on intelligent transportation systems, vol. 11, no. 3, pp. 692-701, 2010) and Dai et. al. (J. Dai, J. Teng, X. Bai, Z. Shen, and D. Xuan, "Mobile phone based drunk driving detection," in 2010 4th International Conference on Pervasive Computing Technologies for Healthcare. IEEE, 2010, pp. 1-8) used Orientation and abnormality-based approaches to assess the vehicle's orientation defined by a set of parameters such as position, velocity, acceleration, etc., and subsequently exploit them to indirectly monitor dangerous driving. The assumption here is that the abnormality in the driving behavior causes irregular patterns in these parameters, eventually inferring dangerous driving.However, several uncontrollable elements, such as weather, traffic, etc., could also cause irregular driving patterns and not necessarily signify dangerous driving. B. Cyganek (B. Cyganek and S. Gruszczy nski, "Hybrid computer vision system for drivers' eye recognition and fatigue monitoring," Neurocomputing, vol. 126, pp. 78-94, 2014) and Borghi et. al. (G. Borghi, M. Venturelli, R. Vezzani, and R. Cucchiara, "Poseidon: Face-from-depth for driver pose estimation,") applied Vision-based approaches. Whereas Dua et. al. (I. Dua, T. A. John, R. Gupta, and C. Jawahar, "Dgaze: Driver gaze mapping on road," in 2020 IEEE / RSJ International Conference on Intelligent Robots and Systems. IEEE, 2020, pp. 5946-5953) leverage computer vision on RGB cameras, Kajiwara et. al. (S. Kajiwara, "Driver-condition detection using a thermal imaging camera and neural networks," International journal of automotive technology, vol. 22, no. 6, pp. 1505-1515, 2021.) implemented thermal imaging camera and Yan et. al. (C. Yan, Y. Wang, and Z. Zhang, "Robust real-time multi-user pupil detection and tracking under various illumination and large-scale head motion," Computer Vision and Image Understanding, vol. 115, no. 8, pp. 1223-1238, 2011) applied Infrared (IR) cameras to detect abnormal driving activities by directly focusing on the driver. The captured images within the environment are processed to determine movements, including facial features such as eye movements, talking, and yawning, as well as movements of other body parts [8], such as head movements, hand movements, etc. These activities, in turn, signify patterns of inattentive or dangerous driving behaviors. However, such vision-based approaches invade privacy limiting their adoption as a practical solution primarily for public vehicles. Wearable-based approaches make use of wearable devices. Considering the pervasiveness and effectiveness of wearable devices, researchers have leveraged them to sense body kinematics and other body-specific signatures. Patterns in IMU (Inertial Measurement Unit) information, EEG (ElectroEncephaloGram) (G. Cisotto, A. V. Guglielmi, L. Badia, and A. Zanella, "Joint compres- sion of eeg and emg signals for wireless biometrics," in 2018 IEEE Global Communications Conference. IEEE, 2018, pp. 1-6.) signals, heart rate, etc., bear various activity-specific signatures and thus are used to assess abnormal and dangerous driving behaviors eventually. However, such signatures often vary between different demographics (e.g., age groups), making it difficult to make such a system generic. Additionally, wearables must be explicitly carried by the users and may have invasive footprints (e.g., EEG sensor fitted on the body) interfering with individuals' day-to-day movements. Several researchers have also explored acoustics for studying distracted and dangerous driving scenarios, which leverages doppler shifts as well as FMCW chirps

[13] . Xu et. al. (X. Xu, H. Gao, J. Yu, Y. Chen, Y. Zhu, G. Xue, and M. Li, "Er: Early recognition of inattentive driving leveraging audio devices on smartphones," in IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 2017, pp. 1-9.) and Xie et. al. (Y. Xie, F. Li, Y. Wu, S. Yang, and Y. Wang, "D 3-guard: Acoustic- based drowsy driving detection using smartphones," in IEEE INFOCOM 2019-IEEE Conference on Computer Communications. IEEE, 2019, pp. 1225-1233.) applied doppler shifts whereas Jiang et. al. (H. Jiang, J. Hu, D. Liu, J. Xiong, and M. Cai, "Driversonar: Fine-grained dangerous driving detection using active sonar," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 5, no. 3, pp. 1-22, 2021.) applied FMCW acoustic chirps to detect the dangerous activity of a driver while driving a vehicle. However, environmental noise has a significant influence over such acoustic-based sensing. Even with pervasive devices such as smartphones being used for acoustic sensing, it is worth noting that different devices have different sensitivity patterns for audio signals that impact the effectiveness of such systems. Other factors, such as the impact of location and orientation of the sensor, and even privacy issues, are the cause of concern for acoustic sensing.In this regard the following patent literatures are also reviewed.Sun et. al. (Title: "Advanced driver assistance system for vehicle." Patent No. US9308914).developed a Driver status Monitor to monitor abnormal condition of the driver. It uses a driver-facing camera to capture driver's face aiming to capture gazing directions or to identify whether the driver is drowsy as an abnormal condition. While driver's other physical conditions are estimated by scanning his / her heart beats using a health scanner. Kagawa (Title: "Traveling auxiliary device for vehicle." Japan Patent No. JPH10315800A. 02 Dec. 1998) developed a driver awakening state monitoring system. leverages a set of sensors for detecting information about the running state and the surrounding environment of the vehicle. This includes a vehicle speed sensor, a steering angle sensor, a yaw rate sensor, and an illuminance sensor. Michael et. al. (Title: "Method and device for characterising the state of the driver of a motor vehicle" WIPO(PCT) Patent No. WO2002096694A1. 5 Dec. 2002.) captured physiological status parameters of the driver using electroencephalogram (EEG) and electrocardiogram, EKG. The driver is observed for wakefulness and heart rate. Other sensors were considered to capture skin temperature and blood pressure. A camera was also considered for capturing face and facial movements. The invention by Shigeyasu (Title: "Driving support device for vehicle and inattentive driving state detector" Japan Patent No. JP2007299048A. 15 Nov. 2011.) uses an in-car camera to monitor a driver's side-by-side gazing or dozing. It also uses a steering touch sensor to monitor temperature change, vibration (due to heartbeat), pressure change, voltage change, and the likes. Masakazu (Title: "Driving support apparatus." Japan Patent No. JP2008097278A. 24 Apr. 2008) developed a method to monitor driver's gazing direction using a camera. Masumi et. al. (Title: "Driving support apparatus." Japan Patent No. JP2011238133A. 24 Nov. 2011.) developed a technique to observe driver health condition using a set of biosensors. The biosensors detect various physiological condition of the driver such as driver's pulse, blood pressure, body temperature, amount of sweat, body weight, body fat percentage, brain wave, line-of-sight direction, blood glucose level, and the like. Additionally, operation state sensors detect steering angle, steering force, steering speed, accelerator pedal depression amount, accelerator pedal depression force, accelerator pedal operation speed, brake pedal depression amount, brake pedal depression force, brake pedal operation speed, transmission pedal, shift lever shift position, transmission shift lever operating force, transmission shift lever operating speed, and the likes. Kin et. al. (Title: "System and method for responding to driver behavior," U.S. Patent No. 0 212 353 A1, Dec. 3, 2012.) use optical / thermal information to assess if the driver is drowsy. Anna et. al. (Title: "Driver monitoring and vehicle control system." United Kingdom Patent No. GB2500690A. 2 Oct. 2013.) developed a camera-based method for monitoring driver movements and behaviors. Yasuhiko et. al. (Title: "Driver's emergency support device and driver's emergency support method." Japan Patent No. JP2014044707A. 13 Mar. 2014.) make use of a driver monitoring system that performs monitoring of the driver's state using the output video of the driver-camera. The invention by Hiroharu et. al. (Title: "Moving body driving support device." Japan Patent No. JP2001219760A. 14 Aug. 2001.) also uses a camera to determine the dozing state of the driver. The driver monitoring system by Hiroaki (Title: "Vehicle deviation alarm device." Japan Patent No. JP2006331323A. 7 Dec. 2006.) also uses a driver-facing camera to detect if the driver is looking sideways. Heyes et. al. (Title: "Method for supporting a driver of a vehicle, in particular a motor vehicle or utility vehicle." Germany Patent No. DE102011011714A1. 23 Aug. 2012) considered driver's state in terms of driver's attention, that can be derived from various measurements such steering wheel angle or SWRR (Steering Wheel Reversal Rate). Takaso (Title: "Vehicle traveling control apparatus." U.S. Patent No. US9944294. 17 Apr. 2018.) consider a driver state detector and inattentive driving in terms of degree of awakening, consciousness, eyesight, driver's direction of facing etc. using several sensing methods such as a camera and an infra-red lamp within the vehicle, operation of steering wheel, angle of steering wheel, data acquired through biological sensors such as blood pressure, heart-rate, sweating, etc.From the above study, it is observed that the existing solutions for driver monitoring either use a privacy-intrusive camera-based approach or require the driver to wear body-mounted sensors. Therefore, there has been a need for developing a Driver Monitoring System (DMS) system to monitor and identify a set of activities of a driver that could be dangerous while driving a vehicle involving minimal intrusive footprint.OBJECT OF THE INVENTION:It is thus the basic object of the present invention is to develop a Driver Monitoring System (DMS) system to assess driving behaviours of a driver using minimal intrusive footprint. Another object of the present invention is to develop a system and method for detecting driving activities.Another object of the present invention is to develop a system and method to identify a number of activities, which can be considered dangerous on the part of a driver, while driving a car.Yet another object of the present invention is to develop a machine learning based pipeline for a Decision-making model that can be hosted on low-power devices for on-board inference without any requirement for a cloud support for inference computation.SUMMARY OF THE INVENTION:Thus, according to the basic aspect of the present invention there is provided a system for monitoring behaviours of driver while driving a vehicle comprisinga millimeter wave (mmWave) sensor unit (1.1) for capturing frequency bins corresponding to driver movements;an Inertial Measurement Unit (IMU) (1.3) for inferring road induced noises such as road-bumps, potholes and like; a single board computer (1.2) for performing computation on data generated by said mmWave sensor unit (1.2) and IMU (1.3) and thereby detecting driver activities and generating necessary decisions involving multimodal data and issuing control signals accordingly.In the present system, the mmWave sensor unit (1.1) includes a Frequency-modulated continuous wave (FMCW) radar-based sensor for capturing signals corresponding to the driver activities, whereby said FMCW-based sensor operates by emitting continuous radio frequency waves and analyzing reflections off objects within its detection range to accurately detect and differentiate the diverse driver activities and behaviors in real-time and to alert or intervene when it detects potentially risky behaviors, thereby enhancing overall safety on the road.In the present system, the IMU (1.3) includes a combination of accelerometers, gyroscopes, and magnetometers to mitigate interference of road bumps and potholes on mmWave sensor operation including utilizing IMU data that involves analyzing the accelerometers to detect acceleration spikes indicative of road irregularities and extracting these patterns, utilizing machine learning or signal processing techniques identify and timestamp instances of potholes or bumps and synchronizing these timestamps with mmWave sensor data that allows targeted exclusion of corresponding segments affected by road noise.In the present system, the single Board computer (1.2) comprisesa speaker (1.4) for generating warning alarms and warning messages on detection of a dangerous driving pattern;a wireless transmitter module (1.5) for transmitting warning messages received over the internet using this module and transmitting sensed information to the cloud for further processing followed by a decision making;warning lights (1.6) for integration on the vehicle and activating which selectively activable to warn the pedestrians and nearby cars about a potential danger.In the present system, the single board computer (1.2) is configured for detecting activity classes, including (a) nodding, (b) yawning, (c) steering anomaly, (d) drinking, (e) communication with other passengers, (f) picking a drop, (g) fetching from the dash, (h) using mobile, (i) talking sideways, and generating necessary decisions based on the multimodal data involvinga preprocessing module for processing of data including range bins, range-doppler heatmap, noise profile as calculated from doppler information at different range bins;a noise removal module for filtering corresponding data collected from the preprocessing module to suppress unwanted noise in feature space;a feature extraction network for capturing temporal variations of the data received from said noise removal module to extract cross channel feature which includes patterns or trends in the data over time, potentially representing significant driving behavior indicators, such as abrupt changes in speed, erratic movements, or repeated patterns in drivers activity;a DVN classifier which utilizes a generic protocol to distinguish normal driving patterns from dangerous ones, said DVN classifier leverages features extracted from the previous stage, potentially looking for deviations from established norms, erratic movements, or behaviors associated with risky driving, triggering an alert or intervention when such patterns are identified;a DDB classifier activable by the DVN classifier upon detecting dangerous driving behavior for determining the particular dangerous driving class, said DDB classifier operates using a generic protocol to determine the specific class of dangerous driving behavior, said DDB classifier relies on a more detailed analysis of the extracted features, identifying and categorizing specific risky behaviors such as aggressive manoeuvres, distracted driving, or other hazardous actions observed from the data.BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS:Figure 1 discloses hardware components of the present Driver Monitoring System (DMS) system (mmDrive).Figure 2 reveals method of data flow through the mmDrive components.Figure 3 shows the range-doppler heatmaps for driving actions.Figure 4 represents the experimental setup of mmDriveFigure 5 depicts the confusion matrix for all the dangerous driving behaviors where (a) represents Fused-CNN, (b) represents RF, (c) represents VGG-16 and (d) represents Acoustic-FMCWFigure 6 (a) depicts the Weighted F1-Score across driving behaviors, and (b) represents AUC-ROC for classifying dangerous vs normal driving.DETAILED DESCRIPTION OF THE INVENTION WITH REFERENCE TO THE ACCOMPANYING DRAWINGS:The present invention thus provides a system called mmDrive using millimeter wave (mmWave)-based passive sensing. The present invention also provides a cooperative method that can monitor real-time driving behaviors of a driver similar to any existing DMSs but with a completely different perspective. Firstly, it uses mmWave-based sensing for driver activity detection, which is novel for driver activity detection. Secondly, it uses a novel technique to identify dangerous activities while driving a vehicle. These two components aim to provide an end-to-end driving activity monitoring solution and have been tested in real-world conditions. Additionally, the compact setup of mmDrive can be hosted in a vehicle facing the driver. It continuously monitors the dynamics of the driver and tries to capture their movement patterns which could be potentially dangerous for the passengers and the people outside. If it detects such practices, it generates warning signals identifying a pre-defined class of hazardous driving behavior.The developed system, mmDrive, captures different activities of driver associated with the attentiveness of the driver. The hardware components of mmDrive is shown in Figure 1. The system is composed of the following primary hardware components.mmWave Sensing Module: The sensing unit is primarily built on top of a single Commercial-Off-The-Shelf (COTS) FMCW mmWave sensor. A generic FMCW (Frequency-modulated continuous wave) sensor uses a linear 'chirp' (continuous waves in the form of periodic signals) or swept frequency transmission, which is reflected by the obstacles in the environment. The radar then performs the dechirping by mixing the transmitted signal with the reflected signal. The resulting Intermediate frequency signal (IF) then undergoes a set of processing to extract the needed information. At this point, the sensor can measure various information from this collected signal. However, mmDrive requires the following information estimated by the mmWave sensor:i) Different range bins in the form of a one-dimensional array signifying range profile. Here range bin refers to a specific distance within the radar's range that is used to detect and measure the presence or absence of objects or targets.ii) The range and velocity information in a 2D matrix called a range-doppler heatmap.iii) noise profile calculated from Doppler information at different range bins. The noise profile is in the same format as range profile, but the profile is at the maximum Doppler bin (maximum body movements)Single Board ComputerIt is computing device on a single circuit board which has microprocessor(s), memory, input / output (I / O) and other features required of a functional computer. Due to its compact size, it is ideal to be used in the mmDrive. It is primarily responsible to perform computation on the data generated by the mmWave sensor. On processing the multimodal data generated by the sensory components, the device generates necessary decisions and therefore issues control signals accordingly.IMU sensor moduleThe Inertial Measurement Unit (IMU) is responsible to measure movements and inertia of bodies using a combination of accelerometers, gyroscopes, and sometimes magnetometers. This is needed to capture movement of the car specific to road conditions such as road bumps and sudden jerks. The captured signatures is used to mitigate the noise generated in the signals captured by the mmWave sensor due to such movements.SpeakerThe speaker generates warning alarms whenever a dangerous driving pattern is detected. The system is programmable to play effective warning messages through the speaker for alert.Wireless Transmitter ModuleThe wireless module requires a connection with the internet. When required, mmDrive can transmit warning messages over the internet using this module. The sensed information can also be transmitted to the cloud for further processing followed by a decision making. Warning lightsThis is the integration of warning / hazard lights of a vehicle. Whenever necessary, mmDrive can activate warning lights to warn the pedestrians and nearby cars about a potential danger.The developed prototype is a fully functional standalone system for monitoring dangerous driving activities within the car. COTS mmWave FMCW sensor, model AWR6843ISK from Texas Instruments have been used. As for the single board computer, a Raspberry Pi model 3B is used for executing the software components of mmDrive. For this evaluation IMU sensors integrated in Nexar 4 car dash cam is used. However, the above mentioned module can be replaced by a commercially available IMU sensor module such as, MPU-6050. The data flow through the mmDrive components is shown in Figure 2. Once the sensor captures the reflected frames, a Pre-processing step concatenates 10 feature frames together (determined based on empirical analysis). Finally, a min-max scaler normalize the features from 0 to 1. A Noise removal module takes the help of an alternate modality from IMU sensors embedded within the device. The z-axis acceleration of the vehicle to determine jerkiness due to poor road conditions or bumps has been applied. Subsequently, the noise removal module filters the corresponding data collected from the mmWave sensor to suppress unwanted noise in the feature space. Feature Extraction (FE) Network concatenates 10 feature frames, to capture the temporal variations. Hence, range and noise profiles are vectors of size 64x1x10, and concatenating them together forms an array of size 64x2x10. On the other hand, the range-doppler heatmap forms a stacked 2D image-like feature with size 16x64x10. The architecture is designed using three Convolutional 2D layers with valid padding and ReLU activation, followed by a global average pooling to extract 96-dimensional range-noise-based embeddings followed by a global average pooling to extract 96-dimensional range-noise-based embeddings. Convolution 2D operation to extract the dependency of neighboring values within all possible k x k regions at each range-doppler frame along with the temporal relationship of past 10 such frames is applied. Further, it computes several cross-channel feature maps, which are helpful for subsequent model layers. Four 2D Convolutional layers with the same padding and ReLU activation, followed by a global averaging to extract 128-dimensional range-doppler feature embeddings is used. Next, the concatenated range-noise and range-doppler feature embeddings are forwarded through two successive modules of the proposed architecture. The DDB (Dangerous Driving Behaviors) classifier takes the cross-channel feature embeddings from the FE Network and passes through two successive Dropout and Dense layers, where the dropout rate is kept as 10% to prevent overfitting. The last layer has 9 neurons with softmax activation to output a joint probability distribution over the nine dangerous driving actions. Note that the FE Network is trained only with the backward gradients from the DDB Classifier to learn efficient feature extraction for classifying 9 dangerous driving behavior. The DVN Classifier (Dangerous Vs Normal driving) utilizes the learned feature embeddings from the pre-trained FE Network to classify all forms of dangerous driving behaviors from normal driving. Finally, during real-world deployment, the DVN Classifier determines whether the driving behavior is normal or dangerous. Upon detecting normal driving behavior, mmDrive differs the execution of the DDB Classifier, reducing computational overhead and energy consumption. The DDB Classifier is queried only if the DVN Classifier detects potentially dangerous driving behavior, further determining the particular dangerous driving class.It is to be mentioned that mmWave sensing can monitor a driver's movements directly rather than indirect observations such as vehicle states and its kinematics. Unlike cameras, mmWave sensing has a minimal invasion of privacy as it does not capture visual features of the environment. Unlike wearables, mmWave sensing does not require a user to mount, or carry any device and measures passively, without restricting the subject's regular movements and activities. With mmWave sensing micro-movements such as yawning can also be detected which is very crucial for determining a sleepy state of a driver. Additionally, mmWave can penetrate potentially light-occluding entities such as clothing. and therefore it can be used even if the driver wearing a face mask. Another reference comes from Figure 3 wherein the Figure depicts the signatures captured (in the form of range-doppler heatmap by an mmWave radar) for different dangerous activities of the driver. In this regard these activities are mainly classified into two basic categories (1) actions caused by fatigue / drowsiness and (2) actions caused by distraction. The first group i.e. actions caused by fatigue / drowsiness typically occurs when the driver is fatigued or drowsy and could potentially cause an accident. In the proposed invention the inventiveness lies in its sensing mechanism and activity assessment methodology to detect the dangerous activity / behaviour of a driver. It is a vehicle-hosted end-to-end system to monitor and identify a well-defined set of activities that could be dangerous while driving a vehicle. In a nutshell it is basically composed of two primary components, which are novel for dangerous driving activity assessment. The first is monitoring driving activities by adopting the mmWave sensing mechanism, and the second is an activity detection technique to identify driver behaviors that could be dangerous while driving. Through empirical trials, we notice that mmWave data can be influenced by signal interferences stemming from poor road conditions like bumps or potholes. To address this, we utilize IMU sensor data to identify and mitigate these disruptions. Additionally, we introduce an innovative Fused-CNN classifier to distinguish between safe and risky driving activities. Remarkably, it not only distinguishes risky driving from regular driving but also categorizes nine distinct instances of hazardous driving activities (which are - (a) Nodding, (b) Yawning, (c) Steering anomaly, (d) Drinking, (e) Talking to the rear passenger, (f) Picking a drop, (g) Fetching from the dash, (h) Using mobile, (i) Talking sideways). TESTING:The experimental setup for mmDrive is shown on Figure 4. mmDrive is implemented on COTS mmWave radar AWR6843ISK, an FMCW-based mmWave radar from Texas InstrumentsTM. The radar works in the frequency range of 60- 64GHz with a range resolution of approximately 4cm, which is adequate for measuring the activities of interest. The radar's maximum range is up to a distance of 10 meters with a Field-of-View of -70° to +70° on azimuthal and elevation planes. This Field-of-View is suitable for detecting dangerous driving behaviors within the car. mmDrive measures variations across 64 range bins, which represent 2.4 meters distance from the dashboard. This brings a range resolution of 3.75 cm. mmDrive collects the doppler information across 16 doppler bins from the 2D-FFT to have a velocity resolution of 0.13 m / s which is sufficient to capture the driver's micro as well as macro body movements in real- world driving scenarios. Moreover, Raspberry Pi 4 Model-B with 8GB RAM for on-device driving behavior detection hasbeen used. The mmWave radar and IMU sensor are connected to the Pi-4 via USB and I2C bus, respectively. The IMU sensor helps in filtering out the road-induced noises. The mmWave serial data is parsed and forwarded to the classification pipeline for inferring the driving behavior.Python 3.9, Tensorflow v2.10, and Scikit- learn v1.1 for implementing the Fused-CNN-based driver behavior classifier model alongside other two state-of-the-art baselines: Random Forest (RF), and VGG-16. VGG-16 network is initialized with the pre- trained weights on the ImageNet dataset to have a transfer learning approach helping in learning feature extraction from high dimensional image-like input data. On top of this base VGG-16 model, global average pooling and successive Dropout and Dense layers in the Fused-CNN architecture havebeen added to classify dangerous driving behaviors. The models are trained with a train-test split of 70%-30% and a validation split of 20% from the training set. Besides these two baselines, A FMCW based-acoustic modality has been applied as the third baseline. Previous works such as, use an acoustic sensing approach to detect dangerous driving behaviors with fewer behavior classes. To understand the feasibility of this approach with nine different dangerous driving behaviors, an acoustic-FMCW using the smartphone-embedded microphone and speaker in the near- ultrasound range between 16-19kHz has been implemented. Raw chirps using the smartphone is transmitted and received and then Range- FFT in the post-processing to generate the amplitude and phase of the IF signal across different range bins is applied. Further 2D-doppler FFT to generate the range-doppler heatmap is applied. Finally, using a Random Forest-based classifier, its performance is compared with mmDrive.Performance of DDB ClassifierFig. 5, depicts the confusion matrix for all the three classifiers. From the confusion matrix, it is evident that the proposed Fused- CNN model shows superior accuracy when compared with the baselines. Fig. 6(a) shows the average weighted F1-Score for all the individual dangerous driving actions. Among the baselines, RF performs better in comparison to VGG-16. The primary reason behind poor performance with VGG-16 is that this model expects a 2D input feature, which is a 2D range-doppler heatmap image in this case. Thus, it cannot take advantage of range or noise profile-based features. Also, VGG-16 is pre-trained with the image net dataset and does not fit well in engineering features from a 2D range-doppler heatmap, which is mostly sparse across the range bins except for the range bin where the driver is present. In the case of RF, the features are passed across different kernel sizes to capture the spatial variation in the range and noise profiles as well as in the range-doppler heatmap. Thus, it is observed a slightly better performance can be achieved in comparison to VGG-16. On the other hand, the proposed Fused-CNN has complete freedom in learning the spatio-temporal cross-features, as the features are not hand-engineered like RF (min, max, mean, standard deviation). Moreover, Fused-CNN shows a lower inference time due to less number of convolutional layers as compared to VGG-16.Performance of DVN ClassifierFig. 6 (b) reveals the ROC curve for the DVN classifier. As shown in the figure, the area under the curve (AUC) is 0.96, which ensures good accuracy in classifying dangerous driving from normal driving behavior. Moreover, a weighted F1-Score of 90(±0.5) % has been observed.The advantages of the present invention can be summarized as hereunder:Camera-less monitoring The existing commercial systems for monitoring dangerous driving activity primarily rely on cameras which is privacy invasive due to their inherent nature. Therefore, avoiding the need for a camera while performing a similar functionality, mmDrive significantly solves privacy concerns.Functional through occlusionsSince mmDrive leverages mmWave signal for its purpose, it can work through different materials such as clothes. As mmWave can penetrate clothes, mmDrive can detect activities, such as yawning while wearing a mask, that cannot be detected by a video capture.Enhanced SafetyCan provide real-time monitoring of driver behavior and activities inside the vehicle. These systems can significantly reduce the risk of accidents, injuries, and fatalities by alerting drivers and taking preventive actions. This enhanced safety can result in reduced insurance costs for individuals and fleets.Insurance Premium ReductionInsurance companies often offer discounts and incentives for drivers who have installed vehicle safety systems. The proposed system can qualify for such discounts, potentially reducing insurance premiums for vehicle owners. Fleet Management Optimization By tracking driver behavior, such as excessive idling, and harsh acceleration, fleet managers can identify areas for improvement and implement driver training programs. Optimized driving behavior can result in lower fuel consumption, reduced maintenance costs, and improved overall fleet performance.Regulatory ComplianceTransportation and logistics companies are often subject to stringent driver safety and compliance regulations. Thus, it can help companies by providing a comprehensive record of driver behavior and adherence to safety protocols. Data Analytics and InsightsThe data collected by the proposed systems can be leveraged for advanced analytics and insights. By analysing the patterns, companies can identify risk factors, develop proactive safety strategies, and optimize their operations.

Claims

1. A system for monitoring behaviours of driver while driving a vehicle comprising a millimeter wave sensor unit (1.1) for capturing signatures of the driver’s activities; an Inertial Measurement Unit (IMU) (1.3) for inferring road induced noises such as road-bumps, potholes and like; a single board computer (1.2) for performing computation on data generated by said mmWave sensor unit (1.2) and IMU (1.3), thereby identifying a specific activity and generating necessary decisions involving multimodal data and issuing control signals accordingly.

2. The system as claimed in claim 1, wherein the millimeter wave sensor unit (1.1) includes a Frequency-modulated continuous wave (FMCW)radar based sensor for capturing signatures of driver’s activity, whereby said FMCW-based sensor operates by emitting continuous radio frequency waves and analyzing reflections off objects within its detection range to accurately detect and differentiate the diverse driver activities and behaviors in real-time and to alert or intervene when it detects potentially risky behaviors, thereby enhancing overall safety on the road.

3. The system as claimed in claim 1 or 2, wherein the IMU (1.3) includes a combination of accelerometers, gyroscopes, and magnetometers to mitigate interference of road bumps and potholes on mmWave sensor operation including utilizing IMU data that involves analyzing the accelerometers to detect acceleration spikes indicative of road irregularities and extracting these patterns, utilizing machine learning or signal processing techniques identify and timestamp instances of potholes or bumps and synchronizing these timestamps with mmWave sensor data that allows targeted exclusion of corresponding segments affected by road noise.

4. The system as claimed in claims 1 to 3, wherein the single Board computer (1.2) comprises a speaker (1.4) for generating warning alarms and warning messages on detection of a dangerous driving pattern; a wireless transmitter module (1.5) for transmitting warning messages received over the internet using this module and transmitting sensed information to the cloud for further processing followed by a decision making; warning lights (1.6) for integration on the vehicle and activating which selectively activable to warn the pedestrians and nearby cars about a potential danger.

5. The system as claimed in claims 1 to 4, wherein the single board computer (1.2) is configured for detecting the type of activities including (a) nodding, (b) yawning, (c) steering anomaly, (d) drinking, (e) communication with other passenger, (f) picking a drop, (g) fetching from the dash, (h) using mobile, (i) talking sideways, and generating necessary decisions based on the multimodal data involving a preprocessing module for processing of data including range bins, range-doppler heatmap, noise profile as calculated from doppler information at different range bins; a noise removal module for filtering corresponding data collected from the preprocessing module to suppress unwanted noise in feature space; a feature extraction network for capturing temporal variations of the data received from said noise removal module to extract cross channel feature which includes patterns or trends in the data over time, potentially representing significant driving behavior indicators, such as abrupt changes in speed, erratic movements, or repeated patterns in drivers activity; a DVN classifier which utilizes a generic protocol to distinguish normal driving patterns from dangerous ones, said DVN classifier leverages features extracted from the previous stage, potentially looking for deviations from established norms, erratic movements, or behaviors associated with risky driving, triggering an alert or intervention when such patterns are identified; a DDB classifier activable by the DVN classifier upon detecting dangerous driving behavior for determining the particular dangerous driving class, said DDB classifier operates using a generic protocol to determine the specific class of dangerous driving behavior, said DDB classifier relies on a more detailed analysis of the extracted features, identifying and categorizing specific risky behaviors such as aggressive manoeuvres, distracted driving, or other hazardous actions observed from the data.