A driver status monitoring system based on intelligent image recognition

Abstract

This invention relates to the field of intelligent driving assistance technology, specifically disclosing a driver state monitoring system based on intelligent image recognition. Built on a Raspberry Pi embedded platform, it includes a video acquisition module, a core algorithm processing module, and a peripheral control module. The video acquisition module is compatible with USB or CSI cameras, acquiring and preprocessing driver facial images. The core algorithm processing module incorporates a lightweight facial key point detection model to extract facial features and identify dangerous behaviors such as closed eyes and head tilting, and features a feature calibration and fatigue grading mechanism. The peripheral control module connects to alarm and display components, triggering corresponding warning actions and can be expanded to include multiple alarm methods. The system can also track continuous driving time and expand remote monitoring functionality. This invention features low hardware cost, low power consumption, convenient deployment, fast edge-side inference response, and accurate monitoring. It is adaptable to multiple scenarios, requires no manual intervention, comprehensively prevents fatigue driving, and is easy to deploy on a large scale.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving assistance technology, specifically to a driver status monitoring system based on intelligent image recognition. Background Technology

[0002] With the rapid development of the transportation industry and the continuous growth of motor vehicle ownership, driving safety issues are becoming increasingly prominent. According to statistics, driver fatigue and distracted driving (such as closing eyes, looking down, yawning, etc.) are among the main causes of traffic accidents, especially for long-distance commercial vehicles and vehicles on night shifts at checkpoints. Drivers are in a high-intensity driving state for long periods of time, making them highly susceptible to fatigue and dangerous behavior, which seriously threatens their own and others' lives and property.

[0003] Existing driver status monitoring systems often employ complex hardware architectures, relying on high-performance servers for image processing. This results in high costs, cumbersome deployment, high response latency, and an inability to achieve real-time edge-side inference, making them unsuitable for small vehicles and low-cost application scenarios. Some simplified monitoring systems can only detect single hazardous behaviors, exhibiting low accuracy and limited functionality. Furthermore, they lack flexible peripheral device adaptation and scenario expansion capabilities, failing to meet the monitoring needs of different driving environments and usage scenarios.

[0004] The Raspberry Pi embedded platform boasts advantages such as small size, low power consumption, low cost, and high scalability, and has been widely used in the development of various embedded intelligent devices. Combined with lightweight image recognition algorithms, it enables real-time image processing and status assessment on the device side, effectively addressing the aforementioned shortcomings of existing monitoring systems. Therefore, developing a driver status monitoring system based on the Raspberry Pi embedded platform that combines low cost, high precision, real-time performance, and high scalability has significant practical importance and application value. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a driver status monitoring system based on intelligent image recognition. The system is built on a Raspberry Pi embedded platform and combined with a lightweight facial key point detection algorithm to realize real-time monitoring and graded early warning of dangerous driver behaviors. At the same time, it has flexible peripheral expansion and scene adaptation capabilities, reduces deployment costs, and improves driving safety.

[0006] To solve the above-mentioned technical problems, the technical solution provided by this invention is: a driver status monitoring system based on intelligent image recognition, comprising a video acquisition module based on a Raspberry Pi embedded platform, a core algorithm processing module, and a peripheral control module. The video acquisition module is compatible with a USB camera or a CSI camera module and is used to acquire driver facial images and perform preliminary image preprocessing. The core algorithm processing module incorporates a lightweight facial key point detection model and feature analysis logic algorithm to achieve end-side reasoning, driver facial feature extraction, and state judgment. The peripheral control module is connected to an alarm component and a display component, and is used to receive judgment signals from the core algorithm processing module and trigger corresponding peripheral actions.

[0007] Furthermore, the core algorithm processing module extracts key features of the driver's eyes, mouth, and facial contours, analyzes and judges in real time whether the driver exhibits potential dangerous behaviors such as closing their eyes, looking down, yawning, or losing their face. When any of these dangerous behaviors are detected, a trigger signal is sent to the peripheral control module.

[0008] Furthermore, the camera module of the video acquisition module is fixed to the vehicle dashboard or a matching bracket, and the camera can be adjusted to clearly capture the driver's facial area.

[0009] Furthermore, the Raspberry Pi embedded platform is connected to the vehicle's 5V power supply. After the power is turned on, it automatically starts and loads the monitoring program to complete the initialization.

[0010] Furthermore, the core algorithm processing module has a facial feature calibration function. Upon first run or after the camera position changes, it captures the baseline facial parameters of the driver in a normal driving posture and uses the baseline facial parameters as a reference for subsequent driver status judgment.

[0011] Furthermore, the alarm component of the peripheral control module is a voice broadcast component, which immediately issues a voice alarm upon receiving a trigger signal; upon receiving the trigger signal, the display component synchronously displays the corresponding warning prompt for dangerous behavior, and the voice alarm and the screen prompt are triggered simultaneously.

[0012] Furthermore, the core algorithm processing module has a fatigue risk level judgment mechanism, which distinguishes between mild fatigue and severe fatigue based on the duration and frequency of the driver's dangerous behavior, and sends trigger signals of different intensities to the peripheral control module, so that the peripheral control module outputs a corresponding level of reminder action.

[0013] Furthermore, the Raspberry Pi embedded platform has a built-in timing module for counting the continuous running time of the system to realize the driver's continuous driving time statistics, and triggers an active rest reminder through the peripheral control module when the preset time is reached.

[0014] Furthermore, the peripheral control module can be extended to connect vibration feedback components, light flashing components, and form multiple alarm mode combinations with voice broadcast components and display components to adapt to driving environments with different noise levels.

[0015] Furthermore, the Raspberry Pi embedded platform also includes a wireless communication module for extending remote monitoring functions, uploading driver status monitoring data and dangerous behavior alarm information to the remote monitoring terminal, enabling extended applications in multiple scenarios such as night shift guard duty and fleet management.

[0016] The advantages of this invention compared to the prior art are: This invention is built on the Raspberry Pi embedded platform, which has low hardware cost, small size and low power consumption. It can be directly adapted to various vehicles without complicated installation and debugging. It can automatically start and run when connected to the vehicle's 5V power supply, which reduces the system deployment and usage costs and facilitates large-scale promotion and application.

[0017] The core algorithm processing module of this invention adopts a lightweight facial key point detection model to achieve real-time inference on the device side without relying on a remote server, resulting in low response latency. It also has a facial feature calibration function to effectively avoid judgment errors caused by various interference factors. It can accurately identify dangerous behaviors such as closing eyes, looking down, yawning, and missing faces, and the monitoring accuracy meets the needs of practical applications.

[0018] This invention features a fatigue risk grading and judgment mechanism that can differentiate between mild and severe fatigue based on the duration and frequency of dangerous behaviors, enabling graded early warning. It also integrates continuous driving time statistics and proactive rest reminders to comprehensively prevent fatigued driving and improve the pertinence and effectiveness of early warnings.

[0019] The peripheral control module of this invention can be expanded to connect to various alarm components to form a combined alarm mode, adapting to driving environments with different noise levels; through the wireless communication module, the remote monitoring function can be expanded, adapting to multiple application scenarios such as night shift guard duty and fleet management, with high flexibility.

[0020] The system of this invention starts automatically, calibrates automatically, and monitors automatically without human intervention; voice alarms and screen prompts are triggered synchronously, enabling drivers to quickly perceive warning information. It is easy to operate and its level of intelligence meets the needs of modern driving assistance. Attached Figure Description

[0021] Figure 1 This is a system block diagram of a driver status monitoring system based on intelligent image recognition according to the present invention. Detailed Implementation

[0022] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0023] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0024] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0025] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0026] The following detailed description of a driver status monitoring system based on intelligent image recognition according to the present invention, with reference to the accompanying drawings, is provided in further detail.

[0027] Combined with appendix Figure 1 This invention will be described in detail below.

[0028] A driver status monitoring system based on intelligent image recognition includes a video acquisition module, a core algorithm processing module, and a peripheral control module based on a Raspberry Pi embedded platform. The modules work together to realize real-time acquisition, analysis, judgment, and early warning of the driver's status.

[0029] The video acquisition module is compatible with USB or CSI camera modules to capture driver facial images and perform preliminary image preprocessing. This preprocessing includes image noise reduction, size normalization, and grayscale conversion, providing clear and standardized image data for subsequent facial feature extraction and state assessment, ensuring the analytical accuracy of the core algorithm processing module. The camera module can be fixed to the vehicle dashboard or a matching bracket and its angle can be flexibly adjusted to ensure a clear and direct view of the driver's face, avoiding feature extraction failures or judgment errors due to shooting angle deviations.

[0030] The core algorithm processing module incorporates a lightweight facial landmark detection model and feature analysis logic algorithm to achieve edge-side inference, driver facial feature extraction, and state judgment. This module extracts key features from the driver's eyes, mouth, and facial contours, and analyzes them in real time to determine if the driver exhibits potentially dangerous behaviors such as closing their eyes, looking down, yawning, or losing sight of their face. When any dangerous behavior is detected, a trigger signal is sent to the peripheral control module. The lightweight facial landmark detection model uses an efficient model adapted for Raspberry Pi edge-side inference, balancing detection accuracy and running speed while avoiding excessive system resource consumption and ensuring real-time response. Lightweight improved versions of open-source efficient models such as SeetaFace6 can be used, and the model is adaptable to complex scenarios such as mask occlusion, exhibiting low false recognition rate and strong robustness.

[0031] The core algorithm processing module also features facial feature calibration. Upon initial operation or after a change in camera position, it automatically captures baseline facial parameters of the driver in a normal driving posture and uses these parameters as a reference for subsequent driver status assessments. This effectively avoids judgment errors caused by changes in driver posture or camera position shifts, improving monitoring accuracy. Simultaneously, the module incorporates a fatigue risk level assessment mechanism. Based on the duration and frequency of dangerous driver behavior, it differentiates between mild and severe fatigue and sends trigger signals of varying intensities to the external control module. This prompts the external control module to output corresponding alerts, achieving tiered warnings and enhancing the targeting and effectiveness of the warnings.

[0032] The peripheral control module is connected to an alarm component and a display component. It receives judgment signals from the core algorithm processing module and triggers corresponding peripheral actions. The alarm component is a voice broadcast component. Upon receiving the trigger signal, it immediately issues a voice alarm. The content of the voice alarm can be differentiated according to the type of dangerous behavior and the fatigue level. For example, it broadcasts "Please take a rest" for mild fatigue and "Severe fatigue, please stop and rest immediately" for severe fatigue. Upon receiving the trigger signal, the display component synchronously displays the corresponding warning prompts for dangerous behaviors, such as "Eye closing warning" and "Yawning warning". The voice alarm and the screen prompt are triggered synchronously to ensure that the driver can perceive the warning information in a timely manner.

[0033] The Raspberry Pi embedded platform connects to the vehicle's 5V power supply. After power is connected, it automatically starts and loads the monitoring program, completing system initialization without manual operation, thus improving ease of use. The platform has a built-in timing module to count the continuous running time of the system to realize the driver's continuous driving time statistics. When the preset time is reached, the external control module triggers an active rest reminder to further prevent fatigue driving.

[0034] The peripheral control module can be extended to connect vibration feedback components, flashing light components, and voice broadcast components to form multiple alarm combinations, adapting to driving environments with different noise levels. For example, when driving at high speeds and in noisy conditions, a combined alarm method of vibration feedback, flashing lights, and voice broadcast can be activated to ensure effective warning delivery. Simultaneously, the Raspberry Pi embedded platform also includes a wireless communication module to extend remote monitoring capabilities. This module uploads driver status monitoring data and dangerous behavior alarm information to a remote monitoring terminal, enabling extended applications in scenarios such as night shift duty and fleet management. This allows managers to monitor driver status in real time and enhance driving safety management.

[0035] The specific implementation process of the driver status monitoring system based on intelligent image recognition of the present invention is as follows: 1. Hardware Components Raspberry Pi Embedded Platform: The Raspberry Pi 4B model is selected. This platform has powerful processing capabilities and rich interfaces, supports USB and CSI interface expansion, and can directly connect to camera modules, peripheral components and wireless communication modules. It also supports 5V vehicle power supply, has low power consumption and strong stability, and is suitable for use in the vehicle environment. The platform has a built-in timing module, which can accurately count the continuous running time of the system, thereby realizing the continuous driving time statistics of the driver.

[0036] Video capture module: A CSI camera module is selected, with a resolution of 1080P and a frame rate of 30fps. It is fixed in the center of the vehicle's dashboard, and the angle is adjusted using a bracket to ensure that the camera is directly facing the driver's face, enabling clear capture of the driver's facial image. The camera module is connected to the Raspberry Pi embedded platform via a CSI interface, which provides fast transmission speed and reduces image transmission latency. At the same time, this module supports preliminary preprocessing operations such as image noise reduction, size normalization, and grayscale conversion. The resolution of the preprocessed image is normalized to 640×480, providing standard data for subsequent feature extraction.

[0037] Core algorithm processing module: A software module based on the Raspberry Pi embedded platform, with built-in lightweight facial landmark detection model and feature analysis logic algorithm; when deploying the model, it is configured through the Python environment of the Raspberry Pi, without the need to pre-install complex dependency libraries, achieving zero dependency deployment and reducing the integration threshold; the feature analysis logic algorithm judges the driver's state by extracting the coordinate changes of key points of the eyes, mouth and facial contours: for example, judging whether the eyes are closed by the degree of closure and duration of the eye key points, judging whether yawning by the degree of opening and duration of the mouth key points, judging whether the head is tilted down by the pitch angle of the facial contour key points, and judging whether the face is missing by the detection results of the facial key points, such as the driver tilting down and obscuring the face or leaving the driver's seat.

[0038] Peripheral control module: including voice broadcast component, using a small vehicle voice module, supporting customized voice broadcast content; display component, using a 3.5-inch TFT-LCD touch screen with a resolution of 480×320, connected to the Raspberry Pi platform via GPIO interface; also includes an extended connection vibration feedback component, installed on the back of the driver's seat; and a light flashing component, installed on the dashboard; forming a multi-mode combined alarm.

[0039] Wireless communication module: A WiFi module is selected, which connects to the Raspberry Pi embedded platform via a USB interface to realize the wireless uploading of monitoring data and alarm information. It can be connected to remote monitoring terminals, such as computers and mobile phone APPs.

[0040] 2. Software Flow System initialization: Connect the Raspberry Pi embedded platform to the vehicle's 5V power supply. After power is connected, the system will start automatically, load the monitoring program, and complete the initialization of the camera module, peripheral components, and wireless communication module. The initialization time will not exceed 10 seconds. After initialization, the system will enter standby mode and wait to collect the driver's facial image.

[0041] Facial Feature Calibration: Upon initial system operation or after a change in camera position, the system will prompt the driver to maintain a normal driving posture. The camera will capture 3-5 frames of normal facial images of the driver. The core algorithm processing module will extract facial key point features from these images and calculate baseline facial parameters, including the standard coordinates of key points of the eyes, mouth, and facial contours, as well as thresholds for eye opening, mouth opening, and facial pitch angle. These baseline parameters will be stored in the Raspberry Pi's local storage module as a reference for subsequent status judgments. The calibration process can be manually triggered for recalibration to ensure the accuracy of the baseline parameters.

[0042] Real-time monitoring and status assessment: The video acquisition module captures the driver's facial image frame by frame. After preliminary preprocessing, the image is transmitted to the core algorithm processing module. The core algorithm processing module extracts facial key point features using a lightweight facial key point detection model, compares and analyzes them with baseline facial parameters, and determines in real time whether the driver exhibits dangerous behaviors such as closing their eyes, looking down, yawning, or losing sight of their face. Eyes-closed judgment: When the opening and closing degree of key points of the eyes is lower than the baseline threshold and the duration exceeds 1 second, it is judged as dangerous behavior of closing the eyes; Head-down judgment: When the tilt angle of key points of facial contours is lower than the baseline threshold, that is, the head-down angle exceeds the preset value and the duration exceeds 2 seconds, it is judged as a dangerous head-down behavior. Yawning detection: When the opening of the mouth at key points exceeds the baseline threshold and lasts for more than 3 seconds, it is judged as a dangerous yawning behavior. Face loss detection: If no valid facial landmarks are detected for 5 consecutive frames, it is determined to be a dangerous behavior of face loss.

[0043] Fatigue Level Judgment and Warning Trigger: The core algorithm processing module distinguishes between mild and severe fatigue based on the duration and frequency of dangerous behaviors. Mild fatigue is defined as a single dangerous behavior occurring only once, or multiple times within a short period. Severe fatigue is defined as a single dangerous behavior lasting more than 5 seconds, or multiple dangerous behaviors occurring simultaneously. When mild fatigue is detected, the core algorithm processing module sends a low-intensity trigger signal to the peripheral control module, which triggers the voice broadcast component to announce "Please take a rest," the display component to display "Mild Fatigue Warning," and the light flashing component to blink slowly. When severe fatigue is detected, a high-intensity trigger signal is sent, triggering the voice broadcast component to repeatedly announce "Severe fatigue, please stop and rest immediately," the display component to display "Severe Fatigue Warning" and flash brightly, the vibration feedback component to activate vibration, and the light flashing component to blink rapidly, ensuring the driver's timely perception.

[0044] Continuous driving time reminder: The timing module of the Raspberry Pi embedded platform starts timing when the system starts up and counts the continuous running time in real time, that is, the driver's continuous driving time. When the continuous time reaches the preset value (such as 4 hours), the core algorithm processing module sends a trigger signal to the peripheral control module, triggering the voice broadcast "Continuous driving has reached 4 hours, please stop and rest", and the display component displays the rest reminder information, realizing proactive rest reminder.

[0045] Remote monitoring extension: The wireless communication module uploads driver status monitoring data, including normal driving status data, dangerous behavior data, fatigue level data, and continuous driving duration data, as well as dangerous behavior alarm information, including alarm time, alarm type, and fatigue level, to the remote monitoring terminal in real time. Managers can view the driver's status and receive alarm reminders through the terminal, enabling remote supervision in scenarios such as fleet management and night shift duty.

[0046] In this embodiment, the video acquisition module can be replaced with a USB camera, connected via the Raspberry Pi's USB interface, to accommodate different types of cameras. The peripheral control module can adjust the alarm mode combination according to the noise level of the driving environment: for example, when driving on city roads (where noise is low), only voice broadcast and display prompts can be enabled; when driving on highways (where noise is high), a combination of voice broadcast, vibration feedback, and flashing lights can be enabled. The wireless communication module can be replaced with a 5G module to achieve long-distance data transmission, adapting to the remote monitoring needs of long-distance operating vehicles.

[0047] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A driver status monitoring system based on intelligent image recognition, characterized in that: It includes a video acquisition module based on the Raspberry Pi embedded platform, a core algorithm processing module, and a peripheral control module. The video acquisition module is compatible with USB camera or CSI camera module and is used to acquire driver facial images and perform preliminary image preprocessing. The core algorithm processing module incorporates a lightweight facial key point detection model and feature analysis logic algorithm to achieve end-side reasoning, driver facial feature extraction, and state judgment. The peripheral control module is connected to an alarm component and a display component, and is used to receive judgment signals from the core algorithm processing module and trigger corresponding peripheral actions.

2. The driver status monitoring system based on intelligent image recognition according to claim 1, characterized in that: The core algorithm processing module extracts key features of the driver's eyes, mouth, and facial contours, and analyzes and judges in real time whether the driver exhibits potential dangerous behaviors such as closing their eyes, looking down, yawning, or losing their face. When any of the aforementioned dangerous behaviors are detected, a trigger signal is sent to the peripheral control module.

3. The driver status monitoring system based on intelligent image recognition according to claim 2, characterized in that: The camera module of the video acquisition module is fixed to the vehicle dashboard or a matching bracket, and the camera can be adjusted to clearly capture the driver's facial area.

4. The driver status monitoring system based on intelligent image recognition according to claim 3, characterized in that: The Raspberry Pi embedded platform is connected to the vehicle's 5V power supply. After the power is turned on, it automatically starts and loads the monitoring program to complete the initialization.

5. A driver status monitoring system based on intelligent image recognition according to claim 4, characterized in that: The core algorithm processing module has a facial feature calibration function. Upon first run or after the camera position changes, it captures the baseline facial parameters of the driver in a normal driving posture and uses the baseline facial parameters as a reference for subsequent driver status judgment.

6. A driver status monitoring system based on intelligent image recognition according to claim 5, characterized in that: The alarm component of the peripheral control module is a voice broadcast component, which immediately issues a voice alarm upon receiving a trigger signal; upon receiving the trigger signal, the display component synchronously displays the corresponding warning prompt for dangerous behavior, and the voice alarm and the screen prompt are triggered simultaneously.

7. A driver status monitoring system based on intelligent image recognition according to claim 6, characterized in that: The core algorithm processing module has a fatigue risk level judgment mechanism, which distinguishes between mild fatigue and severe fatigue based on the duration and frequency of the driver's dangerous behavior, and sends trigger signals of different intensities to the peripheral control module, so that the peripheral control module outputs the corresponding level of reminder action.

8. A driver status monitoring system based on intelligent image recognition according to claim 7, characterized in that: The Raspberry Pi embedded platform has a built-in timing module for counting the continuous running time of the system to realize the driver's continuous driving time statistics, and triggers an active rest reminder through the peripheral control module when the preset time is reached.

9. A driver status monitoring system based on intelligent image recognition according to claim 8, characterized in that: The peripheral control module can be extended to connect vibration feedback components, light flashing components, and form multiple alarm combinations with voice broadcast components and display components to adapt to driving environments with different noise levels.

10. A driver status monitoring system based on intelligent image recognition according to claim 9, characterized in that: The Raspberry Pi embedded platform also includes a wireless communication module for extending remote monitoring functionality. This module uploads driver status monitoring data and dangerous behavior alarm information to the remote monitoring terminal, enabling extended applications in multiple scenarios such as night shift guard duty and fleet management.

Images