A lightweight driver fatigue monitoring method and system

Abstract

The application provides a lightweight driver fatigue monitoring method and system, relates to the technical field of safe driving, and comprises the following steps: collecting the light intensity in the vehicle, adjusting the light intensity based on a set threshold; collecting and preprocessing the real-time image of the driving position area in the vehicle; constructing a lightweight training model; determining the candidate area in the image based on the mask concept, identifying the key parts; setting a determination threshold for the change degree of the key parts, issuing an alarm when the determination index reaches the limit value; bypass shielding and local calibration for false alarm conditions. The light intensity control algorithm is proposed, the light is intelligently adjusted, the light is supplemented or reduced, the fatigue detection area is subtracted, the interference and false alarm rate are reduced to a certain extent, and the occupancy rate of the vehicle-mounted computing resources is reduced.

Description

Technical Field

[0001] This application relates to the field of safe driving technology, and in particular to a lightweight method and system for monitoring driver fatigue. Background Technology

[0002] Driver fatigue refers to the physiological or psychological dysfunction experienced by a driver while driving, resulting in reduced perception of the surrounding environment and decreased vehicle control, leading to deviations from normal driving behavior. Therefore, proactively monitoring driver condition and effectively providing driver fatigue warnings is a pressing traffic safety issue that needs to be addressed.

[0003] Most existing Driver Monitoring Systems (DMS) operate on the principle of processing visual signals and judging driver status through feature extraction. However, in practical applications, some inherent problems exist. For example, video images captured by in-vehicle cameras, the most important data source, are easily affected by ambient light. In strong light, backlight, or low light environments, image quality will decrease to varying degrees, affecting the judgment of driver status. The PERCLOS (Percentage of Eyelid Closure over the pupil over time) algorithm, which subtracts two adjacent frames of facial images captured by the camera sensor to perform image binarization and segment the facial region, has a certain effect. However, if the person shakes their head or changes body position significantly, the recognition effect will be affected. While keypoint localization methods combining mainstream deep neural network algorithms can achieve relatively accurate results, the entire deep learning process, from dataset labeling to training and inference, is very labor-intensive. Furthermore, it consumes a lot of computing resources when deployed on the vehicle, placing high demands on the in-vehicle equipment. Summary of the Invention

[0004] This application addresses the aforementioned problems and aims to provide a lightweight driver fatigue monitoring method and system. By monitoring the light intensity inside the vehicle, a light intensity control algorithm is proposed to intelligently fine-tune the control of supplemental or reduced lighting, thereby reducing the fatigue detection area by eliminating weakly correlated parts. This allows the system to focus on the most critical areas and, by incorporating the concept of "eye diagrams" in communication, to determine the state of the eyes and mouth. This reduces interference and false alarm rates to some extent, lowers the occupancy of onboard computing resources, and allows more computing resources to be allocated to vehicle control logic.

[0005] Specifically, the first aspect of this application provides a lightweight driver fatigue monitoring method, comprising the following steps:

[0006] Step 1: Collect the light intensity inside the vehicle. Based on the set light intensity threshold range, adjust the light intensity using an adjustment device when the light intensity is outside the range.

[0007] The system uses a light intensity sensor to detect the ambient light intensity inside and outside the vehicle. Utilizing a dual-threshold mechanism, it detects when the light intensity inside the vehicle exceeds a certain threshold. Lux, the driver adjusts the angle of the neutral density filter to reduce the amount of light passing through the camera lens; the interior light intensity is lower than... In Lux mode, several infrared LEDs are activated to provide supplementary lighting for the camera's video capture. and It should have a relatively obvious difference.

[0008] Both the neutral density (ND) filter and the infrared LED are mounted on the image acquisition module. The ND filter's light reduction effect is adjusted by rotating its angle; each rotation angle... The camera will then reduce the amount of light accordingly. Lux, default 0° no light reduction; there are N infrared lights, each of which, when lit, effectively adds power to the camera. Lux light intensity controls the number of lights illuminated based on the interior light intensity; by default, all lights are off with no additional illumination.

[0009] Step 2: Real-time image acquisition and preprocessing of the driver's seat area inside the vehicle;

[0010] Step 3: Build a lightweight training model, preprocess the training portrait data and annotate key parts, and use the data to train the model;

[0011] Step 4: Based on the concept of masking, candidate regions in the driver's seat area image data are determined, and key parts are identified through lightweight training model;

[0012] Step 5: Set a judgment threshold for the degree of change in key parts, and set the judgment indicator exceeding the limit value per unit time. When the judgment indicator exceeds the limit value, an alarm will be issued.

[0013] Step Six: Perform bypass shielding and local calibration for false alarms;

[0014] Furthermore, in step one, the formula for adjusting the light intensity is as follows:

[0015] ;

[0016] in: Light-intensity decision-making strategy;

[0017] The switching coefficient is used to control the dimming adjustment strategy, and the elements are in the set {0,1}.

[0018] The switching coefficient is controlled by a strategy to enhance brightness, and the elements are in the set {0,1}.

[0019] This represents the actual light intensity inside the vehicle.

[0020] This represents the upper limit of the light intensity threshold.

[0021] The amount of light reduced when the light-reducing device is adjusted by a unit angle;

[0022] To adjust the angle;

[0023] This is the lower limit of the light intensity threshold;

[0024] The amount of light added when each supplementary lighting device is turned on;

[0025] To round down;

[0026] This is for rounding up.

[0027] like That is, the light intensity inside the car is higher than At that time, the image may be overexposed, then in the formula , Instead of using a supplemental lighting strategy, only the neutral density filter is adjusted to reduce the light.

[0028] like That is, the light intensity inside the car is lower than At that time, it may be difficult to see clearly, then in the formula , Instead of implementing a light reduction strategy, only adjust the fill light.

[0029] Assuming that the light intensity inside the vehicle can reach a moderate range after the light intensity adjustment strategy, we will not consider cases where the light intensity cannot be outside the moderate range after adjustment.

[0030] like That is, when the light intensity inside the car is moderate, there is no need to adjust the amount of light entering the camera. , No need to adjust the light reduction or supplemental lighting strategy.

[0031] Furthermore, the preprocessing specifically includes image grayscale conversion and binarization sparsity.

[0032] The raw data is first stored in local storage module 1 and then transmitted to the recognition and processing module for core function processing. After obtaining a fixed number of video frames, they are grouped into a batch and uniformly and in parallel grayscale processing is performed to obtain the corresponding binary image. This transforms the R, G, and B three-channel image into a single-channel image, reducing the amount of data by 2 / 3 of the dimensions.

[0033] Binarization sparsity specifically involves further sparsifying the image using pixel thresholding to extract non-blank information. By adjusting an empirical threshold to distinguish the opening and closing states of eyes and mouths in most images, images below this threshold are filtered out, while those above are retained, resulting in a binarized image. Binarization can further reduce memory usage and processing overhead, while also reducing noise interference.

[0034] Furthermore, the image grayscale processing specifically involves compressing the R, G, and B three-channel image into a single-channel image. The formula for the value of each pixel during the compression process is as follows:

[0035] ;

[0036] in: It is a single-channel image;

[0037] These are all regulatory factors used to extract key components;

[0038] Image of the red channel;

[0039] Image of the green channel;

[0040] This is the blue channel image.

[0041] Adding a regulatory factor as an adjustable parameter to the classic conversion formula aims to clearly extract the key parts of the eyes and lips. For parts such as the eyes and lips, red and green primary colors are used as the main colors, and the weight value of the regulatory factor can be appropriately increased.

[0042] Furthermore, the annotation of key parts specifically includes: since the elements in the human image data are relatively concentrated, the eyes and mouth, which show significant differences in facial expression compared to when the driver is drowsy and awake, are selected as key parts, and these key parts are marked and identified.

[0043] The eyes and mouth show significant differences in facial expressions when a driver is drowsy compared to when they are fully awake, resulting in high reliability of the recognition results. Therefore, these two areas are the focus of recognition. Supervised learning training is implemented by marking the driver's eyes and mouth in the image.

[0044] The selection of training image data is not limited to driver images; any image containing a human head can be used as the training set, with a primary focus on the state of the eyes and mouth. Image binarization is mainly for compressing storage space and reducing processing overhead. However, during manual annotation, binarized images may be less clear than the original images, hindering accurate annotation. Therefore, during the training phase, the original images are directly annotated to obtain the position information of the annotation label boxes. The binarized images are then used as the corresponding images in the training set, completing the mapping of the training set.

[0045] Furthermore, the process of determining candidate regions for image data based on the concept of a mask specifically includes: using the image captured by the driver under normal driving conditions as a mask according to the camera's installation orientation, and taking full account of the driver's body tilt margin to define the driver's position range, while clearing the remaining areas.

[0046] Furthermore, the body tilt margin is specifically defined as expanding the driver's image outwards by 5% to 15%.

[0047] The image acquisition module primarily captures the driver's upper body, but also includes the area behind them. If there are passengers in the back seat, they will also be captured, interfering with driver recognition. Therefore, borrowing the concept of a "mask" commonly used in image processing, the image captured under normal driving conditions is used as a mask based on the camera's installation position, taking into full account the driver's body tilt margin to roughly determine the driver's position. The remaining areas are cleared to reduce recognition interference from rear-seat passengers appearing in the frame. It is important to note that the image is not cropped to maintain consistent dimensions for unified processing.

[0048] After being trained by a convolutional neural network, the system can be put into use and collect driver image information. It can then use reasoning to identify the opening and closing states of the driver's eyes and mouth, thus making a preliminary judgment on the driver's state at each moment.

[0049] Furthermore, the setting of a judgment threshold for the degree of change in key parts and the setting of the limit value of the judgment index per unit time specifically includes: drawing on the concept of "eye diagram" in communication, setting a judgment threshold for eye closing and mouth opening, prioritizing the number of blinks as the judgment index, and taking the number of mouth openings as a progressive supplementary judgment index, and setting the limit value of these two indicators.

[0050] Furthermore, the alarm is triggered when the judgment indicator exceeds the limit. Specifically, this includes: using a progressive judgment method, first judging the number of blinks within a unit of time. If it exceeds the limit, it is considered that the driver may have fatigue problems and an alarm is triggered; if it does not exceed the limit, it turns to the number of mouth openings. If this indicator exceeds the limit, the system judges that the driver may have fatigue problems and an alarm is triggered.

[0051] After training, the system can recognize the states of eyes open and closed, and mouth open and closed. Borrowing the concept of an "eye diagram" from communications, thresholds are set for eye closure and mouth opening. A progressive judgment method is used per unit time, prioritizing the number of blinks as the judgment indicator, and then using the number of mouth openings as a supplementary indicator. That is, the system first judges the number of blinks; if it exceeds the limit, it assumes the driver may be fatigued and issues an alarm; if it does not exceed the limit, it then judges the number of mouth openings; if this indicator exceeds the limit, the system judges the driver may be fatigued and issues an alarm. The determination of the threshold values ​​for both indicators is based on extensive biological research, and relatively mature recommended reference values ​​can be used.

[0052] When a driver is eating or drinking while covering their mouth, or wearing sunglasses while obscuring their eyes, making it impossible to detect the mouth or eyes, this is recorded as an abnormal situation. In rare cases, if the driver has physiological characteristics or habits such as small eyes or habitually opening their mouth, false alarms can be bypassed and the system can be fed back. During idle periods, the system uses data stored during driving, combined with driver feedback, for secondary training to calibrate the system's judgment.

[0053] A second aspect of this application provides a lightweight driver fatigue monitoring system, comprising:

[0054] Light sensing module: used to monitor the light intensity inside the vehicle in real time and transmit the signal to the recognition and processing module;

[0055] Lighting and light reduction module: Used to control the infrared lighting and light reduction mirror, and adjust the light intensity according to the light intensity control calculation of the recognition and processing module;

[0056] Image acquisition module: monitors key facial features of the driver, acquires raw data, and sends it to the recognition processing module and local storage module 1;

[0057] Local storage module 1: Stores the raw data transmitted from the image acquisition module;

[0058] Recognition processing module: used for light intensity control calculation and facial key part recognition training, controls the reminder and warning module when fatigue is detected and judged, and uses local storage module 2 data for local calibration training when false alarms are reported;

[0059] Local storage module 2: Used to store the result data and false alarm information data after the identification processing module processes the original data;

[0060] Server: Used to remotely store raw detection data, identification and processing result data, and false alarm information data, to perform secondary verification of decision results, and is electrically connected to the identification and processing module;

[0061] Alert and warning module: Used to provide voice and vibration alerts when fatigue is detected and assessed;

[0062] Bypass module: Used to manually disable alerts when false alarms occur in the system.

[0063] Secondly, this application also provides a computing device that has the functionality to implement the method described in the first aspect. The beneficial effects are described in the first aspect and will not be repeated here. The functionality can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functionality. In one possible design, the device structure includes an acquisition module, a training module, and optionally, a construction module. These modules can implement the function of training nodes in the method example of the first aspect, as detailed in the method example, and will not be repeated here.

[0064] Thirdly, this application also provides a computing device for implementing the functions of the method described in the first aspect above. The beneficial effects are described in the first aspect and will not be repeated here. The computing device includes a processor and a memory, with the memory storing instructions and / or data. The memory is coupled to the processor, and when the processor executes the program instructions stored in the memory, it can implement the function of the training node in the example of the first aspect above. The computing device also includes a communication interface for communicating with other devices.

[0065] Fourthly, this application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect and various possible designs of the first aspect.

[0066] Fifthly, this application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods described in the first aspect and various possible designs of the first aspect.

[0067] In a sixth aspect, this application also provides a computing chip connected to a memory, which is used to read and execute a software program stored in the memory to perform the methods described in the first aspect and various possible implementations of the first aspect. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this drawing or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this drawing. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0069] Figure 1 A flowchart illustrating the steps of a lightweight driver fatigue monitoring method;

[0070] Figure 2 A detailed flowchart of a lightweight driver fatigue monitoring method;

[0071] Figure 3 A schematic diagram of data acquisition markings for a lightweight driver fatigue monitoring method;

[0072] Figure 4 A schematic diagram illustrating the discrimination limits for key components in a lightweight driver fatigue monitoring method;

[0073] Figure 5 A schematic diagram of image changes during the process of a lightweight driver fatigue monitoring method;

[0074] Figure 6 A schematic diagram illustrating the technical means of a lightweight driver fatigue monitoring method;

[0075] Figure 7 A schematic diagram of the structural composition of a lightweight driver fatigue monitoring system;

[0076] The purpose, features, and advantages of this accompanying drawing will be further explained in conjunction with the embodiments and with reference to the accompanying drawing. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0078] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0079] Unless otherwise specified, all embodiments and optional embodiments of this application can be combined to form new technical solutions.

[0080] Unless otherwise specified, all technical features and optional technical features of this application may be combined to form new technical solutions.

[0081] Unless otherwise specified, all steps in this application may be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), indicating that the method may include steps (a) and (b) performed sequentially, or it may include steps (b) and (a) performed sequentially. For example, the mention that the method may also include step (c) indicates that step (c) may be added to the method in any order. For example, the method may include steps (a), (b), and (c), or it may include steps (a), (c), and (b), or it may include steps (c), (a), and (b), etc.

[0082] Unless otherwise specified, the terms "comprising" and "including" as used in this application can be open-ended or closed-ended. For example, "comprising" and "including" can mean that other components not listed may also be included, or that only the listed components may be included.

[0083] Unless otherwise specified, the term "or" is inclusive in this application. For example, the phrase "A or B" means "A, B, or both A and B". More specifically, the condition "A or B" is satisfied by any of the following conditions: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); or both A and B are true (or exist).

[0084] To better understand the solutions of the embodiments of this application, some related terms and concepts that may be involved in the embodiments of this application will be introduced below.

[0085] (1) Artificial intelligence (AI), also known as intelligent machinery or machine intelligence, refers to machines created by humans that can exhibit intelligence. Generally, artificial intelligence refers to the technology of presenting human intelligence through ordinary computer programs.

[0086] (2) Machine Learning (ML): Machine learning is the core of artificial intelligence. Machine learning theory mainly involves designing and analyzing algorithms that allow computers to learn automatically. Machine learning algorithms are a class of algorithms that automatically analyze data to obtain patterns and use these patterns to predict unknown data. Therefore, the core of machine learning is data, algorithms (models), and computing power (computer processing ability). Machine learning has a wide range of applications, including data mining, data classification, computer vision, natural language processing (NLP), biometrics, search engines, medical diagnosis, credit card fraud detection, securities market analysis, DNA sequencing, speech and handwriting recognition, strategy games, and robotics. Machine learning involves designing an algorithm model to process data and output the desired results. Users can continuously optimize the algorithm model to achieve more accurate data processing capabilities.

[0087] (3) Deep Learning (DL): Deep learning is a method of machine learning. Its concept originated from the research of artificial neural networks. A multilayer perceptron with multiple hidden layers is a type of deep learning structure, so deep learning is often referred to as a deep neural network. Compared with general machine learning, deep learning can automatically extract features, that is, automatically combine simple features into more complex features, and use these combinations to learn multilayer weights to solve problems. The motivation for researching deep learning is to build neural networks that simulate the human brain's analytical learning, which imitate the mechanisms of the human brain to interpret data, such as images, sounds, and text. Deep learning first emerged in image recognition, but in just a few years, it has been promoted to various fields of machine learning and has performed excellently. It has applications in image recognition, speech recognition, audio processing, natural language recognition, robotic bioinformatics processing, search engines, human-computer interaction, targeted online advertising, automated medical diagnosis, and finance, among other major fields.

[0088] (4) Neural networks (NNs) are mathematical models that mimic the behavioral characteristics of animal neural networks to perform distributed parallel information processing. They can process information by adjusting the connections between a large number of nodes within the neural network, and possess self-learning and adaptive capabilities. Neural networks are commonly used in artificial intelligence model training and data derivation processing.

[0089] Specifically, neural networks typically contain multiple interconnected layers, such as convolutional layers, fully connected layers (FC), activation layers, or pooling layers.

[0090] (5) Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with multiple hidden layers. DNNs can be divided into three categories based on the position of each layer: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs seem complex, the operation of each layer is actually quite simple, represented by the following linear relationship: where is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer simply performs this simple operation on the input vector to obtain the output vector.

[0091] (6) A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer refers to the layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn reasonable weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0092] In this embodiment, as Figure 1 As shown, a lightweight driver fatigue monitoring method includes the following steps:

[0093] Step 1: Collect the light intensity inside the vehicle. Based on the set light intensity threshold range, adjust the light intensity using an adjustment device when the light intensity is outside the range.

[0094] The system uses a light intensity sensor to detect the ambient light intensity inside and outside the vehicle. Utilizing a dual-threshold mechanism, it detects when the light intensity inside the vehicle exceeds a certain threshold. Lux, the driver adjusts the angle of the neutral density filter to reduce the amount of light passing through the camera lens; the interior light intensity is lower than... In Lux mode, several infrared LEDs are activated to provide supplementary lighting for the camera's video capture. and It should have a relatively obvious difference.

[0095] Both the neutral density (ND) filter and the infrared LED are mounted on the image acquisition module. The ND filter's light reduction effect is adjusted by rotating its angle; each rotation angle... The camera will then reduce the amount of light accordingly. Lux, default 0° no light reduction; there are N infrared lights, each of which, when lit, effectively adds power to the camera. Lux light intensity controls the number of lights illuminated based on the interior light intensity; by default, all lights are off with no additional illumination.

[0096] Step 2: Real-time image acquisition and preprocessing of the driver's seat area inside the vehicle;

[0097] Step 3: Build a lightweight training model, preprocess the training portrait data and annotate key parts, and use the data to train the model;

[0098] Step 4: Based on the concept of masking, candidate regions in the driver's seat area image data are determined, and key parts are identified through lightweight training model;

[0099] Step 5: Set a judgment threshold for the degree of change in key parts, and set the judgment indicator exceeding the limit value per unit time. When the judgment indicator exceeds the limit value, an alarm will be issued.

[0100] Step Six: Perform bypass shielding and local calibration for false alarms;

[0101] Furthermore, in step one, the formula for adjusting the light intensity is as follows:

[0102] ;

[0103] like That is, the light intensity inside the car is higher than At that time, the image may be overexposed, then in the formula , Instead of using a supplemental lighting strategy, only the neutral density filter is adjusted to reduce the light.

[0104] like That is, the light intensity inside the car is lower than At that time, it may be difficult to see clearly, then in the formula , Instead of implementing a light reduction strategy, only adjust the fill light.

[0105] Assuming that the light intensity inside the vehicle can reach a moderate range after the light intensity adjustment strategy, we will not consider cases where the light intensity cannot be outside the moderate range after adjustment.

[0106] like That is, when the light intensity inside the car is moderate, there is no need to adjust the amount of light entering the camera. , No need to adjust the light reduction or supplemental lighting strategy.

[0107] In this embodiment, a detailed flowchart of a lightweight driver fatigue monitoring method is shown below. Figure 2 The specific process of the fatigue testing method provided in this application is illustrated in the figure.

[0108] Further, preprocessing specifically includes image grayscale conversion and binarization sparsity reduction.

[0109] The raw data is first stored in local storage module 1 and then transmitted to the recognition and processing module for core function processing. After obtaining a fixed number of video frames, they are grouped into a batch and uniformly and in parallel grayscale processing is performed to obtain the corresponding binary image. This transforms the R, G, and B three-channel image into a single-channel image, reducing the amount of data by 2 / 3 of the dimensions.

[0110] Binarization sparsity specifically involves further sparsifying the image using pixel thresholding to extract non-blank information. By adjusting an empirical threshold to distinguish the opening and closing states of eyes and mouths in most images, images below this threshold are filtered out, while those above are retained, resulting in a binarized image. Binarization can further reduce memory usage and processing overhead, while also reducing noise interference.

[0111] Furthermore, the image grayscale processing specifically involves compressing the R, G, and B three-channel image into a single-channel image. The formula for the value of each pixel during the compression process is as follows:

[0112] ;

[0113] In this embodiment, a regulatory factor is added as an adjustable parameter to the classic conversion formula. The purpose is to clearly extract the key parts of the eyes and lips. For parts such as the eyes and lips, the red and green primary colors are used as the main colors, and the weight value of the regulatory factor can be appropriately increased.

[0114] Furthermore, key areas are labeled, specifically including: since the elements in the facial data are relatively concentrated, the eyes and mouth, which show significant differences in facial expression compared to when the driver is drowsy and awake, are selected as key areas, and these key areas are marked and identified.

[0115] In this embodiment, a schematic diagram illustrating the changes in the acquired image during processing is shown below. Figure 5 As shown, the specific techniques used in image processing are as follows: Figure 6 As shown.

[0116] The eyes and mouth show significant differences in facial expressions when a driver is drowsy compared to when they are fully awake, resulting in high reliability of the recognition results. Therefore, these two areas are the focus of recognition. Supervised learning training is implemented by marking the driver's eyes and mouth in the image.

[0117] The selection of training image data is not limited to driver images; any image containing a human head can be used as the training set, with a primary focus on the state of the eyes and mouth. Binarization of the images is mainly for compressing storage space and reducing processing overhead. However, during manual annotation, binarized images may be less clear than the original images, hindering accurate annotation. Therefore, during the training phase, the original images are directly annotated to obtain the position information of the annotation label boxes. The binarized images are then used as the corresponding images in the training set, completing the mapping of the training set. The image data collection and labeling process is as follows: Figure 3 As shown.

[0118] Furthermore, candidate regions for image data are determined based on the concept of a mask. Specifically, this includes: using images captured by the driver under normal driving conditions as a mask according to the camera's installation orientation, taking full account of the driver's body tilt margin, defining the driver's position range, and clearing the remaining areas.

[0119] The image acquisition module primarily captures the driver's upper body, but also includes the area behind them. If there are passengers in the back seat, these areas will also be captured, interfering with driver recognition. Therefore, drawing on the concept of a "mask" commonly used in image processing, the image captured under normal driving conditions is used as a mask based on the camera's installation orientation, while fully considering the driver's body tilt margin.

[0120] In this embodiment, the body tilt margin is 10% of the dimension extending outwards from the driver, roughly defining the driver's position. The remaining area is cleared to reduce recognition interference caused by rear passengers appearing in the frame. It is important to note that the image is not cropped to maintain consistency in size before and after processing.

[0121] After being trained by a convolutional neural network, the system can be put into use and collect driver image information. It can then use reasoning to identify the opening and closing states of the driver's eyes and mouth, thus making a preliminary judgment on the driver's state at each moment.

[0122] Furthermore, thresholds are set for the degree of change in key areas, and limits are set for the judgment indicators within a unit of time. Specifically, this includes: drawing on the concept of "eye diagram" in communications, setting thresholds for eye closure and mouth opening, since both eyes and mouths are divided into upper and lower edges and are basically symmetrical about the midline, thus reducing workload, only the upper edge needs to be studied. The maximum opening area (maximum distance from the midline of the eye diagram) and the minimum closing area (minimum distance from the midline of the eye diagram) of the upper edge of the eye diagram are detected separately. If the number of times the upper edge of the eye diagram moves from the maximum opening area to the minimum closing area exceeds c times within a unit of time, it is considered that the blinking frequency is too high, indicating fatigue driving. If the upper edge of the eye diagram remains in the minimum closing area within a period of time Δt, it is also considered fatigue driving. Similarly, the judgment for the mouth eye diagram is the same, except that if the upper edge of the mouth eye diagram remains in the maximum opening area, it is considered fatigue driving, which is the opposite of the logic for the eye eye diagram. The number of blinks is given priority as the judgment indicator, and the number of mouth openings is taken as a progressive supplementary judgment indicator, and limits are set for the over-limit of these two indicators.

[0123] Furthermore, the system determines whether an indicator exceeds its limit and issues an alarm. Specifically, it uses a progressive determination method, first determining the number of blinks within a unit of time. If the number exceeds the limit, the system assumes the driver may be fatigued and issues an alarm. If the number does not exceed the limit, the system determines the number of times the driver opens their mouth wide. If this indicator exceeds the limit, the system assumes the driver may be fatigued and issues an alarm.

[0124] After training, the system can recognize when the eyes are open and closed, and when the mouth is open and closed. For example... Figure 4 As shown, this application borrows the concept of "eye diagram" in communications and sets a threshold for determining whether the eyes are closed or the mouth is open, i.e., when the eyes are closed... Figure 4 The midline is the threshold for judgment; the mouth must be opened wide enough to reach the specified threshold. Figure 4 The maximum value of the vertical line is the judgment threshold. A progressive judgment method is used per unit time, prioritizing the number of blinks as the judgment indicator, and using the number of mouth openings as a supplementary progressive judgment indicator. That is, the blink count is judged first; if it exceeds the limit, the system assumes the driver may be fatigued and issues an alarm; if it does not exceed the limit, the system moves to the number of mouth openings. If this indicator exceeds the limit, the system judges the driver may be fatigued and issues an alarm. There is considerable biological research on determining the limits for both indicators, and relatively mature recommended reference values ​​can be used. In this embodiment, 20 blinks per minute is considered an excessive value, and 2 mouth openings per minute is considered an excessive value.

[0125] When a driver is eating or drinking while covering their mouth, or wearing sunglasses while obscuring their eyes, making it impossible to detect the mouth or eyes, this is recorded as an abnormal situation. In rare cases, if the driver has physiological characteristics or habits such as small eyes or habitually opening their mouth, false alarms can be bypassed and the system can be fed back. During idle periods, the system uses data stored during driving, combined with driver feedback, for secondary training to calibrate the system's judgment.

[0126] like Figure 7 As shown, a second aspect of this application provides a lightweight driver fatigue monitoring system, comprising:

[0127] Light sensing module: used to monitor the light intensity inside the vehicle in real time and transmit the signal to the recognition and processing module;

[0128] Lighting and light reduction module: Used to control the infrared lighting and light reduction mirror, and adjust the light intensity according to the light intensity control calculation of the recognition and processing module;

[0129] Image acquisition module: monitors key facial features of the driver, acquires raw data, and sends it to the recognition processing module and local storage module 1;

[0130] Local storage module 1: Stores the raw data transmitted from the image acquisition module;

[0131] Recognition processing module: used for light intensity control calculation and facial key part recognition training, controls the reminder and warning module when fatigue is detected and judged, and uses local storage module 2 data for local calibration training when false alarms are reported;

[0132] Local storage module 2: Used to store the result data and false alarm information data after the identification processing module processes the original data;

[0133] Server: Used to remotely store raw detection data, identification and processing result data, and false alarm information data, to perform secondary verification of decision results, and is electrically connected to the identification and processing module;

[0134] Alert and warning module: Used to provide voice and vibration alerts when fatigue is detected and assessed;

[0135] Bypass module: Used to manually disable alerts when false alarms occur in the system.

[0136] It should be noted that this application is not limited to the above-described embodiments. The above embodiments are merely examples, and any embodiments with the same structure and effect as the technical concept within the scope of this application are included in the technical scope of this application. Furthermore, various modifications that can be conceived by those skilled in the art to the embodiments, and other ways of constructing by combining some of the constituent elements of the embodiments, without departing from the spirit of this application, are also included in the scope of this application.

Claims

1. A lightweight driver fatigue monitoring method characterized by, The method comprises the following steps: Step 1: Collect the light intensity in the vehicle, and based on the set light intensity threshold range, adjust the light intensity by adjusting the device when the light intensity is not in the range, wherein the light intensity adjustment formula is: ； wherein is a light intensity regulation decision policy; to control the on-off relationship of the dimming adjustment strategy, for the element in the set {0, 1}; For the gain adjustment strategy control on relationship number, for the set {0,1} element; Ii is the actual light intensity in the car; is an upper light intensity threshold value; light quantity reduced when adjusting a unit angle for the light reduction device; To adjust the angle; Ith is the lower threshold of the light intensity; the light amount added for each lighting of the light supplement device; rounded down; is rounded up; If No light compensation strategy is performed, only dimmer dimming is adjusted. If No light reduction strategy is performed, only the fill light is adjusted. If There is no need to adjust the dimming or light compensation strategy; Step 2: Real-time image acquisition and preprocessing of the driver's seat area in the vehicle; Step 3: Construct a lightweight training model, preprocess the portrait data for training, and label the key parts, and use the data to train the model; Step 4: Determine the candidate region in the image data of the driver's seat area based on the mask concept, and identify the key parts through the lightweight training model; The specific steps of determining the candidate region in the image data based on the mask concept include: according to the installation direction of the camera, taking the image collected under the normal driving condition of the driver as a mask, and fully considering the body tilt allowance, delimiting the driver's position range, and emptying the rest; Step 5: Referring to the eye diagram concept in communication, set the determination threshold for eye closure and mouth opening, preferentially take the number of eye blinks as the determination index, and take the number of mouth opening as the progressive supplementary determination index, and set the over-limit value of the two indexes, and the alarm is issued when the determination index reaches the over-limit value; Step 6: Bypass shielding and local calibration for false positives.

2. The method of claim 1, wherein The preprocessing specifically includes image grayscale processing and binary sparse.

3. The method of claim 2, wherein The image grayscale processing specifically compresses and converts the R, G, and B three-channel images into a single-channel image.

4. The method of claim 1, wherein The labeling of the key parts specifically includes: the elements in the portrait data are relatively concentrated, and the eyes and mouth, which have significant differences in facial expressions between the driver's drowsiness and normal wakefulness, are selected as the key parts, and the key parts are labeled and identified.

5. The method of claim 1, wherein The body tilt allowance specifically extends the driver's image by 5% to 15% around.

6. The method of claim 1, wherein The alarm is issued when the determination index reaches the over-limit value, which specifically includes: using a progressive determination method, first determine the number of eye blinks within a unit time, if it is over-limit, it is considered that the driver may have fatigue problem, and an alarm is issued; if it is not over-limit, turn to the number of mouth opening, if the index is over-limit, the system determines that the driver may have fatigue problem, and an alarm is issued.

7. A lightweight driver fatigue monitoring system characterized by, The system is used to realize the lightweight driver fatigue monitoring method according to any one of claims 1-6, comprising: An illumination sensing module for real-time monitoring of the light intensity in the vehicle, transmitting signals to the recognition processing module; A light supplementing and reducing module for controlling the infrared light supplementing lamp and the light reducing mirror, and adjusting the light intensity according to the light intensity control calculation of the recognition processing module; An image acquisition module for monitoring the key features of the driver's face, obtaining original data, and sending to the recognition processing module and the local storage module 1; A local storage module 1 for storing the original data transmitted by the image acquisition module; A recognition processing module for light intensity control calculation and face key part recognition training, controlling the warning module when detecting fatigue, and using the local storage module 2 data for local calibration training when false positives are fed back; A local storage module 2 for storing the result data and false information data of the original data processed by the recognition processing module; A server is configured to remotely store original detection data, identification processing result data, false alarm information data, and perform secondary verification of decision results, and is electrically connected with the identification processing module; A reminding and warning module is configured to perform voice and vibration prompting when fatigue is detected and determined; A bypass module is configured to manually close the reminding and warning through the bypass module when a false alarm occurs in the system.

Images