Driver fatigue detection method, system, device and storage medium

Abstract

The present application relates to a driver fatigue detection method, system, device and storage medium, the method comprising: obtaining a driver's facial image; identifying an eye region image in the facial image according to a pre-trained eye recognition model, and identifying a mouth region image in the facial image according to a pre-trained mouth recognition model; identifying the posture-related information of the driver's eyes and mouth through an image sequence based on the eye region image and the mouth region image; detecting whether the driver is in a fatigue state based on the posture-related information of the driver's eyes and mouth and based on preset rules. The present application can solve the technical problem of low accuracy in driver fatigue detection.

Description

Technical Field

[0001] The present application relates to the technical field of fatigue status monitoring, and in particular to a driver fatigue detection method, system, device and storage medium. Background Art

[0002] At present, with the rapid development of the economy, the number of motor vehicles has increased rapidly. When people are tired, their reactions become slow and their body control ability decreases. In certain environments, such as fatigue driving, drivers will have inattention and other phenomena, unable to accurately judge and quickly deal with various abnormal situations, which can easily cause traffic accidents. Therefore, it is very necessary to monitor the fatigue state of drivers while driving motor vehicles.

[0003] The existing technology generally adopts a solution of counting driving time, and if the continuous driving time is too long, it is judged that the driver is driving fatigued. However, different drivers may have certain differences in the fatigue caused by driving time. Therefore, this detection method based only on continuous driving time is not targeted for identifying whether the driver is fatigued, and the recognition accuracy is low. Summary of the invention

[0004] In view of this, the present application provides a driver fatigue detection method, system, device and storage medium to solve the technical problem of low accuracy of driver fatigue detection.

[0005] In order to solve the above problems, in a first aspect, the present application provides a driver fatigue detection method, the method comprising:

[0006] Acquire a facial image of the driver;

[0007] Identify an eye region image in the facial image according to a pre-trained eye recognition model, and identify a mouth region image in the facial image according to a pre-trained mouth recognition model;

[0008] Based on the eye region image and the mouth region image, respectively identifying the posture-related information of the driver's eyes and mouth through the image sequence;

[0009] Based on the information related to the driver's eye and mouth postures and based on preset rules, it detects whether the driver is in a state of fatigue.

[0010] Optionally, obtaining the driver's facial image includes:

[0011] Obtaining image information of the driver collected in real time by a camera installed in the cab;

[0012] The collected image information is recognized using a face recognition algorithm based on deep learning to obtain the driver's facial image.

[0013] Optionally, the face recognition algorithm based on deep learning is a convolutional neural network algorithm, and the step of using the face recognition algorithm based on deep learning to recognize the collected image information to obtain the driver's facial image includes:

[0014] The convolutional neural network algorithm is used to extract features from the collected image information, and the different convolutional layer features in the convolutional neural network are connected in stages to obtain the extracted feature map;

[0015] The extracted feature graphs are classified using the support vector machine classification algorithm with improved kernel function;

[0016] According to the classification result, the driver's facial image is determined from the collected image information;

[0017] The expression of the improved kernel function K(m,n) is as follows:

[0018]

[0019] (m,n) represents the coordinates of the feature points extracted by the convolutional neural network, exp represents the exponential function, and δ represents the standard deviation;

[0020] The support vector machine classification algorithm using the improved kernel function to classify the extracted feature graphs also includes calculating extreme values ​​using the following function constraint optimization problem to classify the extracted feature graphs:

[0021]

[0022]

[0023]

[0024] Where L(a,b,c) is the Lagrangian function, (x i ,y i ) is the coordinate of the point on the classification hyperplane, a is the normal vector of the classification hyperplane, b is the displacement term, c is the Lagrange multiplier, and T is the transposition symbol.

[0025] Optionally, the eye recognition model is a convolutional neural network model obtained by pre-labeling and training a preset first number of eye area and non-eye area sample images; the mouth recognition model is a convolutional neural network model obtained by pre-labeling and training a preset second number of mouth area and non-mouth area sample images; the step of recognizing the eye area image in the facial image according to the pre-trained eye recognition model and recognizing the mouth area image in the facial image according to the pre-trained mouth recognition model includes:

[0026] Cropping the facial image to normalize the facial image to a preset first size;

[0027] The eye image features in the cropped facial image are extracted using a pre-trained eye recognition model, and an activation function is used for classification and recognition to obtain an eye region image in the facial image;

[0028] The mouth image features in the cropped facial image are extracted using a pre-trained mouth recognition model, and the activation function is used for classification and recognition to obtain the mouth area image in the facial image.

[0029] Optionally, the step of respectively identifying the information related to the postures of the driver's eyes and mouth through an image sequence based on the eye region image and the mouth region image includes:

[0030] The eye region image is cropped and normalized to a preset second size; an eye state recognition model is used to recognize the eye blinking state in the cropped eye region image, and the number of blinks in the eye region image sequence within a preset first time period is obtained by counting; wherein the eye state recognition model is a convolutional neural network model obtained by pre-labeling and training a preset third number of eye blinking state sample images and eye normal state sample images;

[0031] The mouth area image is cropped and normalized to a preset third size; the yawning state in the cropped mouth area image is identified using a mouth state recognition model, and the number of yawning events in the mouth area image sequence within a preset second time period is obtained by counting; wherein the mouth state recognition model is a convolutional neural network model obtained by pre-labeling and training a preset fourth number of mouth yawning state sample images and mouth normal state sample images.

[0032] Optionally, the detecting whether the driver is in a fatigue state based on information related to the postures of the driver's eyes and mouth and based on preset rules includes:

[0033] Obtaining the number of blinks and yawns of the driver within a preset third time period;

[0034] Calculating the blinking frequency and yawning frequency of the driver in the preset third time period according to the number of blinks and yawns of the driver in the preset third time period;

[0035] Detecting whether the blinking frequency exceeds a preset first threshold, and whether the yawning frequency exceeds a preset second threshold;

[0036] If so, it is determined that the driver is in a fatigue state.

[0037] Optionally, after determining that the driver is in a fatigue state, the method further includes:

[0038] Play a preset voice warning message, and / or access a preset assisted driving system to assist in controlling the vehicle.

[0039] In a second aspect, the present application provides a driver fatigue detection system, the system comprising:

[0040] An acquisition module, used for acquiring a facial image of the driver;

[0041] A first recognition module, configured to recognize an eye region image in the facial image according to a pre-trained eye recognition model, and to recognize a mouth region image in the facial image according to a pre-trained mouth recognition model;

[0042] A second recognition module is used to recognize the posture-related information of the driver's eyes and mouth respectively based on the eye area image and the mouth area image through an image sequence;

[0043] The detection module is used to detect whether the driver is in a fatigue state based on information related to the driver's eyes and mouth postures and based on preset rules.

[0044] In a third aspect, the present application provides a computer device, which adopts the following technical solution:

[0045] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the driver fatigue detection method when executing the computer program.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0047] A computer-readable storage medium stores a computer program, which implements the steps of the driver fatigue detection method when executed by a processor.

[0048] The beneficial effect of the above embodiment is that by acquiring the eye area image and the mouth area image in the driver's facial image and identifying the information related to the posture of the driver's eyes and mouth, it is detected whether the driver is in a fatigue state according to the information related to the posture of the driver's eyes and mouth. Since fatigue detection is performed based on the driver's facial feature information such as the information related to the posture of the driver's eyes and mouth, it is possible to comprehensively judge whether different drivers are fatigued according to the posture of their eyes and mouth when driving a vehicle, and the recognition accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1A method flow chart of an embodiment of a driver fatigue detection method provided by the present application;

[0050] Figure 2 A schematic diagram of obtaining a driver's facial image in an embodiment of a driver fatigue detection method provided in this application;

[0051] Figure 3 A schematic diagram of identifying an eye region image and a mouth region image in an embodiment of a driver fatigue detection method provided in the present application;

[0052] Figure 4 A functional block diagram of an embodiment of a driver fatigue detection system provided in this application;

[0053] Figure 5 This is a functional block diagram of an embodiment of a computer device provided in this application. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of the present application and are used together with the embodiments of the present application to illustrate the principles of the present application, and are not used to limit the scope of the present application.

[0055] In the description of the present application, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0056] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] The present application provides a driver fatigue detection method, system, device and storage medium, which are described in detail below.

[0058] Reference Figure 1 A method flow chart of an embodiment of a driver fatigue detection method provided in the present application, the driver fatigue detection method comprises the following steps:

[0059] Step S101, obtaining a facial image of the driver;

[0060] Step S102, identifying an eye region image in the facial image according to a pre-trained eye recognition model, and identifying a mouth region image in the facial image according to a pre-trained mouth recognition model;

[0061] Step S103, identifying the posture-related information of the driver's eyes and mouth respectively through the image sequence based on the eye region image and the mouth region image;

[0062] Step S104, detecting whether the driver is in a fatigue state based on information related to the posture of the driver's eyes and mouth and based on preset rules.

[0063] This embodiment obtains the eye area image and the mouth area image in the driver's face image, and identifies the information related to the posture of the driver's eyes and mouth, and detects whether the driver is in a fatigue state according to the information related to the posture of the driver's eyes and mouth. Since fatigue detection is performed based on the driver's facial feature information such as the information related to the posture of the driver's eyes and mouth, it can comprehensively judge whether different drivers are fatigued according to the posture of their eyes and mouth when driving a vehicle, and the recognition accuracy is higher.

[0064] Reference Figure 2 , Figure 2 A schematic diagram of obtaining a driver's facial image in an embodiment of a driver fatigue detection method provided by the present application. In one embodiment, step S101 includes:

[0065] Obtaining image information of the driver collected in real time by a camera installed in the cab;

[0066] The collected image information is recognized using a face recognition algorithm based on deep learning to obtain the driver's facial image.

[0067] Specifically, the image information of the driver can be collected in real time by a camera installed in front of or on the side of the cab. For example, a camera with a viewing angle of 30 degrees and a focal length of 35 mm can be used to collect the driver's facial image. The collected image information is recognized by a face recognition algorithm based on deep learning. The face recognition algorithm is a convolutional neural network algorithm. The convolutional neural network algorithm can be used to extract features from the collected image information, and different convolutional layer features in the convolutional neural network are connected in stages to obtain an extracted feature map; the extracted feature map is classified by a support vector machine classification algorithm with an improved kernel function; and the driver's facial image is determined from the collected image information according to the classification result.

[0068] Specifically, a convolutional neural network with a 15-layer network, a 5×5 convolution template, and a 3×3 maximum pooling template can be used to extract the image information collected by the camera, and the features of different convolution layers can be connected in stages to further enhance the robustness of the extracted features. Then, the support vector machine classification algorithm with an improved kernel function is used to classify the extracted feature maps. The expression of the improved kernel function K(m,n) is as follows:

[0069]

[0070] (m,n) represents the coordinates of the feature points extracted by the convolutional neural network, exp represents the exponential function, and δ represents the standard deviation;

[0071] The support vector machine classification algorithm using the improved kernel function to classify the extracted feature graphs also includes calculating extreme values ​​using the following function constraint optimization problem to classify the extracted feature graphs:

[0072]

[0073]

[0074]

[0075] Where L(a,b,c) is the Lagrangian function, (x i ,y i ) is the coordinate of the point on the classification hyperplane, a is the normal vector of the classification hyperplane, b is the displacement term, c is the Lagrange multiplier, and T is the transposition symbol.

[0076] According to the classification result, the driver's face area, that is, the face image, is selected from the collected image information.

[0077] Further, refer to Figure 3 , Figure 3 A schematic diagram of identifying eye area images and mouth area images in an embodiment of the driver fatigue detection method provided by the present application. In one embodiment, the eye recognition model is a convolutional neural network model obtained by pre-labeling and training a preset number of eye area and non-eye area sample images; the mouth recognition model is a convolutional neural network model obtained by pre-labeling and training a preset number of mouth area and non-mouth area sample images; the eye area image in the facial image is recognized according to the pre-trained eye recognition model, and the mouth area image in the facial image is recognized according to the pre-trained mouth recognition model, including:

[0078] Cropping the facial image to normalize the facial image to a preset first size;

[0079] The eye image features in the cropped facial image are extracted using a pre-trained eye recognition model, and an activation function is used for classification and recognition to obtain an eye region image in the facial image;

[0080] The mouth image features in the cropped facial image are extracted using a pre-trained mouth recognition model, and the activation function is used for classification and recognition to obtain the mouth area image in the facial image.

[0081] For example, taking the eye recognition model as an example, the training process of the eye recognition model is as follows:

[0082] A. Set a preset number of sample images corresponding to the eye area and non-eye area (such as nose, mouth, ears and other facial features) (for example, 100 images for each area), and label each sample image with a corresponding label (for example, the sample image of the eye area is labeled as "eye", the sample image of the nose area is labeled as "nose", etc.);

[0083] B. Preprocess each sample image to obtain training images for model training;

[0084] C. Divide all training images into a training set of the first scale and a validation set of the second scale;

[0085] D. training the predetermined eye recognition model using the training set;

[0086] E. Use the verification set to verify the accuracy of the trained eye recognition model. If the accuracy is greater than or equal to the preset accuracy, the training ends. Alternatively, if the accuracy is less than the preset accuracy, increase the number of sample images corresponding to each eye area and non-eye area and re-execute the above steps B, C, D, and E.

[0087] The training methods of other convolutional neural network models in this embodiment can refer to the above-mentioned eye recognition model. After the eye recognition model is trained, the eye recognition model can be used to recognize the eye area image in the facial image. Specifically, the facial image is first cropped, and the cropped image size is normalized to a 256×256 pixel image; according to the previously trained eye recognition model, image features are extracted through a 5-layer convolutional network, a 7×7 convolutional template, and a 2×2 maximum pooling template network, and the activation function softmax is used to recognize the eye area image.

[0088] Similarly, the facial image is first cropped and the cropped image size is normalized to a 256×256 pixel image. Based on the previously trained mouth recognition model, image features are extracted through a 5-layer convolutional network, a 7×7 convolutional template, and a 2×2 maximum pooling template network, and the activation function softmax is used to identify the mouth area image.

[0089] Furthermore, the step of identifying the posture-related information of the driver's eyes and mouth respectively based on the eye region image and the mouth region image through an image sequence includes:

[0090] The identified eye region image is cropped and normalized to a preset second size, for example, the cropped eye region image size can be normalized to a 64×64 pixel image. The eye state recognition model is used to recognize the eye blinking state in the cropped eye region image, and the number of blinks in the eye region image sequence within the preset first time period is statistically obtained, wherein the eye region image sequence is a series of images of the target, i.e., the eye region, acquired sequentially and continuously at different times and orientations. The eye state recognition model is a convolutional neural network model obtained by pre-labeling and training a preset third number of eye blinking state sample images and eye normal state sample images. For example, the eye blinking state image can be pre-set as an image in which the eyes are completely closed or the angle of the eyes being opened is less than a certain threshold, and the eye normal state image can be pre-set as an image in which the eyes are normally open. To avoid confusion between the eye blinking state and the eye normal state, the set threshold can be adjusted as low as possible. Prepare a preset number (e.g., 1,000 each) of sample images of eye blinking states and sample images of eye normal states, and annotate corresponding labels for each sample image (e.g., sample images of eye blinking states are annotated as "blinking", and sample images of eye normal states are annotated as "normal"); divide all sample images into a training set and a validation set, wherein both the training set and the validation set include sample images of eye blinking states and sample images of eye normal states. Use a convolutional neural network with deep learning to train the sample images in the training set to obtain a preliminary trained model; then use the preliminary trained model to repeatedly verify and train the sample images in the validation set until the model converges, and use the converged model as the final trained eye state recognition model. For example, in an optional implementation, the eye state recognition network model with a 3-layer convolutional network, a 5×5 convolutional template, and a 2×2 maximum pooling template can be used to extract the image features of the eye blinking state in the cropped eye area image, and the activation function softmax is used to identify the state of the eye.

[0091] The identified mouth area image is cropped and normalized to a preset third size. For example, the size of the cropped mouth area image can be normalized to a 64×64 pixel image. The mouth state recognition model is used to recognize the yawning state of the mouth in the cropped mouth area image, and the number of yawns in the mouth area image sequence within the preset second time period is statistically obtained, wherein the mouth area image sequence is a series of images of the target, i.e., the mouth area, obtained in sequence and continuously at different times and orientations. The mouth state recognition model is a convolutional neural network model obtained by pre-labeling and training a preset fourth number of mouth yawning state sample images and mouth normal state sample images. For example, the mouth yawning state image can be pre-set as an image with a mouth opening angle exceeding a certain threshold, and the mouth normal state image can be pre-set as an image with a mouth opening normally. In order to avoid confusion between the mouth yawning state and the mouth normal state, the set opening angle threshold can be adjusted as high as possible. Prepare a preset number (e.g., 1,000 each) of sample images of the mouth in a yawning state and sample images of the mouth in a normal state, and annotate each sample image with a corresponding label (e.g., sample images of the mouth in a yawning state are labeled as "yawning", and sample images of the mouth in a normal state are labeled as "normal"); divide all sample images into a training set and a validation set, wherein both the training set and the validation set include sample images of the mouth in a yawning state and sample images of the mouth in a normal state. Use a convolutional neural network with deep learning to train the sample images in the training set to obtain a preliminarily trained model; then use the preliminarily trained model to repeatedly verify and train the sample images in the validation set until the model converges, and use the converged model as the final trained mouth state recognition model. For example, in an optional implementation, a mouth state recognition network model with a 3-layer convolutional network, a 5×5 convolutional template, and a 2×2 maximum pooling template can be used to extract the image features of the mouth in the cropped mouth area image, and the excitation function softmax is used to identify the state of the mouth.

[0092] Further, in one embodiment, the detecting whether the driver is in a fatigue state based on the information related to the posture of the driver's eyes and mouth and based on preset rules includes:

[0093] Obtaining the number of blinks and yawns of the driver within a preset third time period;

[0094] Calculating the blinking frequency and yawning frequency of the driver in the preset third time period according to the number of blinks and yawns of the driver in the preset third time period;

[0095] Detecting whether the blinking frequency exceeds a preset first threshold, and whether the yawning frequency exceeds a preset second threshold;

[0096] If so, it is determined that the driver is in a fatigue state.

[0097] In this embodiment, the driver's fatigue is mainly judged by combining the frequency of the driver's blinking and yawning within a period of time. For example, when the driver's blinking frequency exceeds the preset first threshold 30 times within a time period of 60 minutes, and the frequency of yawning exceeds the preset second threshold 3 times, it can be comprehensively judged that the driver is in a fatigue driving state. Among them, the time period, the preset first threshold, and the preset second threshold can be adjusted and set according to the actual application scenario to achieve an accurate detection effect. Of course, it can also be personalized according to the behavioral habits of different drivers, which is not limited here.

[0098] In addition, in another optional implementation, after comprehensively judging the driver's fatigue through blinking frequency and yawning frequency, a voice message can be automatically sent to the driver for confirmation, and the driver's feedback on the voice message can be used to further confirm whether the driver is in a fatigued state, thereby more accurately detecting the driver's fatigue and preventing the system from misjudging.

[0099] Furthermore, in one embodiment, after determining that the driver is in a fatigue state, the method further includes:

[0100] Play preset voice warning information, such as playing a loud voice to remind the driver that he is in a fatigued driving state and needs to pay attention to driving safety. Or, automatically connect to the preset auxiliary driving system to assist in controlling the vehicle, such as automatically turning on the anti-collision warning and lane keeping functions, or slowly reduce the speed to ensure the driver's safety.

[0101] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0102] This embodiment also provides a driver fatigue detection system, which corresponds one-to-one with the driver fatigue detection method in the above embodiment. Figure 4 As shown, the driver fatigue detection system includes an acquisition module 401, a first recognition module 402, a second recognition module 403, and a detection module 404. Each functional module is described in detail as follows:

[0103] An acquisition module 401 is used to acquire a facial image of a driver;

[0104] A first recognition module 402, configured to recognize an eye region image in the facial image according to a pre-trained eye recognition model, and recognize a mouth region image in the facial image according to a pre-trained mouth recognition model;

[0105] The second recognition module 403 is used to recognize the posture-related information of the driver's eyes and mouth respectively based on the eye region image and the mouth region image through the image sequence;

[0106] The detection module 404 is used to detect whether the driver is in a fatigue state based on information related to the posture of the driver's eyes and mouth and based on preset rules.

[0107] For the specific definition of each module of the driver fatigue detection system, please refer to the definition of the driver fatigue detection method above, which will not be repeated here. Each module in the above-mentioned driver fatigue detection system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0108] Reference Figure 5 This embodiment also provides a computer device, which may be a computing device such as a mobile terminal, a desktop computer, a notebook, a palm computer, a server, etc. The computer device includes a processor 10, a memory 20, and a display 30. Figure 5 Only some components of the computer device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0109] In some embodiments, the memory 20 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 20 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Furthermore, the memory 20 may also include both an internal storage unit of the computer device and an external storage device. The memory 20 is used to store application software and various types of data installed in the computer device, such as program codes installed in the computer device. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a driver fatigue detection computer program 40 is stored on the memory 20.

[0110] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing a driver fatigue detection method.

[0111] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the computer device and to display a visual user interface. The components 10-30 of the computer device communicate with each other through a system bus.

[0112] In one embodiment, when the processor 10 executes the driver fatigue detection computer program 40 in the memory 20, the following steps are implemented:

[0113] Acquire a facial image of the driver;

[0114] Identify an eye region image in the facial image according to a pre-trained eye recognition model, and identify a mouth region image in the facial image according to a pre-trained mouth recognition model;

[0115] Based on the eye region image and the mouth region image, respectively identifying the posture-related information of the driver's eyes and mouth through the image sequence;

[0116] Based on the information related to the driver's eye and mouth postures and based on preset rules, it detects whether the driver is in a state of fatigue.

[0117] This embodiment also provides a computer-readable storage medium on which a driver fatigue detection computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0118] Acquire a facial image of the driver;

[0119] Identify an eye region image in the facial image according to a pre-trained eye recognition model, and identify a mouth region image in the facial image according to a pre-trained mouth recognition model;

[0120] Based on the eye region image and the mouth region image, respectively identifying the posture-related information of the driver's eyes and mouth through the image sequence;

[0121] Based on the information related to the driver's eye and mouth postures and based on preset rules, it detects whether the driver is in a state of fatigue.

[0122] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0123] Any reference to memory, storage, database or other medium used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0124] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

Claims

1. A driver fatigue detection method, characterized in that: The method comprises: Acquire a facial image of the driver; Identify an eye region image in the facial image according to a pre-trained eye recognition model, and identify a mouth region image in the facial image according to a pre-trained mouth recognition model; Based on the eye region image and the mouth region image, respectively identifying the posture-related information of the driver's eyes and mouth through the image sequence; Detects whether the driver is in a fatigue state based on information related to the driver's eyes and mouth posture and preset rules; The step of obtaining the driver's facial image comprises: Obtaining image information of the driver collected in real time by a camera installed in the cab; The collected image information is recognized using a face recognition algorithm based on deep learning to obtain the driver's facial image; The face recognition algorithm based on deep learning is a convolutional neural network algorithm. The face recognition algorithm based on deep learning is used to recognize the collected image information to obtain the driver's facial image, which includes: The convolutional neural network algorithm is used to extract features from the collected image information, and the different convolutional layer features in the convolutional neural network are connected in stages to obtain the extracted feature map; The extracted feature graphs are classified using the support vector machine classification algorithm with improved kernel function; According to the classification result, the driver's facial image is determined from the collected image information; The expression of the improved kernel function K(m,n) is as follows: (m,n) represents the coordinates of the feature points extracted by the convolutional neural network, exp represents the exponential function, δ represents the standard deviation; The support vector machine classification algorithm using the improved kernel function to classify the extracted feature graphs also includes calculating extreme values ​​using the following function constraint optimization problem to classify the extracted feature graphs: Where L(a,b,c) is the Lagrangian function, ( x i , y i ) are the coordinates of the points on the classification hyperplane, a is the normal vector of the classification hyperplane, b is the displacement term, c is the Lagrange multiplier, and T is the transposition symbol.

2. The driver fatigue detection method according to claim 1, characterized in that: The eye recognition model is a convolutional neural network model obtained by pre-marking and training a preset first number of eye area and non-eye area sample images; the mouth recognition model is a convolutional neural network model obtained by pre-marking and training a preset second number of mouth area and non-mouth area sample images; the eye area image in the facial image is recognized according to the pre-trained eye recognition model, and the mouth area image in the facial image is recognized according to the pre-trained mouth recognition model, including: Cropping the facial image to normalize the facial image to a preset first size; The eye image features in the cropped facial image are extracted using a pre-trained eye recognition model, and an activation function is used for classification and recognition to obtain an eye region image in the facial image; The mouth image features in the cropped facial image are extracted using a pre-trained mouth recognition model, and the activation function is used for classification and recognition to obtain the mouth area image in the facial image.

3. The driver fatigue detection method according to claim 1 or 2, characterized in that: The step of respectively identifying the posture-related information of the driver's eyes and mouth through an image sequence based on the eye region image and the mouth region image includes: The eye region image is cropped and normalized to a preset second size; an eye state recognition model is used to recognize the eye blinking state in the cropped eye region image, and the number of blinks in the eye region image sequence within a preset first time period is obtained by counting; wherein the eye state recognition model is a convolutional neural network model obtained by pre-labeling and training a preset third number of eye blinking state sample images and eye normal state sample images; The mouth area image is cropped and normalized to a preset third size; the yawning state in the cropped mouth area image is identified using a mouth state recognition model, and the number of yawning events in the mouth area image sequence within a preset second time period is obtained by counting; wherein the mouth state recognition model is a convolutional neural network model obtained by pre-labeling and training a preset fourth number of mouth yawning state sample images and mouth normal state sample images.

4. The driver fatigue detection method according to claim 3, characterized in that: The method of detecting whether the driver is in a fatigue state based on the information related to the posture of the driver's eyes and mouth and based on preset rules includes: Obtaining the number of blinks and yawns of the driver within a preset third time period; Calculating the blinking frequency and yawning frequency of the driver in the preset third time period according to the number of blinks and yawns of the driver in the preset third time period; Detecting whether the blinking frequency exceeds a preset first threshold, and whether the yawning frequency exceeds a preset second threshold; If so, it is determined that the driver is in a fatigue state.

5. The driver fatigue detection method according to claim 4, characterized in that: After determining that the driver is in a fatigue state, the method further includes: Play a preset voice warning message, and / or access a preset assisted driving system to assist in controlling the vehicle.

6. A driver fatigue detection system, characterized in that: The system comprises: An acquisition module, used for acquiring a facial image of the driver; A first recognition module, configured to recognize an eye region image in the facial image according to a pre-trained eye recognition model, and to recognize a mouth region image in the facial image according to a pre-trained mouth recognition model; A second recognition module is used to recognize the posture-related information of the driver's eyes and mouth respectively based on the eye area image and the mouth area image through an image sequence; A detection module, used to detect whether the driver is in a fatigue state based on information related to the driver's eyes and mouth postures and preset rules; The step of obtaining the driver's facial image comprises: Obtaining image information of the driver collected in real time by a camera installed in the cab; The collected image information is recognized using a face recognition algorithm based on deep learning to obtain the driver's facial image; The face recognition algorithm based on deep learning is a convolutional neural network algorithm. The face recognition algorithm based on deep learning is used to recognize the collected image information to obtain the driver's facial image, which includes: The convolutional neural network algorithm is used to extract features from the collected image information, and the different convolutional layer features in the convolutional neural network are connected in stages to obtain the extracted feature map; The extracted feature graphs are classified using the support vector machine classification algorithm with improved kernel function; According to the classification result, the driver's facial image is determined from the collected image information; The expression of the improved kernel function K(m,n) is as follows: (m,n) represents the coordinates of the feature points extracted by the convolutional neural network, exp represents the exponential function, δ represents the standard deviation; The support vector machine classification algorithm using the improved kernel function to classify the extracted feature graphs also includes calculating extreme values ​​using the following function constraint optimization problem to classify the extracted feature graphs: Where L(a,b,c) is the Lagrangian function, ( x i , y i ) are the coordinates of the points on the classification hyperplane, a is the normal vector of the classification hyperplane, b is the displacement term, c is the Lagrange multiplier, and T is the transposition symbol.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the driver fatigue detection method as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the driver fatigue detection method according to any one of claims 1 to 5 are implemented.

Images