Karwning behavior detection method and system, electronic equipment and vehicle

Abstract

The invention relates to a yawning behavior detection method and system, electronic equipment and a vehicle. The method comprises the following steps: acquiring an original video containing the face of a driver; generating a corresponding face cutting image according to a video frame in the original video; according to a pre-trained detection model based on a convolutional neural network, classifying the face clipping image to obtain a classification result of whether yawning occurs; wherein the detection model is obtained by training sample images simulating various head postures and various yawn angles. According to the scheme provided by the invention, the yawning behavior of the driver can be efficiently and accurately detected through the convolutional neural network trained by adopting the sample images with rich facial features, and the situation of false detection and missing detection is avoided.

Description

Technical Field

[0001] The present application relates to the field of computer vision processing technology, and in particular to a yawning behavior detection method, system, electronic device and vehicle. Background Art

[0002] Driver fatigue is an important factor in causing traffic accidents. Many traffic accidents are caused by driver fatigue and lack of concentration. With the advancement of artificial intelligence technology in recent years, vision-based fatigue driving detection technology has received more and more attention. Yawning is a key feature of fatigue, and the accuracy of yawn recognition will have a great impact on fatigue detection.

[0003] Traditional yawn detection methods generally monitor the mouth aspect ratio (MAR) or only focus on the mouth area to determine whether the driver is yawning, which has limitations. MAR-based detection methods rely heavily on the accuracy of facial landmark detection, which is often unreliable when the facial landmarks are partially occluded or in a non-frontal face direction. Similarly, methods that only detect the mouth fail to take into account the related facial movements that accompany yawning, and are prone to misidentifying other mouth opening movements (such as singing, laughing, etc.) as yawns.

[0004] Therefore, how to obtain an efficient and accurate yawn detection method is a technical problem that needs to be solved at present. Summary of the invention

[0005] In order to solve or partially solve the problems existing in the related art, the present application provides a yawning behavior detection method, system, electronic device and vehicle, which can efficiently and accurately detect the driver's yawning behavior and avoid false detection and missed detection.

[0006] The first aspect of the present application provides a yawning behavior detection method, which comprises:

[0007] Get the original video containing the driver's face;

[0008] Generate a corresponding facial cropped image according to the video frames in the original video;

[0009] Classifying the facial cropped image according to a pre-trained convolutional neural network-based detection model to obtain a classification result of whether the face is yawning;

[0010] The detection model is obtained by training sample images simulating various head postures and various yawning angles.

[0011] In some embodiments, the method further comprises:

[0012] In the detection unit of the preset time length, when the number of frames of the facial cropped image classified as yawning is greater than a preset threshold, an alarm message is generated.

[0013] In some embodiments, the convolutional neural network includes a residual network and a maximum pooling layer, and the multiple residual blocks of the residual network each include a convolutional layer, a batch normalization layer, and a ReLU activation layer.

[0014] In some embodiments, weights are assigned to the convolutional layer output channels corresponding to facial features associated with yawning to influence the classification results of the detection model in combination with facial details; and / or

[0015] During the pre-training process of the detection model, the convolutional neural network also includes a Dropout layer, and the Dropout layer is used to discard some weight parameters in the pre-training process.

[0016] In some implementations, generating a corresponding facial cropped image based on a video frame in the original video includes:

[0017] Performing facial recognition and cropping on the video frame to obtain a facial cropped image of a preset size; and / or

[0018] The video frame or the facial cropped image is flipped to obtain a facial cropped image that is consistent with the shooting perspective of the sample image.

[0019] In some implementations, at least part of the sample images used in the training of the detection model are obtained according to the following method:

[0020] The original image displayed in the front face is rotated and transformed into sample images displayed in the side face, so that each of the sample images includes a variety of head postures and a variety of yawning angles.

[0021] In some implementations, rotating the original image displayed in the front face to transform it into a sample image displayed in the side face includes:

[0022] Obtaining initial two-dimensional coordinates of key facial points in the original image displayed frontally, wherein the key facial points include eyes, nose, and mouth, and the initial two-dimensional coordinates include X-axis coordinates and Y-axis coordinates;

[0023] Constructing a three-dimensional frontal face model according to the initial two-dimensional coordinates and the assigned Z-axis coordinates;

[0024] Rotate the three-dimensional frontal face model along the Y axis to obtain a corresponding three-dimensional side face model;

[0025] A sample image of the side face display is obtained by projecting the facial key points in the three-dimensional side face model onto a two-dimensional plane.

[0026] A second aspect of the present application provides a yawning behavior detection system, which comprises:

[0027] A video acquisition module, used to acquire the original video containing the driver's face;

[0028] An image processing module, used for generating a corresponding facial cropped image according to a video frame in the original video;

[0029] A classification prediction module is used to classify the facial cropped image according to a pre-trained detection model based on a convolutional neural network to obtain a classification result of whether the face is yawning; wherein the detection model is trained based on sample images simulating multiple head postures and multiple yawning angles.

[0030] A third aspect of the present application provides an electronic device, including:

[0031] Processor; and

[0032] A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the yawning behavior detection method as described in the first aspect above.

[0033] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the yawning behavior detection method as described in the first aspect above.

[0034] The fifth aspect of the present application provides a vehicle, comprising the electronic device as described in the third aspect above and / or the computer-readable storage medium provided in the fourth aspect.

[0035] The technical solution provided by this application may have the following beneficial effects:

[0036] The yawning behavior detection method of the present application uses a convolutional neural network trained with sample images rich in facial features to classify facial cropped images as "yawning" or "not yawning" based on comprehensive facial features. It can achieve efficient and accurate classification in more complex scenarios, and then accurately determine the driver's fatigue level, give risk warnings in a timely manner, and reduce the risk of traffic accidents.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0039] Figure 1 It is a flowchart of a yawning behavior detection method provided in an embodiment of the present application;

[0040] Figure 2 It is a flowchart of the training data enhancement method provided in the embodiment of the present application;

[0041] Figure 3 It is a flowchart of a yawning behavior detection method provided in an embodiment of the present application;

[0042] Figure 4 is a structural schematic diagram of a yawning behavior detection system provided in an embodiment of the present application;

[0043] Figure 5 is another structural schematic diagram of the yawning behavior detection system provided in an embodiment of the present application;

[0044] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0046] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0047] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0048] In the related art, in driver fatigue monitoring, the traditional yawn detection method generally determines whether the driver yawns by monitoring the aspect ratio of the mouth or only focusing on the mouth area, which may result in inaccurate judgment results.

[0049] In response to the above problems, the present application provides a yawning behavior detection method, which can efficiently and accurately detect the driver's yawning behavior and avoid false detection or missed detection.

[0050] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0051] See also Figure 1 , a yawning behavior detection method shown in the present application includes:

[0052] S110, obtaining an original video including the driver's face.

[0053] In the present application, the driver's head area can be photographed according to the camera installed in the vehicle to obtain the original video including the driver's face. In some embodiments, the camera can be an ordinary camera or an infrared (IR) camera. Among them, the corresponding camera positions are arranged according to the distribution position of the cockpit in the car. For example, when the cockpit is a left-hand drive, the camera can be installed on the inside of the left A-pillar of the vehicle and the shooting angle is aimed at the driver's head area. Similarly, when the cockpit is a right-hand drive, the camera can be installed on the inside of the right A-pillar of the vehicle. In other embodiments, the camera can also be installed in front of the cockpit to obtain the original video of the driver's face.

[0054] Furthermore, the driver's head is photographed at a preset frame rate to obtain the corresponding original video. By controlling the camera frame rate, the data processing volume of the system is controlled, the processing efficiency is improved, and the collection of key information is ensured.

[0055] In order to capture the driver's facial video with high quality, in some embodiments, an illuminator is also installed in the vehicle to ensure that a clear image can be obtained under different lighting conditions. For example, the illuminator can be a near infrared (NIR) illuminator or an infrared (IR) illuminator.

[0056] S120, generating a corresponding facial cropped image according to the video frames in the original video.

[0057] In some implementations, the original video is divided into frames according to a preset duration to obtain corresponding video frames. The preset duration may be 1 second to 5 seconds. For example, with the real-time acquisition of the original video, every 3 seconds of the original video may be used as a detection unit, and the video frames in the corresponding original video may be used to generate a facial cropped image. For example, the first second to the third second may be used as a detection unit, the second second to the fourth second may be used as a detection unit, the third second to the fifth second may be used as a detection unit, and so on.

[0058] In some embodiments, face recognition and cropping are performed on the video frame to obtain a face cropped image of a preset size. Specifically, the face area in the video frame can be detected according to the face detection model in the relevant technology, and then the video frame can be further cropped, and the video frame can be cropped to a preset size while saving the face image. Such a design can remove redundant image information and reduce the data processing load of the system. In some embodiments, the preset sizes of different face cropped images can be the same or similar. For example, all face cropped images have a uniform preset size, and the preset size can be a square size or a rectangular size. Specifically, according to the length and width of the image, the preset size can be, for example, (96-512) x (96-512) pixels. For example, the preset size can be 96x96 pixels, 128x128 pixels, 96x128 pixels, etc., which are only exemplified here. By controlling the size of the face cropped image, while ensuring the required accuracy of the detection, the amount of redundant data processing is reduced and the data processing efficiency is improved.

[0059] In order to further improve the system processing efficiency, in some embodiments, when the detection unit that is later in time sequence has a video frame at the same time as the detection unit that is earlier in time sequence, the facial cropped image of the same video frame can be used as the facial cropped image of the video frame of the current detection unit. Such a design can reduce the recognition processing of the same frame adaptation frame, reduce the data processing load of the system, and improve the processing efficiency. In other words, each video frame has a unique timing mark. For the same video frame, there is no need to repeat facial recognition and size cropping. The facial cropped image generated previously can be called as the facial cropped image required by the current detection unit, reducing the redundant operation of the system.

[0060] In some embodiments, the method of the present application further includes: flipping the video frame or the facial cropped image to obtain a facial cropped image that is consistent with the shooting angle of the sample image. Before obtaining the facial cropped image, the video frame can be flipped. After obtaining the facial cropped image, the facial cropped image can be flipped. It can be understood that in the training process of the detection model, most sample images generally select images with a unified shooting angle, such as images uniformly shot from the left side of the left steering wheel, or images uniformly shot from the right side of the right steering wheel. If the current vehicle is left-hand steering, and the sample image is shot from the right side of the right steering wheel, the video frame or facial cropped image can be flipped to the right shooting angle to adapt to the training effect of the detection model. Similarly, if the current vehicle is right-hand steering, and the sample image is shot from the left side of the left steering wheel, the video frame or facial cropped image can be flipped to the left shooting angle. It can be understood that when the shooting angle of the video frame is consistent with the shooting angle of the sample image, the video frame or facial cropped image does not need to be flipped.

[0061] S130, classifying the cropped facial image according to a pre-trained detection model based on a convolutional neural network to obtain a classification result of whether the face is yawning; wherein the detection model is trained based on sample images simulating multiple head postures and multiple yawning angles.

[0062] The detection model of the present application is a model using a convolutional neural network architecture. The detection model is pre-trained based on training data. The training data includes sample images and corresponding sample labels, and the sample labels are "yawning" or "not yawning", which are marked according to the yawning situation of the sample image.

[0063] In order to train a detection model that can make predictions based on full-face features, each frame of the sample image includes a variety of head postures and yawning angles. It can be understood that during driving, the driver's shooting angle with the camera is not fixed, that is, the head postures captured by the camera are diverse, such as the degree of lowering and raising the head, and the steering angle of the head are diverse; when yawning, the yawning angle includes the angles of various facial features and other facial details, such as the closure of the eyes shows dynamic changes, and the opening amplitude of the lips from the beginning to the end is not constant. In other words, yawning is a process in which the face undergoes dynamic changes, accompanied by changes in the details of multiple facial features throughout the process. Based on this, the sample images selected in this application include various yawning angles under various head postures, so that the detection model can comprehensively learn the facial details of the yawning image, rather than focusing only on whether the mouth is open or closed or the aspect ratio of the mouth.

[0064] Further, in some embodiments, the convolutional neural network of the present application includes a residual network (ResNet) and a maximum pooling layer, and the residual block of the residual network includes a convolution layer, a batch normalization layer, and a ReLU activation layer. Among them, the convolutional layer (Convolutional Layer) is used to perform a convolution operation, using multiple convolution kernels to extract local features in each frame of facial cropped images, such as edges and textures, to generate a feature map; each residual block may include at least two layers of convolutional layers. The batch normalization layer (Batch Normalization Layer) is used to normalize the feature map output by the convolution layer, reduce internal covariate shift, and improve training speed and stability. The ReLU activation layer (Rectified Linear Unit Layer) applies a nonlinear activation function to activate the normalized feature map and increase the nonlinear representation capability of the model. The maximum pooling layer (MaxPooling Layer) is set between the two residual blocks to downsample the activated feature map, reduce the size of the feature map, while retaining the main features and reducing the computational complexity. It should be noted that not every two residual blocks must include an independent maximum pooling layer, and the number of maximum pooling layers can be customized according to the dimensionality reduction requirements.

[0065] Furthermore, in order to strengthen the influence of facial details, in some embodiments, weights are assigned to the output channels of the convolutional layer corresponding to the facial features related to yawning, so as to combine the classification results of the detection model with the facial details. That is to say, by adding modules such as Squeeze-and-Excitation (SE) Block or Convolutional Block Attention Module (CBAM) to the residual block of the residual network, dynamic adjustment of channel and spatial attention is achieved. The module can automatically learn the importance of each channel and assign weights according to the features in the input image, thereby enhancing the network's attention to specific areas. For example, larger weights are assigned to specific areas such as eyes, mouth, eyebrows, nostrils, etc., thereby enhancing the network's attention to these specific areas. For example, through SE Block, the network can more effectively capture facial features related to yawning, such as the amplitude of the mouth opening, the degree of eye closure, the expansion of the nostrils and other details, thereby improving the accuracy of the classification results.

[0066] By inputting each frame of facial cropped image of a preset size into the detection model, the classification results of whether each frame of facial cropped image is "yawning" or "not yawning" are output respectively.

[0067] The yawning behavior detection method of the present application uses a convolutional neural network trained with sample images rich in facial features to classify facial cropped images as "yawning" or "not yawning" based on comprehensive facial features. It can achieve efficient and accurate classification in more complex scenarios, and then accurately determine the driver's fatigue level, give risk warnings in a timely manner, and reduce the risk of traffic accidents.

[0068] See also Figure 2 An embodiment of the present application also provides a training method for the detection model of the present application based on a convolutional neural network. The detection model of the present application can be obtained by training according to the following method.

[0069] S210, obtaining training data, where the training data includes multiple frames of sample images and corresponding sample labels.

[0070] In order to obtain sample images with a unified shooting angle, sample images of yawning in the left-hand steering wheel or sample images of yawning in the right-hand steering wheel under the same shooting angle can be collected. By flipping the image, sample images of different shooting angles can be unified so that the detection model can focus on learning the facial features of yawning under a specific shooting angle. Based on this, left-hand steering training data can be set to train a detection model for identifying yawns of left-hand steering drivers; right-hand steering training data can be set to train a detection model for identifying yawns of right-hand steering drivers.

[0071] Furthermore, in order to distinguish yawning from facial features when singing and talking, some sample images also include facial images when singing and talking and set the sample label as "not yawning". However, under the premise of a small number of samples, in order to enrich the head posture and yawning angle changes in the sample images, affine transformation can be performed based on the original yawning image to obtain sample images with multiple head postures and multiple yawning angles.

[0072] In order to enrich the sample images as much as possible, in some embodiments, the original image displayed in the front face is rotated and transformed into a sample image displayed in the side face, so that each sample image includes a variety of head postures and a variety of yawning angles. Among them, the original image displayed in the front face can be a plurality of frames of original images, and different original images contain a variety of yawning angles when the front face is displayed, such as A1, A2, ...An. By rotating the front face display image of each yawning angle at different angles, such as B1, B2, ...Bn, sample images of side face display with various head postures can be obtained, such as A1B1, A1B2, A1Bn, A2B1, A2B2, A2Bn, AnB1, An1B2, AnBn, etc. For example, the sample image A1B1 refers to the facial image at the yawning angle shown in A1, combined with the head rotation angle of B1, and so on. Such a design expands the data volume of the sample images and enhances the training data, thereby ensuring that the model learns to accurately classify yawns under various head postures and environmental conditions.

[0073] In order to obtain the above-mentioned diverse sample images, in some implementations, enhanced training data is obtained according to the following method:

[0074] S211, obtaining initial two-dimensional coordinates of key facial points in the original image displayed in the front face, the key facial points including eyes, nose, and mouth, and the initial two-dimensional coordinates including X-axis coordinates and Y-axis coordinates.

[0075] It is understandable that, since the camera captures a two-dimensional image, when acquiring an original image showing the face, the initial two-dimensional coordinates of each key point can be obtained according to relevant technologies such as a facial key point detection algorithm. Optionally, facial key points may also include other parts such as eyebrows and ears, which are only used as examples and are not intended to be limiting.

[0076] Among them, in the three-dimensional coordinate system of the image, a three-dimensional coordinate system consisting of an X-axis coordinate axis, a Y-axis coordinate axis and a Z-axis coordinate axis can be preset. The X-axis extends from left to right along the image, the Y-axis extends from bottom to top along the image, and the Z-axis extends from the camera to the direction of the photographed target. Accordingly, the initial two-dimensional coordinates of each facial key point are (x, y).

[0077] S212, constructing a three-dimensional frontal face model according to the initial two-dimensional coordinates and the assigned Z-axis coordinates.

[0078] This step performs 3D modeling based on the 2D image. In this step, some key points are preset with specific depth values, i.e., z values, so that the 3D coordinates (x, y, z) of each key point can be obtained; for example, the z value of the nose key point is 0, and the eyes and mouth use the z value coordinate of the nose as a reference and obtain z values ​​according to common sense. Based on the 3D coordinates of these key points, a simplified 3D frontal face image can be constructed.

[0079] S213, rotating the three-dimensional frontal face model along the Y axis to obtain a corresponding three-dimensional side face model.

[0080] In this step, the key points are rotated at different angles using a rotation matrix around the Y axis to obtain multiple 3D side face models of corresponding angles. Accordingly, the key points of each 3D side face model have a rotated 3D coordinate (x', y', z').

[0081] S214, projecting facial key points in the three-dimensional side face model onto a two-dimensional plane to obtain a sample image of the side face display.

[0082] It can be understood that the three-dimensional side face model similarly has facial key points corresponding to the three-dimensional front face model. In this step, the three-dimensional coordinates (x', y', z') of the facial key points of the three-dimensional side face model are projected onto the two-dimensional plane of the image to obtain new two-dimensional coordinates (x", y") corresponding to each key point, and a new two-dimensional image is obtained through affine transformation, i.e., the sample image displayed in side face. It should be noted that, according to the principle of what you see is what you get, for the facial key points of the three-dimensional side face model that are no longer displayed in front, there is no need to obtain the corresponding new two-dimensional coordinates, and thus these hidden facial key points will not be displayed in the sample image displayed in side face.

[0083] In summary, according to the above-mentioned image change method, a large number of sample images with various head postures and various yawning angles can be quickly obtained for the detection model to learn, and the association between different facial detail features and yawning can be learned from the enhanced training data to improve the recognition accuracy of the model; at the same time, the training data of specific shooting angles are distinguished to train the detection model separately, and the trained detection model can be deployed in vehicles in different countries / regions to meet the yawn detection needs under different driving habits.

[0084] S220, deploying a network architecture of a detection model according to the residual network, and obtaining a detection model by training according to the training data.

[0085] The architecture of the detection model can refer to the relevant introduction of the above step S130, which will not be repeated here. In addition, the convolution layer output channel corresponding to the facial features related to yawning has a pre-assigned larger weight to combine the facial details to influence the classification result of the detection model

[0086] During the pre-training process of the detection model, the convolutional neural network also includes a Dropout layer, which is used to discard some weight parameters during the pre-training process.

[0087] Specifically, in order to avoid overfitting and further improve the generalization ability of the model, especially when the sample size is small, the present application introduces Dropout regularization between the fully connected layers of the neural network. During the pre-training process of the model, specifically in each round of forward propagation, a portion of the network parameters are randomly zeroed (dropout) according to a random proportion (such as 5% to 20%) to avoid overfitting of the model parameters and make the model more generalizable. Furthermore, the Dropout layer of the present application is applied to the last fully connected layer to ensure that the classifier can maintain strong robustness when classifying yawns and effectively reduce overfitting on small data sets.

[0088] In summary, the pre-trained detection model of this application, by adding data types of various face angles during the training process, especially in the head tilt scene, can accurately identify yawning at different face angles, thereby significantly improving the detection accuracy under various conditions. By introducing the residual attention mechanism, it is possible to more effectively process complex facial expression features, especially in the recognition of small facial movements (such as eyes, eyebrows, and nostrils), which significantly improves the classification accuracy. By adding the Dropout layer, the generalization ability of the detection model has been effectively enhanced, and it can adapt to face images of different people and different lighting conditions, avoiding the problem of overfitting. The improvement of the overall network architecture only makes minor modifications to the original ResNet model, retaining the efficient characteristics of ResNet, while improving the classification performance, which is suitable for deployment on resource-limited devices, such as mobile terminals or embedded devices.

[0089] See also Figure 3 Based on the pre-trained detection model of the above embodiment, the yawning behavior detection method of the present application will be further introduced in combination with the embodiment below, which includes:

[0090] S310, obtaining an original video including the driver's face according to a preset frame rate from a camera installed on the A-pillar of the vehicle.

[0091] In this embodiment, taking a left-hand drive vehicle as an example, an IR camera installed on the left A-pillar of the vehicle is used to continuously capture the driver's face in real time with the assistance of a near-infrared illuminator, and acquires the original video containing the driver's face at a preset frame rate, such as 15 frames per second, according to advanced driver monitoring system (DMS) imaging technology. The preset frame rate is only used as an example and is not intended to be limiting.

[0092] S320, performing facial recognition and cropping on the video frames in the original video to obtain a facial cropped image of a preset size.

[0093] According to the face detection model in the related art, the face boundary box in the captured image is identified and cropped to obtain a face cropped image of a preset size, for example, a uniform size of 96x96 pixels, to improve the image quality.

[0094] Flip the face crop left / right if it was taken from a different perspective than the one the detection model was trained to accommodate.

[0095] S330, classifying the facial cropped image according to the pre-trained residual network-based detection model to obtain a classification result of whether the face is yawning; wherein, weights are assigned to the convolutional layer output channels corresponding to the facial features related to yawning to combine the facial details to affect the classification result of the detection model.

[0096] The detection model of the present application is pre-trained according to the above-mentioned training method. A greater weight is assigned to the convolutional layer output channel corresponding to the facial features related to yawning, and the facial features include but are not limited to the features of the five senses, and then the classification results of the detection model are affected by the facial details, so that the classification results are more accurate. It can be understood that if the image is detected to have yawning behavior, the classification result is "yawning", otherwise it is "not yawning".

[0097] S340, in the detection unit of the preset time length, when the number of frames of the facial cropped image classified as yawning is greater than a preset threshold, an alarm message is generated.

[0098] For example, the preset time length may be 3 seconds, that is, when each detection unit is 3 seconds, the classification result of whether yawning is present in each frame of the image is monitored. If, for example, 70% of the frames in the detection unit are classified as "yawning", an alarm message is generated. Of course, the preset threshold is only used as an example here and is not a limitation.

[0099] Optionally, the yawning alarm information can be triggered by auditory or visual signals, such as flashing the screen, lights in the car, playing alarm prompts, music, etc., to remind the driver to pay attention to fatigue driving.

[0100] In summary, the yawning behavior detection method of the present application is based on a detection model obtained by training with richer training data, and will not mistakenly identify other mouth opening movements (such as singing, laughing, etc.) as yawning, ensuring accurate detection results in head tilt scenarios. By using a comprehensive facial image as input, the movement characteristics of other parts of the face (such as eyes and eyebrows) are comprehensively considered when detecting yawns, thereby comprehensively analyzing the yawning characteristics and effectively avoiding false alarms. At the same time, the detection method of the present application also has good scalability and can be widely used in facial expression recognition, fatigue detection and other fields.

[0101] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a yawning behavior detection system and corresponding embodiments.

[0102] Figure 4 It is a structural schematic diagram of the yawning behavior detection system shown in this application.

[0103] See also Figure 4 The yawning behavior detection system shown in the present application includes a video acquisition module 410, an image processing module 420 and a classification prediction module 430. Among them:

[0104] The video acquisition module 410 is used to acquire the original video containing the driver's face.

[0105] The image processing module 420 is used to generate corresponding facial cropped images according to the video frames in the original video.

[0106] The classification prediction module 430 is used to classify the facial cropped image according to a pre-trained convolutional neural network-based detection model to obtain a classification result of whether the face is yawning; wherein the detection model is trained based on sample images simulating multiple head postures and multiple yawning angles.

[0107] In some specific implementations, the video acquisition module 410 is used to acquire the original video including the driver's face according to a preset frame rate based on a camera installed on the A-pillar of the vehicle.

[0108] See also Figure 5 In some specific implementations, the image processing module 420 includes a recognition and cropping module 421, which is used to perform facial recognition and cropping on the video frame to obtain a facial cropped image of a preset size.

[0109] In some specific implementations, the image processing module 420 includes an image flipping module 422, which is used to flip the video frame or the facial cropped image to obtain a facial cropped image that is consistent with the shooting perspective of the sample image.

[0110] In some specific implementations, the apparatus of the present application further includes a fatigue alarm module 440 for generating an alarm message when the number of frames of the facial cropped image classified as yawning is greater than a preset threshold within a detection unit of a preset duration.

[0111] From this example, it can be seen that the yawning behavior detection system of the present application can efficiently and accurately identify the driver's yawning behavior based on the pre-trained detection model, and promptly initiate an alarm to remind the driver to pay attention to rest, avoid fatigue driving, reduce the risk of traffic accidents, and improve driving safety.

[0112] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0113] An embodiment of the present application also provides a vehicle, on which the above-mentioned system or the following electronic device is deployed to accurately and efficiently detect the driver's fatigue driving behavior and promptly remind the driver of fatigue driving.

[0114] Figure 6 It is a schematic diagram of the structure of the electronic device shown in this application.

[0115] See also Figure 6 , the electronic device 1000 includes a memory 1010 and a processor 1020 .

[0116] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0117] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0118] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0119] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0120] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0121] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A yawning behavior detection method, characterized in that: include: Get the original video containing the driver's face; Generate a corresponding facial cropped image according to the video frames in the original video; Classifying the facial cropped image according to a pre-trained convolutional neural network-based detection model to obtain a classification result of whether the face is yawning; The detection model is obtained by training sample images simulating various head postures and various yawning angles.

2. The method according to claim 1, characterized in that: The method further comprises: In the detection unit of the preset time length, when the number of frames of the facial cropped image classified as yawning is greater than a preset threshold, an alarm message is generated.

3. The method according to claim 1, characterized in that The convolutional neural network includes a residual network and a maximum pooling layer, and the multiple residual blocks of the residual network each include a convolutional layer, a batch normalization layer, and a ReLU activation layer.

4. The method according to claim 3, characterized in that Assigning weights to the convolutional layer output channels corresponding to facial features associated with yawning to influence the classification results of the detection model in combination with facial details; and / or During the pre-training process of the detection model, the convolutional neural network also includes a Dropout layer, and the Dropout layer is used to discard some weight parameters in the pre-training process.

5. The method according to claim 1, characterized in that The step of generating a corresponding facial cropped image according to the video frames in the original video includes: Performing facial recognition and cropping on the video frame to obtain a facial cropped image of a preset size; and / or The video frame or the facial cropped image is flipped to obtain a facial cropped image that is consistent with the shooting perspective of the sample image.

6. The method according to claim 1, characterized in that At least part of the sample images used by the detection model during training is obtained according to the following method: The original image displayed in the front face is rotated and transformed into sample images displayed in the side face, so that each of the sample images includes a variety of head postures and a variety of yawning angles.

7. The method according to claim 6, characterized in that The method of rotating the original image displayed in the front face to transform it into a sample image displayed in the side face includes: Obtaining initial two-dimensional coordinates of key facial points in the original image displayed frontally, wherein the key facial points include eyes, nose, and mouth, and the initial two-dimensional coordinates include X-axis coordinates and Y-axis coordinates; Constructing a three-dimensional frontal face model according to the initial two-dimensional coordinates and the assigned Z-axis coordinates; Rotate the three-dimensional frontal face model along the Y axis to obtain a corresponding three-dimensional side face model; A sample image of the side face display is obtained by projecting the facial key points in the three-dimensional side face model onto a two-dimensional plane.

8. A yawning behavior detection system, characterized in that: include: A video acquisition module, used to acquire the original video containing the driver's face; An image processing module, used for generating a corresponding facial cropped image according to a video frame in the original video; A classification prediction module is used to classify the facial cropped image according to a pre-trained detection model based on a convolutional neural network to obtain a classification result of whether the face is yawning; wherein the detection model is trained based on sample images simulating multiple head postures and multiple yawning angles.

9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the yawning behavior detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the yawning behavior detection method according to any one of claims 1 to 7.

11. A vehicle, characterized in that: Includes the electronic device according to claim 9 and / or the computer-readable storage medium according to claim 10.

Images