Method for recognizing fatigue driving behavior, model training method, and server

Abstract

The method comprises the steps of: obtaining target low-resolution video data; preprocessing the target low-resolution video data to obtain a face video image sequence; processing the face video image sequence by using a preset fatigue driving behavior recognition model to obtain a fatigue driving behavior recognition result in the target low-resolution video data; and wherein the fatigue driving behavior recognition model is obtained by updating parameters of a preset model according to a label type and a prediction type of to-be-trained low-resolution video data. The present application can train a fatigue driving behavior recognition model according to a dynamic video, and utilize the time sequence relationship of face features between multiple frames of images in the dynamic video, so as to improve the recognition accuracy of the dynamic video scene to a certain extent.

Description

Technical Field

[0001] This disclosure relates to dynamic video recognition technology, and more particularly to a method for recognizing fatigued driving behavior, a model training method, and a server. Background Technology

[0002] With the rapid development of the national economy, the increasing popularity of logistics and transportation and private cars, road safety has become paramount. In particular, unsafe driving behaviors of large logistics truck drivers, such as fatigued driving, often lead to very serious personal and economic harm. Therefore, how to effectively monitor driver behavior and provide real-time warnings for fatigued driving is an urgent problem to be solved.

[0003] In existing technologies, the identification of fatigued driving behavior can be achieved by first acquiring a driver's facial image, then using traditional image algorithms or deep convolutional neural networks to extract the geometric features of key facial points in the facial image (such as the degree of eye opening, blinking frequency, and the degree of mouth opening), and finally using machine learning algorithms to identify fatigued driving behavior based on the extracted key point features.

[0004] However, the above methods are not suitable for recognizing low-resolution video data, and the recognition accuracy in dynamic video scenes needs to be further improved. Summary of the Invention

[0005] This disclosure provides a method for recognizing fatigued driving behavior, a model training method, and a server to improve the recognition accuracy of dynamic video scenes, and is applicable to low-resolution video scenes.

[0006] According to a first aspect of this disclosure, a method for identifying fatigued driving behavior is provided, comprising:

[0007] Acquire target low-resolution video data;

[0008] The target low-resolution video data is preprocessed to obtain a sequence of face video images;

[0009] The facial video image sequence is processed using a preset fatigue driving behavior recognition model to obtain fatigue driving behavior recognition results in the target low-resolution video data;

[0010] The fatigue driving behavior recognition model is obtained by updating the parameters of a preset model based on the annotation type and prediction type of the low-resolution video data to be trained; the prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model.

[0011] According to a second aspect of this disclosure, a training method for a fatigue driving behavior recognition model is provided, comprising:

[0012] Obtain a training dataset; the training dataset includes multiple low-resolution video data sets to be trained; the low-resolution video data sets to be trained have a labeled type;

[0013] The low-resolution video data to be trained is preprocessed to obtain a sequence of face video images;

[0014] The face video image sequence is processed using a preset model to obtain the prediction type corresponding to the low-resolution video data to be trained;

[0015] Based on the annotation type and prediction type of the low-resolution video data to be trained, the parameters of the preset model are updated to obtain a fatigue driving behavior recognition model.

[0016] The fatigue driving behavior recognition model is used to identify whether fatigue driving behavior exists in low-resolution video data.

[0017] According to a third aspect of this disclosure, a device for identifying fatigued driving behavior is provided, comprising:

[0018] The acquisition unit is used to acquire target low-resolution video data.

[0019] The preprocessing unit is used to preprocess the target low-resolution video data to obtain a face video image sequence;

[0020] The recognition unit is used to process the face video image sequence using a preset fatigue driving behavior recognition model to obtain the fatigue driving behavior recognition result in the target low-resolution video data.

[0021] The fatigue driving behavior recognition model is obtained by updating the parameters of a preset model based on the annotation type and prediction type of the low-resolution video data to be trained; the prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model.

[0022] According to a fourth aspect of this disclosure, a training device for a fatigue driving behavior recognition model is provided, comprising:

[0023] An acquisition unit is used to acquire a training dataset; the training dataset includes multiple low-resolution video data to be trained; the low-resolution video data to be trained has a labeling type;

[0024] The preprocessing unit is used to preprocess the low-resolution video data to be trained to obtain a sequence of face video images.

[0025] The training unit is used to process the face video image sequence using a preset model to obtain the prediction type corresponding to the low-resolution video data to be trained.

[0026] The training unit is also used to update the parameters of the preset model according to the annotation type and prediction type of the low-resolution video data to be trained, so as to obtain a fatigue driving behavior recognition model.

[0027] The fatigue driving behavior recognition model is used to identify whether fatigue driving behavior exists in low-resolution video data.

[0028] According to a fifth aspect of this disclosure, a server is provided, including a memory and a processor; wherein,

[0029] The memory is used to store computer programs;

[0030] The processor is configured to read a computer program stored in the memory and execute the method described in the first aspect and the second aspect according to the computer program in the memory.

[0031] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the methods described in the first and second aspects.

[0032] According to a seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in the first and second aspects.

[0033] The fatigue driving behavior recognition method, model training method, and server disclosed herein include: acquiring target low-resolution video data; preprocessing the target low-resolution video data to obtain a sequence of facial video images; processing the facial video image sequence using a preset fatigue driving behavior recognition model to obtain fatigue driving behavior recognition results in the target low-resolution video data; wherein, the fatigue driving behavior recognition model is obtained by updating the parameters of a preset model based on the annotation type and prediction type of the low-resolution video data to be trained; the prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model. The fatigue driving behavior recognition method, model training method, and server provided in this solution can train the fatigue driving behavior recognition model based on dynamic video, and utilizes the temporal relationship of facial features between multiple frames in the dynamic video, which can improve the recognition accuracy of dynamic video scenes to a certain extent, and is applicable to low-resolution video scenes. Furthermore, preprocessing the target low-resolution video data can further improve the recognition accuracy. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic flowchart illustrating a method for identifying fatigued driving behavior, as shown in an exemplary embodiment of this disclosure.

[0036] Figure 2 A flowchart illustrating a method for identifying fatigued driving behavior, as shown in another exemplary embodiment of this disclosure;

[0037] Figure 3 A schematic diagram of the structure of an in-vehicle low-resolution fatigue video classification system is shown as an exemplary embodiment of this disclosure;

[0038] Figure 4 This is a flowchart illustrating a training method for a fatigue driving behavior recognition model, as shown in an exemplary embodiment of this disclosure.

[0039] Figure 5 A flowchart illustrating a training method for a fatigue driving behavior recognition model, as shown in another exemplary embodiment of this disclosure;

[0040] Figure 6 This is a structural diagram of a fatigue driving behavior recognition device shown in an exemplary embodiment of the present disclosure;

[0041] Figure 7 A structural diagram of a training device for a fatigue driving behavior recognition model, as illustrated in an exemplary embodiment of this disclosure;

[0042] Figure 8 This is a structural diagram of a server shown as an exemplary embodiment of the present disclosure. Detailed Implementation

[0043] With the rapid development of the national economy, the increasing popularity of logistics and transportation and private cars, road safety has become paramount. In particular, unsafe driving behaviors of large logistics truck drivers, such as fatigued driving, often lead to very serious personal and economic harm. Therefore, how to effectively monitor driver behavior and provide real-time warnings for fatigued driving is an urgent problem to be solved.

[0044] In existing technologies, the identification of fatigued driving behavior can be achieved by first acquiring a driver's facial image, then using traditional image algorithms or deep convolutional neural networks to extract the geometric features of key facial points in the facial image (such as the degree of eye opening, blinking frequency, and the degree of mouth opening), and finally using machine learning algorithms to identify fatigued driving behavior based on the extracted key point features.

[0045] However, the above methods do not consider low-resolution video scenarios, making them unsuitable for such situations. Furthermore, these methods primarily use single-frame image features to train the model, failing to model temporal features across multiple frames, resulting in low accuracy in dynamic video scenarios. Additionally, these methods require the localization and annotation of facial landmarks, a complex process prone to errors and inefficient.

[0046] To address the aforementioned technical issues, the solution provided in this disclosure allows for the training of a fatigue driving behavior recognition model based on dynamic video. It leverages the temporal relationships of facial features across multiple frames in the dynamic video, thereby improving the recognition accuracy in dynamic video scenes to a certain extent, and is applicable to low-resolution video scenarios. Furthermore, preprocessing the target low-resolution video data can further enhance the recognition accuracy. This solution only requires assigning a label type to each low-resolution video data point; the labeling process is simple, error-free, and highly efficient.

[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0048] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0049] Figure 1 This is a schematic flowchart illustrating a method for identifying fatigued driving behavior, as shown in an exemplary embodiment of this disclosure.

[0050] like Figure 1 As shown, the fatigue driving behavior identification method provided in this embodiment includes:

[0051] Step 101: Obtain the target low-resolution video data.

[0052] The method provided in this disclosure can be executed by a server.

[0053] This server is capable of acquiring target low-resolution video data.

[0054] Specifically, the target low-resolution video data can be low-resolution video data of the driver's cab in the vehicle collected by the driver monitor system (DMS) installed in the vehicle. The DMS can then send this target low-resolution video data to a server.

[0055] The length of a single target low-resolution video data point can range from 10 seconds to 30 seconds.

[0056] Step 102: Preprocess the target low-resolution video data to obtain a sequence of face video images.

[0057] Specifically, some preprocessing operations can be performed on the received target low-resolution video data. For example, face recognition can be performed on each image included in the target low-resolution video data to identify the face region in the image and crop the face region to generate a face video image sequence.

[0058] Furthermore, for example, to improve facial recognition, video enhancement operations can be performed on the low-resolution video data of the target before facial recognition to improve video clarity, which in turn improves the detection rate of facial recognition.

[0059] Step 103: Process the face video image sequence using a preset fatigue driving behavior recognition model to obtain fatigue driving behavior recognition results in the target low-resolution video data; wherein, the fatigue driving behavior recognition model is obtained by updating the parameters of the preset model according to the annotation type and prediction type of the low-resolution video data to be trained; the prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model.

[0060] Specifically, a pre-set fatigue driving behavior recognition model can be used to process facial video image sequences. This model can identify whether the facial movements of a person in the sequence constitute normal driving behavior, thus obtaining the fatigue driving behavior recognition result from the target low-resolution video data. This result can include either the presence or absence of fatigue driving behavior. For example, if the facial video image sequence shows a person looking left and right, this action can be determined as not being normal driving behavior, and the fatigue driving behavior recognition result is "fatigue driving behavior present." Conversely, if the facial video image sequence shows a person looking straight ahead normally, this action can be determined as normal driving behavior, and the fatigue driving behavior recognition result is "no fatigue driving behavior."

[0061] Furthermore, if the fatigue driving behavior identification results confirm fatigue driving behavior, a warning message can be sent to the vehicle to alert the driver and prevent potential danger.

[0062] Specifically, the fatigue driving behavior recognition model is obtained by updating the parameters of a pre-set model based on the annotation type and prediction type of the low-resolution video data to be trained. This pre-set model can be a pre-defined neural network model.

[0063] Specifically, the low-resolution video data to be trained can be manually labeled in advance to determine the labeling type of the low-resolution video data to be trained. The labeling type can include those showing fatigued driving behavior and those not showing fatigued driving behavior.

[0064] Next, the low-resolution video data to be trained can be preprocessed. For example, face recognition can be performed on each image included in the low-resolution video data to identify the face region in the image and crop that face region to generate a sequence of face video images. Furthermore, for example, to improve face recognition, video enhancement operations can be performed on the low-resolution video data to be trained before face recognition to improve video clarity, thereby improving the detection rate of face recognition.

[0065] Then, the generated facial video image sequence can be processed using a pre-defined model to obtain the prediction type corresponding to the low-resolution video data to be trained. The prediction type can include whether there is fatigued driving behavior or not.

[0066] Then, based on the annotation type and prediction type of the low-resolution video data to be trained, the parameters of the pre-set model are updated to obtain the fatigue driving behavior recognition model.

[0067] The fatigue driving behavior recognition method disclosed herein includes: acquiring target low-resolution video data; preprocessing the target low-resolution video data to obtain a sequence of facial video images; processing the facial video image sequence using a preset fatigue driving behavior recognition model to obtain fatigue driving behavior recognition results in the target low-resolution video data; wherein, the fatigue driving behavior recognition model is obtained by updating the parameters of a preset model based on the annotation type and prediction type of the low-resolution video data to be trained; the prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model. The fatigue driving behavior recognition method provided by this solution can train the fatigue driving behavior recognition model based on dynamic video, and utilizes the temporal relationship of facial features between multiple frames in dynamic video, which can improve the recognition accuracy of dynamic video scenes to a certain extent, and is applicable to low-resolution video scenes. Furthermore, preprocessing the target low-resolution video data can further improve the recognition accuracy. This solution only requires one annotation type for each low-resolution video data point; the annotation process is simple, less prone to errors, and highly efficient.

[0068] Figure 2 This is a flowchart illustrating a method for identifying fatigued driving behavior, as shown in another exemplary embodiment of this disclosure.

[0069] like Figure 2 As shown, the fatigue driving behavior identification method provided in this embodiment includes:

[0070] Step 201: Obtain the target low-resolution video data.

[0071] Specifically, the principle and implementation of step 201 are similar to those of step 101, and will not be repeated here.

[0072] Step 202: Perform video super-resolution reconstruction on the target low-resolution video data to obtain the video data after video super-resolution reconstruction.

[0073] Specifically, a pre-set method can be used to perform video super-resolution reconstruction on the target low-resolution video data to obtain the video data after video super-resolution reconstruction, thereby improving the video clarity and thus improving the detection rate of subsequent face recognition.

[0074] For example, the Real-time Super-Resolution System of 4K Video Based on Deep Learning (EGVSR) algorithm can be used to perform video super-resolution reconstruction on target low-resolution video data.

[0075] Alternatively, video super-resolution reconstruction can be performed using the Temporally Coherent Gans for Video Super-Resolution (TECoGAN) algorithm, the Frame-Recurrent Video Super-Resolution (FRVSR) algorithm, or the Learning for Video Super-Resolution through High-Resolution Optical Flow Estimation (SOF-VSR) algorithm.

[0076] Step 203: According to a preset method, extract a preset number of video frame images from the video data after video super-resolution reconstruction, and determine the extracted first number of video frame images as the target video frame images.

[0077] The preset first frame number is a value set in advance based on the actual situation.

[0078] Specifically, a preset method can be used to extract a preset number of video frame images from the video data after video super-resolution reconstruction, and the extracted first number of video frame images can be determined as the target video frame images.

[0079] In one possible implementation, frames are extracted from the video data after video super-resolution reconstruction, according to a preset number of second frames extracted every preset time interval, to obtain the second number of video frame images.

[0080] The preset second frame count is a value set in advance based on the actual situation. For example, the preset second frame count can be 5 frames, 1 frame, or 3 frames.

[0081] The preset time is a time length that is pre-set according to the actual situation. For example, the preset time can be set to 1 second.

[0082] Specifically, frames can be extracted from the video data after video super-resolution reconstruction, according to the method of extracting a preset second number of frames per second, to obtain the second number of video frame images.

[0083] A preset number of video frames are uniformly extracted from the second number of video frames, and the extracted first number of video frames are determined as the target video frames.

[0084] Specifically, a total of a preset number of video frames can be uniformly extracted from the second number of video frames, and the extracted first number of video frames can be determined as the target video frame images.

[0085] For example, if the intermediate video data contains 200 video frames, and the first frame is preset to be 25 frames, then one video frame can be extracted from the intermediate video data every 8 frames, for a total of 25 video frames, and the extracted video frame images are determined as the target video frame images.

[0086] Step 204: Identify and extract face images from the target video frame; and perform face super-resolution reconstruction on the face images to obtain a face video image sequence.

[0087] Specifically, in order to reduce the interference of a large amount of background irrelevant information on subsequent fatigue driving behavior recognition, a pre-set method can be used to identify the face image in the target video frame image, and the face region where the face image is located can be identified. The face region can be extracted from the target video frame image to obtain the extracted face image.

[0088] For example, a high-accuracy CPU real-time face detector (Faceboxes) algorithm can be used to identify and extract face images from target video frames. Alternatively, a face detection and alignment algorithm using multitask cascaded convolutional networks (MTCNN) or a practical facial landmark detector (PFLD) algorithm can be used to identify and extract face images from target video frames.

[0089] Since the aforementioned video super-resolution reconstruction mainly targets global video information and does not perform more refined reconstruction of local facial regions, a pre-set method is used here to perform face super-resolution reconstruction on the face images to obtain a sequence of face video images. Specifically, using face super-resolution reconstruction can improve the accuracy of facial feature extraction, facilitating subsequent direct identification of facial video fatigue states based on action recognition, and avoiding the complex post-processing procedures in existing technologies.

[0090] For example, face super-resolution reconstruction can be performed using the self-supervised photo upsampling via latent space exploration of generative models (PULSE) algorithm.

[0091] Alternatively, face super-resolution reconstruction can be performed using the Photo-realistic single image super-resolution using a generative adversarial network (SRGAN) algorithm.

[0092] In one possible implementation, if it is determined that no face image is recognized in the target video frame image, then another target video frame image that is closest in time to the target video frame image and can recognize a face image is obtained.

[0093] Specifically, if it is determined that no face image is identified in the target video frame image, other target video frame images that are closest in time to the target video frame image and can identify face images can be obtained first.

[0094] Then, if it is determined that there are two other target video frame images that are closest to the target video frame image in time and can both identify face images, then according to the preset selection method, one of the other target video frame images is selected, and the face image corresponding to the selected other target video frame image is determined as the face image corresponding to the target video frame image.

[0095] Specifically, if it is determined that there are two other target video frame images that are closest to the target video frame image in time and both can identify face images, then according to the preset selection method, one of the two other target video frame images can be selected, and the face image corresponding to the selected other target video frame image can be determined as the face image corresponding to the target video frame image.

[0096] Specifically, for example, one could select the target video frame image from the two other target video frame images that appears before the time of the target video frame image, and use the face image corresponding to the selected target video frame image as the face image corresponding to the target video frame image. Alternatively, one could select the target video frame image from the two other target video frame images that appears after the time of the target video frame image, and use the face image corresponding to the selected target video frame image as the face image corresponding to the target video frame image.

[0097] or,

[0098] If it is determined that there is one and only one other target video frame image that is closest to the target video frame image in time and can identify a face image, then the face image corresponding to the other target video frame image is determined as the face image corresponding to the target video frame image.

[0099] Specifically, if it is determined that there is one and only other target video frame image that is closest to the target video frame image in time and can identify a face image, then the face image corresponding to the other target video frame image can be used as the face image corresponding to the target video frame image.

[0100] Optionally, all acquired target video frame images can be sorted in chronological order. If it is determined that no face image is recognized in the Nth target video frame image, first determine whether the Nith target video frame image exists. If the Nith target video frame image exists, then check whether a face image can be recognized in the Nith target video frame image. If a face image can be recognized in the Nith target video frame image, then the face image corresponding to the Nith target video frame image can be used as the face image in the Nth target video frame image; if a face image cannot be recognized in the Nith target video frame image, then... First, determine if the target video frame (N+i) exists. If it does, check if a face image can be identified within it. If a face image is identified, it is used as the face image in the Nth frame. If no face image is identified, increment i and continue the process of determining if the target video frame (Ni) exists until the face image corresponding to the Nth frame is found. Here, i = 1, 2, 3, 4, ...; N is a positive integer.

[0101] Step 205: Process the face video image sequence using a preset fatigue driving behavior recognition model to obtain fatigue driving behavior recognition results in the target low-resolution video data; wherein, the fatigue driving behavior recognition model is obtained by updating the parameters of the preset model according to the annotation type and prediction type of the low-resolution video data to be trained; the prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model.

[0102] Specifically, a pre-set fatigue driving behavior recognition model can be used to process facial video image sequences. This model can identify whether the facial movements of a person in the sequence constitute normal driving behavior, thus obtaining the fatigue driving behavior recognition result from the target low-resolution video data. This result can include either the presence or absence of fatigue driving behavior. For example, if the facial video image sequence shows a person looking left and right, this action can be determined as not being normal driving behavior, and the fatigue driving behavior recognition result is "fatigue driving behavior present." Conversely, if the facial video image sequence shows a person looking straight ahead normally, this action can be determined as normal driving behavior, and the fatigue driving behavior recognition result is "no fatigue driving behavior."

[0103] Since most existing technologies rely on single-frame images for fatigue driving behavior recognition, neglecting temporal information between multiple frames, they are prone to false detections. Therefore, extensive post-processing is employed to smooth the results of single frames at the video level, complicating the overall process and reducing accuracy. In contrast, the solution provided in this embodiment preprocesses the low-resolution video data to be trained using the aforementioned method to obtain a sequence of face video images. Then, a pre-set fatigue driving behavior recognition model is used to process the face video image sequence to obtain the fatigue driving behavior recognition result from the target low-resolution video data. This significantly improves recognition accuracy and simplifies the overall process in existing technologies.

[0104] Furthermore, if the fatigue driving behavior identification results confirm fatigue driving behavior, a warning message can be sent to the vehicle to alert the driver and prevent potential danger.

[0105] Specifically, the fatigue driving behavior recognition model is obtained by updating the pre-set model parameters based on the annotation type and prediction type of the low-resolution video data to be trained.

[0106] Specifically, the low-resolution video data to be trained can be manually labeled in advance to determine the labeling type of the low-resolution video data to be trained. The labeling type can include those showing fatigued driving behavior and those not showing fatigued driving behavior.

[0107] Next, the low-resolution video data to be trained can be preprocessed to obtain face video image sequences. The preprocessing method can be found in the above embodiments and will not be repeated here.

[0108] Then, the generated facial video image sequence can be processed using a pre-set model to obtain the prediction type corresponding to the low-resolution video data to be trained. The prediction type can include whether or not there is fatigued driving behavior. Then, based on the annotation type and prediction type of the low-resolution video data to be trained, the parameters of the pre-set model are updated, thereby obtaining the fatigued driving behavior recognition model.

[0109] The preset model can be a pre-set neural network model.

[0110] For example, the Temporal Difference Networks (TDN) algorithm can be used to train the model.

[0111] Alternatively, a pre-defined model can be trained using Temporal Segment Networks (TSN) algorithms or Multipath excitation for action recognition (ActionNet) algorithm modules.

[0112] In one possible implementation, the preset fatigue driving behavior recognition model is obtained by training the preset model using a weighted cross-entropy loss function with added focus loss.

[0113] Specifically, since fatigued samples are far fewer than non-fatigue samples in real-world scenarios, a weighted cross-entropy loss function can be used during training to reduce the impact of training imbalance caused by the imbalance between fatigued and non-fatigue samples. Furthermore, to reduce the impact of the imbalance between easy and difficult samples on model training, a focal loss can be further introduced into the loss function to automatically balance the contribution of easy and difficult samples to the loss during training.

[0114] Furthermore, the overall process of this solution can be easily migrated to other in-vehicle video status analysis scenarios, such as smoking and using a mobile phone, and it still has good accuracy in low-resolution scenarios.

[0115] Figure 3 This is a schematic diagram of the structure of an in-vehicle low-resolution fatigue video classification system, which is an exemplary embodiment of this disclosure.

[0116] like Figure 3 As shown, the DMS vehicle system can collect low-resolution video data from the vehicle's cab (i.e., Figure 3 The system retrieves low-resolution video data and uploads it to the server. The server can receive this low-resolution video data.

[0117] Then, the low-resolution video data can be manually labeled, and the labeled low-resolution video data can be put into a pre-set video frame image library for use when training the model.

[0118] Next, a preset method can be used to perform video super-resolution reconstruction on the low-resolution video data to obtain an enhanced video. Then, a preset method can be used to extract frames from this enhanced video, obtaining video frames. Face detection is then performed on the extracted video frames to identify and extract facial images. These facial images are then cropped, and a preset method is used to perform face super-resolution reconstruction on these facial images (i.e.,...). Figure 3 (Face super-resolution in the image) to obtain a sequence of face video images (i.e. Figure 3 The face video sequence is stored in a video frame image library for use in training the model. The face video image sequence in the video frame image library can be divided into a training set and a test set according to a certain ratio (e.g., 6:4, 7:3, or 8:2); the model is trained using the training set and tested using the test set, thus obtaining a fatigue driving behavior recognition model.

[0119] A pre-set model can be used to process face video image sequences in a video frame image library to obtain a prediction type corresponding to the low-resolution video data. Using this prediction type and the corresponding annotation type of the low-resolution video data in the video frame image library, the parameters in the pre-set model are updated to obtain a fatigue driving behavior recognition model (i.e., Figure 3 (fatigue classification model in the text).

[0120] Furthermore, the obtained fatigue driving behavior recognition model can be used to identify whether fatigue driving behavior exists in target low-resolution video data.

[0121] Figure 4 This is a flowchart illustrating a training method for a fatigue driving behavior recognition model, which is an exemplary embodiment of this disclosure.

[0122] like Figure 4 As shown, the training method for the fatigue driving behavior recognition model provided in this embodiment includes:

[0123] Step 401: Obtain the training dataset; the training dataset includes multiple low-resolution video data to be trained; the low-resolution video data to be trained has a labeling type.

[0124] Step 402: Preprocess the low-resolution video data to be trained to obtain a sequence of face video images.

[0125] Step 403: Process the face video image sequence using a preset model to obtain the prediction type corresponding to the low-resolution video data to be trained.

[0126] Step 404: Update the parameters of the preset model according to the annotation type and prediction type of the low-resolution video data to be trained, and obtain the fatigue driving behavior recognition model; wherein, the fatigue driving behavior recognition model is used to identify whether fatigue driving behavior exists in the low-resolution video data.

[0127] For example, the process of this embodiment can be referred to the above embodiments, and will not be repeated here.

[0128] Figure 5 This is a flowchart illustrating a training method for a fatigue driving behavior recognition model, which is another exemplary embodiment of this disclosure.

[0129] like Figure 5 As shown, the training method for the fatigue driving behavior recognition model provided in this embodiment includes:

[0130] Step 501: Obtain the training dataset; the training dataset includes multiple low-resolution video data to be trained; the low-resolution video data to be trained has a labeling type.

[0131] Step 502: Perform video super-resolution reconstruction on the low-resolution video data to be trained, and obtain the video data after video super-resolution reconstruction.

[0132] Step 503: According to a preset method, extract a preset third number of video frame images from the video data after video super-resolution reconstruction, and determine the extracted third number of video frame images as the target video frame images.

[0133] In one possible approach, frames are extracted from the video data after video super-resolution reconstruction, according to the method of extracting a preset fourth frame number every preset time period, to obtain the fourth frame number of video frame images.

[0134] A preset third number of video frames are uniformly extracted from the fourth number of video frames, and the extracted third number of video frames is determined as the target video frame image.

[0135] Step 504: Identify and extract face images from the target video frame; and perform face super-resolution reconstruction on the face images to obtain a face video image sequence.

[0136] In one possible implementation, if it is determined that no face image is recognized in the target video frame image, then another target video frame image that is closest in time to the target video frame image and can recognize a face image is obtained.

[0137] If it is determined that there are two other target video frame images that are closest to the target video frame image in time and can both identify face images, then according to the preset selection method, one of the two other target video frame images is selected, and the face image corresponding to the selected other target video frame image is determined as the face image corresponding to the target video frame image.

[0138] or,

[0139] If it is determined that there is one and only one other target video frame image that is closest to the target video frame image in time and can identify a face image, then the face image corresponding to the other target video frame image is determined as the face image corresponding to the target video frame image.

[0140] Step 505: Process the face video image sequence using a preset model to obtain the prediction type corresponding to the low-resolution video data to be trained.

[0141] Step 506: Update the parameters of the preset model according to the annotation type and prediction type of the low-resolution video data to be trained, and obtain the fatigue driving behavior recognition model; wherein, the fatigue driving behavior recognition model is used to identify whether fatigue driving behavior exists in the low-resolution video data.

[0142] In one possible implementation, the fatigue driving behavior recognition model is obtained by training a preset model using a weighted cross-entropy loss function with added focus loss.

[0143] For example, the process of this embodiment can be referred to the above embodiments, and will not be repeated here.

[0144] Figure 6 This is a structural diagram of a fatigue driving behavior recognition device illustrated in an exemplary embodiment of the present disclosure.

[0145] like Figure 6 As shown, the fatigue driving behavior identification device 600 provided in this disclosure includes:

[0146] Acquisition unit 610 is used to acquire target low-resolution video data;

[0147] The preprocessing unit 620 is used to preprocess the target low-resolution video data to obtain a face video image sequence;

[0148] The recognition unit 630 is used to process the face video image sequence using a preset fatigue driving behavior recognition model to obtain the fatigue driving behavior recognition result in the target low-resolution video data.

[0149] Among them, the fatigue driving behavior recognition model is obtained by updating the parameters of the preset model based on the annotation type and prediction type of the low-resolution video data to be trained; the prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model.

[0150] The preprocessing unit 620 is specifically used to perform video super-resolution reconstruction on the target low-resolution video data and obtain the video data after video super-resolution reconstruction.

[0151] According to a preset method, a preset number of video frame images are extracted from the video data after video super-resolution reconstruction, and the extracted first number of video frame images are determined as the target video frame images.

[0152] Identify and extract facial images from target video frames; perform face super-resolution reconstruction on the facial images to obtain a sequence of facial video images.

[0153] The preprocessing unit 620 is specifically used to extract frames from the video data after video super-resolution reconstruction, according to the method of extracting a preset number of second frames every preset time period, to obtain the video frame image of the second frame number.

[0154] A preset number of video frames are uniformly extracted from the second number of video frames, and the extracted first number of video frames are determined as the target video frames.

[0155] The preprocessing unit 620 is specifically used to acquire other target video frame images that are closest in time to the target video frame image and can identify face images if it is determined that no face image is identified in the target video frame image.

[0156] If it is determined that there are two other target video frame images that are closest to the target video frame image in time and can both identify face images, then according to the preset selection method, one of the two other target video frame images is selected, and the face image corresponding to the selected other target video frame image is determined as the face image corresponding to the target video frame image.

[0157] or,

[0158] If it is determined that there is one and only one other target video frame image that is closest to the target video frame image in time and can identify a face image, then the face image corresponding to the other target video frame image is determined as the face image corresponding to the target video frame image.

[0159] In one possible implementation, the preset fatigue driving behavior recognition model is obtained by training the preset model using a weighted cross-entropy loss function with added focus loss.

[0160] Figure 7 This is a structural diagram of a training device for a fatigue driving behavior recognition model, as illustrated in an exemplary embodiment of the present disclosure.

[0161] like Figure 7 As shown, the training device 700 for the fatigue driving behavior recognition model provided in this disclosure includes:

[0162] The acquisition unit 710 is used to acquire the training dataset; the training dataset includes multiple low-resolution video data to be trained; the low-resolution video data to be trained has a labeling type;

[0163] The preprocessing unit 720 is used to preprocess the low-resolution video data to be trained and obtain face video image sequences.

[0164] Training unit 730 is used to process face video image sequences using a preset model to obtain the prediction type corresponding to the low-resolution video data to be trained.

[0165] The training unit 730 is also used to update the parameters of the preset model based on the annotation type and prediction type of the low-resolution video data to be trained, so as to obtain the fatigue driving behavior recognition model.

[0166] Among them, the fatigue driving behavior recognition model is used to identify whether fatigue driving behavior exists in low-resolution video data.

[0167] The preprocessing unit 720 is specifically used to perform video super-resolution reconstruction on the low-resolution video data to be trained, and to obtain the video data after video super-resolution reconstruction.

[0168] According to a preset method, a preset third number of video frame images are extracted from the video data after video super-resolution reconstruction, and the extracted third number of video frame images are determined as the target video frame images.

[0169] Identify and extract facial images from target video frames; perform face super-resolution reconstruction on the facial images to obtain a sequence of facial video images.

[0170] The preprocessing unit 720 is specifically used to extract frames from the video data after video super-resolution reconstruction, according to the method of extracting a preset fourth frame number every preset time period, to obtain the video frame image of the fourth frame number.

[0171] A preset third number of video frames are uniformly extracted from the fourth number of video frames, and the extracted third number of video frames is determined as the target video frame image.

[0172] The preprocessing unit 720 is specifically used to acquire other target video frame images that are closest in time to the target video frame image and can identify face images if it is determined that no face image is identified in the target video frame image;

[0173] If it is determined that there are two other target video frame images that are closest to the target video frame image in time and can both identify face images, then according to the preset selection method, one of the two other target video frame images is selected, and the face image corresponding to the selected other target video frame image is determined as the face image corresponding to the target video frame image.

[0174] or,

[0175] If it is determined that there is one and only one other target video frame image that is closest to the target video frame image in time and can identify a face image, then the face image corresponding to the other target video frame image is determined as the face image corresponding to the target video frame image.

[0176] In one possible implementation, the preset fatigue driving behavior recognition model is obtained by training the preset model using a weighted cross-entropy loss function with added focus loss.

[0177] Figure 8 This is a structural diagram of a server shown as an exemplary embodiment of the present disclosure.

[0178] like Figure 8 As shown, the server provided in this embodiment includes:

[0179] Memory 801;

[0180] Processor 802; and

[0181] Computer programs;

[0182] The computer program is stored in memory 801 and configured to be executed by processor 802 to implement any of the methods described above.

[0183] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement any of the methods described above.

[0184] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.

[0185] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying fatigued driving behavior, characterized in that, include: Acquire target low-resolution video data; The target low-resolution video data is preprocessed to obtain a sequence of face video images; The facial video image sequence is processed using a preset fatigue driving behavior recognition model to obtain fatigue driving behavior recognition results in the target low-resolution video data; The fatigue driving behavior recognition model is obtained by updating the parameters of a preset model based on the annotation type and prediction type of the low-resolution video data to be trained; the prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model. The step of preprocessing the target low-resolution video data to obtain a face video image sequence includes: Perform video super-resolution reconstruction on the target low-resolution video data to obtain the video data after video super-resolution reconstruction; According to a preset method, a preset first number of video frame images are extracted from the video data after video super-resolution reconstruction, and the extracted first number of video frame images are determined as the target video frame images. Identify and extract facial images from the target video frame; and perform face super-resolution reconstruction on the facial images to obtain a facial video image sequence.

2. The method according to claim 1, characterized in that, The step of extracting a preset first number of video frame images from the video data after super-resolution reconstruction according to a preset method, and determining the extracted first number of video frame images as the target video frame images, includes: From the video data after super-resolution reconstruction, frames are extracted according to the method of extracting a preset number of second frames every preset time period to obtain the video frame image of the second frame number. A preset first number of video frame images are uniformly extracted from the second number of video frame images, and the extracted first number of video frame images are determined as the target video frame images.

3. The method according to claim 1, characterized in that, If it is determined that no face image is identified in the target video frame image, then other target video frame images that are closest in time distance to the target video frame image and can identify face images are obtained; If it is determined that there are two other target video frame images that are closest to the target video frame image in time and can both identify face images, then according to the preset selection method, one of the two other target video frame images is selected, and the face image corresponding to the selected other target video frame image is determined as the face image corresponding to the target video frame image. or, If it is determined that there is one and only one other target video frame image that is closest to the target video frame image in time and can identify a face image, then the face image corresponding to the other target video frame image is determined as the face image corresponding to the target video frame image.

4. A training method for a fatigue driving behavior recognition model, characterized in that, include: Obtain the training dataset; The training dataset includes multiple low-resolution video data sets to be trained. The low-resolution video data to be trained has a labeled type; The low-resolution video data to be trained is preprocessed to obtain a sequence of face video images; The face video image sequence is processed using a preset model to obtain the prediction type corresponding to the low-resolution video data to be trained; Based on the annotation type and prediction type of the low-resolution video data to be trained, the parameters of the preset model are updated to obtain a fatigue driving behavior recognition model. The fatigue driving behavior recognition model is used to identify whether fatigue driving behavior exists in low-resolution video data. The step of preprocessing the low-resolution video data to be trained to obtain a face video image sequence includes: Perform video super-resolution reconstruction on the low-resolution video data to be trained, and obtain the video data after video super-resolution reconstruction. According to a preset method, a preset third number of video frame images are extracted from the video data after video super-resolution reconstruction, and the extracted third number of video frame images are determined as the target video frame images. Identify and extract facial images from the target video frame; and perform face super-resolution reconstruction on the facial images to obtain a facial video image sequence.

5. A device for identifying fatigued driving behavior, characterized in that, include: The acquisition unit is used to acquire target low-resolution video data. The preprocessing unit is used to preprocess the target low-resolution video data to obtain a face video image sequence; The recognition unit is used to process the face video image sequence using a preset fatigue driving behavior recognition model to obtain the fatigue driving behavior recognition result in the target low-resolution video data. The fatigue driving behavior recognition model is obtained by updating the parameters of a preset model based on the annotation type and prediction type of the low-resolution video data to be trained. The prediction type of the low-resolution video data to be trained is obtained by processing the data obtained after preprocessing the low-resolution video data to be trained based on the preset model; The preprocessing unit is specifically used for: Perform video super-resolution reconstruction on the target low-resolution video data to obtain the video data after video super-resolution reconstruction; According to a preset method, a preset first number of video frame images are extracted from the video data after video super-resolution reconstruction, and the extracted first number of video frame images are determined as the target video frame images. Identify and extract facial images from the target video frame; and perform face super-resolution reconstruction on the facial images to obtain a facial video image sequence.

6. A training device for a fatigue driving behavior recognition model, characterized in that, include: The acquisition unit is used to acquire the training dataset. The training dataset includes multiple low-resolution video data sets to be trained. The low-resolution video data to be trained has a labeled type; The preprocessing unit is used to preprocess the low-resolution video data to be trained to obtain a sequence of face video images. The training unit is used to process the face video image sequence using a preset model to obtain the prediction type corresponding to the low-resolution video data to be trained. The training unit is also used to update the parameters of the preset model according to the annotation type and prediction type of the low-resolution video data to be trained, so as to obtain a fatigue driving behavior recognition model. The fatigue driving behavior recognition model is used to identify whether fatigue driving behavior exists in low-resolution video data. The preprocessing unit is specifically used for: Perform video super-resolution reconstruction on the low-resolution video data to be trained, and obtain the video data after video super-resolution reconstruction. According to a preset method, a preset third number of video frame images are extracted from the video data after video super-resolution reconstruction, and the extracted third number of video frame images are determined as the target video frame images. Identify and extract facial images from the target video frame; and perform face super-resolution reconstruction on the facial images to obtain a facial video image sequence.

7. A server, characterized in that, Includes memory and processor; among which, The memory is used to store computer programs; The processor is configured to read the computer program stored in the memory and execute the method described in any one of claims 1-3 or 4 according to the computer program in the memory.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method described in any one of claims 1-3 or 4.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-3 or 4.

Images