A method and device for identifying fatigue driving, a storage medium and an electronic device

Abstract

This application discloses a method, apparatus, storage medium, and electronic device for identifying fatigued driving, comprising: acquiring N frames of images arranged sequentially and at equal intervals in a target video, and extracting regions of interest including the driver from the N frames of images to obtain N first images; inputting the N first images into a pre-trained classification model, and determining whether the target video represents a classification result of fatigued driving after processing by the classification model; wherein the processing of the classification model includes: jointly extracting spatiotemporal features from the N first images to obtain first image features, performing encoding and decoding processing based on the first image features, and outputting a classification result, which is correlated with a preset action group used to characterize the driver's fatigue level. Applying this application can effectively improve the recognition rate of fatigued driving.

Description

Technical Field

[0001] This application relates to technologies in the field of intelligent transportation, and in particular to a method, device, storage medium, and electronic device for identifying fatigued driving. Background Technology

[0002] With the continuous advancement of image technology and neural network technology, intelligent recognition technology in the transportation field has been increasingly widely applied.

[0003] Currently, in the field of intelligent transportation, fatigue driving detection and identification has become an effective proactive safety measure. Current fatigue driving detection and identification methods are typically based on single-frame image processing. Even in fatigue driving detection using multiple frames, features are extracted independently from each frame, and then the features from multiple frames are directly stitched together in chronological order for classification and determination of whether fatigue driving has occurred. This type of fatigue driving detection and identification method has a low recognition rate for fatigue driving. Summary of the Invention

[0004] This application provides a method, device, storage medium, and electronic device for identifying fatigued driving, which can effectively improve the recognition rate of fatigued driving.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] A method for identifying fatigued driving includes:

[0007] Obtain N frames of images arranged sequentially and at equal intervals from the target video, and extract the region of interest, including the driver, from the N frames to obtain the first N frames; wherein, N is a natural number greater than 1.

[0008] The first N frames of images are input into a pre-trained classification model. After processing by the classification model, it is determined whether the target video represents the classification result of fatigued driving.

[0009] The classification model processing includes: jointly extracting spatiotemporal features from the N frames of the first image to obtain first image features; performing encoding and decoding processing based on the first image features; and outputting the classification result, which is related to a preset action group used to characterize the driver's fatigue level.

[0010] Preferably, the way to associate the classification result with the preset action group is as follows:

[0011] The classification model further includes extracting driver state temporal features from the N frames of the first image to determine the classification result; wherein the driver state temporal features include facial features related to the preset action group and arranged in chronological order.

[0012] Preferably, the way to associate the classification result with the preset action group is as follows:

[0013] During the training of the classification model, during the decoding process, a first set of actions that occur sequentially in the N frames of images to characterize the driver's fatigue level is further predicted. A first loss function is determined based on the comparison between the first set of actions and the actual set of actions, and the first loss function is used as the basis for updating the parameters of the classification model. The actions included in the first set of actions are derived from the preset set of actions.

[0014] Preferably, before performing the encoding and decoding processes, the method further includes: determining the temporal features of the driver's state based on the N frames of the first image;

[0015] The encoding based on the first image features includes: encoding the first image features and the driver state temporal features respectively, and then fusing the encoding results to obtain fused features;

[0016] Decoding processing includes: performing the decoding processing on the fused features.

[0017] Preferably, the method further includes:

[0018] During the training of the classification model, the decoding process further predicts the first set of actions that occur sequentially in the N frames of images to characterize the driver's fatigue level. A first loss function is determined based on the comparison between the first set of actions and the actual set of actions, and the first loss function is used as the basis for updating the parameters of the classification model.

[0019] Preferably, determining the temporal features of the driver's state based on the N frames of the first image includes:

[0020] For each frame in the N frames of the first image, the value of a preset feature quantity is determined by a target detection network;

[0021] Based on the preset feature values ​​of the N frames of the first image, calculate the change in the value of each feature value in different frame images;

[0022] The values ​​of each feature quantity and the information on the changes in those values ​​in the same frame image are spliced ​​together, and the splicing results of N frames are arranged in chronological order to form the N*M-dimensional temporal features of the driver's state.

[0023] The preset feature quantities include: the position of the driver's face and the preset key area related to the driver's fatigue level, the head posture angle, and the target action feature quantity determined by the position of the preset key area, where M is the sum of the number of preset feature quantities and their value changes.

[0024] Preferably, the preset key areas include: the eye area, the mouth area, and the tip of the nose;

[0025] The target motion features include the degree of eye opening and closing and the degree of mouth opening and closing.

[0026] Preferably, the step of fusing the encoding results to obtain the fused features includes:

[0027] The encoding results of the first image features and the encoding results of the driver state temporal features are respectively subjected to self-attention encoding to obtain the first encoding feature and the second encoding feature;

[0028] The encoding results of the first image features and the encoding results of the driver state temporal features are subjected to mutual attention encoding to obtain joint encoded features;

[0029] The fused feature is obtained by concatenating the joint coding feature, the first coding feature, and the second coding feature.

[0030] Preferably, when performing the mutual attention encoding, the encoding result of the driver state temporal features is used as the query key, and the encoding result of the first image features is used as the query key and value key to calculate the weighting coefficient.

[0031] Preferably, the method further includes: when training the classification model, after determining the first loss function, using the fusion result of the first loss function and the second loss function as the loss function of the classification model; wherein the second loss function is a loss function determined based on classification information of whether it is fatigued driving.

[0032] Preferably, the first loss function is a loss function with alignment functionality.

[0033] A fatigue driving identification device includes: an image region extraction module and a classification processing module;

[0034] The image region extraction unit is used to acquire N frames of images arranged sequentially and at equal intervals in the target video, and extract the region of interest, including the driver, from the N frames of images to obtain N first frames; wherein, N is a natural number greater than 1.

[0035] The classification processing module is used to input the N frames of the first image into a pre-trained classification model, and after processing by the classification model, determine whether the target video represents the classification result of fatigue driving.

[0036] The classification processing module includes a first image feature extraction unit, an encoding unit, and a decoding unit;

[0037] The first image feature extraction unit is used to jointly extract spatiotemporal features from the N frames of the first image to obtain the first image features;

[0038] The encoding unit is used to perform encoding processing based on the first image features;

[0039] The decoding unit is used to decode the output of the encoding unit and output the classification result; the classification result is related to a preset action group used to characterize the driver's fatigue level.

[0040] Preferably, the classification processing module further includes a driver state temporal feature extraction unit, used to determine the driver state temporal features based on the N frames of the first image; wherein, the driver state temporal features include facial features related to the preset action group and arranged in chronological order;

[0041] The encoding unit is further configured to encode the first image features and the driver state temporal features respectively, and then fuse the encoding results to obtain fused features for output.

[0042] Preferably, the device further includes a training module for training and generating the classification model; and during the training of the classification model, during the decoding process, it further predicts a first set of actions that occur sequentially in the N frames of images to characterize the driver's fatigue level, determines a first loss function based on the comparison between the first set of actions and the actual set of actions, and uses the first loss function as the basis for updating the parameters of the classification model; wherein the actions included in the first set of actions are derived from the preset set of actions.

[0043] Preferably, in the driver state temporal feature extraction unit, determining the driver state temporal features based on the N frames of the first image includes:

[0044] For each frame in the N frames of the first image, the value of a preset feature quantity is determined by a target detection network;

[0045] Based on the preset feature values ​​of the N frames of the first image, calculate the change in the value of each feature value in different frame images;

[0046] The values ​​of each feature quantity and the information on the changes in those values ​​in the same frame image are spliced ​​together, and the splicing results of N frames are arranged in chronological order to form the N*M-dimensional temporal features of the driver's state.

[0047] The preset feature quantities include: the position of the driver's face and the preset key area related to the driver's fatigue level, the head posture angle, and the target action feature quantity determined by the position of the preset key area, where M is the sum of the number of preset feature quantities and their value changes.

[0048] Preferably, the preset key areas include: the eye area, the mouth area, and the tip of the nose;

[0049] The target motion features include the degree of eye opening and closing and the degree of mouth opening and closing.

[0050] Preferably, in the coding unit, the step of fusing the coding results to obtain the fused feature includes:

[0051] The encoding results of the first image features and the encoding results of the driver state temporal features are respectively subjected to self-attention encoding to obtain the first encoding feature and the second encoding feature;

[0052] The encoding results of the first image features and the encoding results of the driver state temporal features are subjected to mutual attention encoding to obtain joint encoded features;

[0053] The fused feature is obtained by concatenating the joint coding feature, the first coding feature, and the second coding feature.

[0054] Preferably, in the encoding unit, when performing the mutual attention encoding, the encoding result of the driver state temporal feature is used as the query key, and the encoding result of the first image feature is used as the query key and value key to calculate the weighting coefficient.

[0055] Preferably, when training the classification model in the training module, after determining the first loss function, the fusion result of the first loss function and the second loss function is used as the loss function of the classification model; wherein, the second loss function is a loss function determined based on classification information of whether it is fatigued driving.

[0056] Preferably, the first loss function is a loss function with alignment functionality.

[0057] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, can implement the fatigue driving identification method described above.

[0058] An electronic device, comprising at least a computer-readable storage medium and a processor;

[0059] The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the fatigue driving identification method described in any of the preceding claims.

[0060] As can be seen from the above technical solution, in this application, N frames of images arranged sequentially and at equal intervals in the target video are acquired, and the region of interest, including the driver, is extracted from the N frames to obtain N first frames. The N first frames are then input into a pre-trained classification model, and after processing by the classification model, it is determined whether the target video represents a classification result of fatigued driving. The processing of the classification model includes: jointly extracting spatiotemporal features from the N first frames to obtain first image features. Therefore, the feature extraction in this application is not merely arranging the features of a single frame in time, but rather jointly extracting spatiotemporal features from multiple frames. This feature extraction result not only reflects the feature information of each frame, but also reflects the intrinsic relationship between multiple frames through spatiotemporal joint information, providing cross-features of multiple frames in the time and space dimensions, thus providing richer information references including the time and space dimensions for subsequent classification of fatigued driving. Next, encoding and decoding processing is performed based on the first image features to output the classification result. A set of actions to characterize the driver's fatigue level is preset, and the classification result output by the classification model is related to this preset set of actions. This approach utilizes spatiotemporal joint features to provide richer information for the classification model, while also relating the classification results to preset sub-actions that reflect the degree of fatigue. This enables more accurate classification of fatigued driving and improves the recognition rate of fatigued driving. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the basic process of the fatigue driving identification method in this application;

[0062] Figure 2 This is a schematic diagram of the specific process of the fatigue driving recognition method in a specific embodiment of this application;

[0063] Figure 3 This is a schematic diagram of the encoding and fusion processing in a specific embodiment of this application;

[0064] Figure 4 This is a schematic diagram of the basic structure of the fatigue driving detection device in this application;

[0065] Figure 5 This is a schematic diagram of the basic structure of the electronic device provided in this application. Detailed Implementation

[0066] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0067] The basic idea of ​​this application is to use the spatiotemporal features of multiple frames of images to participate in the classification of fatigued driving, and to introduce action groups that represent the driver's fatigued state as the basis for classification, thereby improving the accuracy of fatigued driving classification and the recognition rate of fatigued driving.

[0068] Figure 1 This is a schematic diagram illustrating the basic process of the fatigue driving identification method in this application. Figure 1 As shown, the method includes:

[0069] Step 101: Obtain N frames of images arranged sequentially and at equal intervals in the target video, and extract the region of interest, including the driver, from the N frames to obtain the first N frames.

[0070] Fatigue driving identification is typically performed on a target video segment (e.g., the video length can be greater than or equal to 5 seconds). This application provides a fatigue driving identification method for a target video segment.

[0071] In the target video, some or all images can be selected for fatigue driving identification. When selecting some images, N frames can be selected sequentially and at equal intervals from the target video. Here, N is a preset natural number greater than 1, and its value can be set according to hardware conditions and the required accuracy of fatigue driving identification. Smaller intervals and larger N result in higher accuracy of fatigue driving identification, but also place higher demands on hardware conditions.

[0072] For the selected N frames of images, a region of interest, including the driver's region, is extracted to obtain N sub-images, hereinafter referred to as the first image, which are used for fatigue driving identification. Optionally, to more effectively identify fatigue driving, the extracted region of interest may include only the driver's region, thereby excluding background interference.

[0073] Step 102: Input the N frames of the first image into the pre-trained classification model. After processing by the classification model, determine whether the target video represents the classification result of fatigued driving.

[0074] The specific processing of the classification model includes:

[0075] A. Perform spatiotemporal feature extraction on N frames of the first image to obtain the first image features.

[0076] Here, spatiotemporal feature extraction is performed by combining N frames of the first image, and the extracted features are called the first image features. These features not only reflect the feature information of each frame, but also reflect the intrinsic relationship between multiple frames through spatiotemporal joint information, providing cross-features of multiple frames in the time and space dimensions. This provides richer information references, including time and space dimensions, for subsequent classification and determination of fatigue driving.

[0077] When performing spatiotemporal feature extraction, various existing spatiotemporal feature extraction methods can be used, such as spatiotemporal joint self-attention networks like SWIN or 3D CNN networks. The dimension of the extracted first image feature can be represented as N*C*H*W*D, where H, W, and D represent the height, width, and length of the first image feature, respectively, C represents the number of features, and N is the number of first image frames.

[0078] B. Encode and decode based on the first image features to output classification results; the classification results are related to a preset action group used to characterize the driver's fatigue level.

[0079] Here, the classification model performs encoding and decoding based on the first image features to determine the binary classification result: either the target video represents fatigued driving, or the target video does not represent fatigued driving. In this application, a set of preset action groups (usually facial actions), hereinafter referred to as the preset action group, is used to characterize the driver's fatigue level. For example, an action group consisting of opening eyes, closing eyes, squinting, speaking with mouth open, yawning, looking down, and looking up can be used as the preset action group. The actions in the aforementioned action group can be used to characterize the driver's fatigue level. For example, when a driver is fatigued, they usually make actions such as closing eyes, looking down, and yawning, while when a driver is not fatigued, they usually make actions such as opening eyes and looking up. Based on this, these actions can be used to reflect the driver's fatigue level. The classification processing in this application ensures that the classification result is related to the preset action group, thereby more accurately and comprehensively expressing the fatigue level through action descriptions of fatigue level, improving the accuracy of fatigued driving identification, and thus improving the fatigued driving recognition rate.

[0080] More specifically, to ensure that the classification results are related to the preset action groups, there are two possible methods:

[0081] 1) The classification model further includes: extracting the driver's state temporal features from the first N frames of images, and using the driver's state temporal features to determine the classification result; wherein, the driver's state temporal features include facial features related to the preset action group and arranged in chronological order;

[0082] The above method is used to further obtain the temporal features of the driver's state. These features are related to the preset action group and include detailed features that can directly reflect the driver's state (such as the position of the corner of the eye or slight changes in the head angle). They are a supplement to the first image features and are used in the classification model. Through the model training process, the classification model learns the feature information that reflects the degree of fatigue, thereby ensuring that the classification result is related to the preset action group. By supplementing the detailed action features, the classification accuracy is improved.

[0083] 2) When training the classification model, during the decoding process, the first action group that occurs sequentially in N frames of images and is used to characterize the driver's fatigue level is further predicted. The first loss function is determined based on the comparison between the first action group and the actual action group, and the first loss function is used as the basis for updating the classification model parameters; wherein, the actions included in the first action group are derived from the preset action group.

[0084] In the above method, when training the classification model, by detecting the first action group and setting the first loss function, the classification model can identify features related to the preset action group from the input spatiotemporal features, that is, features reflecting the driver's fatigue level. In other words, it identifies actions related to driving fatigue and uses them as the basis for binary classification, thereby ensuring that the classification result of binary classification is related to the preset action group. By judging the timing of actions that constitute fatigue events, the classification accuracy is improved.

[0085] The two methods described above for ensuring the classification results match the preset action groups can be used in this application, or both methods can be used.

[0086] This concludes the fatigue driving identification method process in this application. In the aforementioned processing of this application, on the one hand, richer feature information is provided for fatigue driving identification through joint spatiotemporal feature extraction of multiple frames, thereby improving the accuracy of fatigue driving identification; on the other hand, the accuracy of fatigue driving is improved by associating the classification results with preset action groups.

[0087] The specific implementation of the fatigue driving recognition method in this application is described below through specific embodiments. In this embodiment, a combination of the above two methods is used to ensure that the classification results are correlated with the preset action groups. Figure 2 This is a schematic diagram illustrating the specific process of the fatigue driving recognition method in a particular embodiment of this application. Figure 2 As shown, the method includes:

[0088] Step 201: Obtain N frames of images arranged sequentially and at equal intervals in the target video, extract the region of interest including the driver from the N frames of images, and obtain the first N frames of images.

[0089] The processing in this application requires the use of multi-frame joint feature extraction. Considering that a series of continuous actions reflecting fatigue conditions have a certain duration, optionally, in this embodiment, a video of 5 seconds or more is selected for fatigue driving identification. This step selects N frames from the target video for region of interest extraction.

[0090] The extraction method for the region of interest in this embodiment may specifically include:

[0091] 1) Detect the driver target by passing the complete single-frame image through the target detection framework, and calculate the outer envelope of the driver target bounding boxes detected in N frames of images;

[0092] For ease of processing, the outer envelope is usually taken as a rectangle. That is, the smallest rectangle corresponding to all detected target boxes is taken, including all detected target boxes.

[0093] 2) For each frame in the N frames of images, the outer envelope region is taken as the region of interest (ROI), and the ROI region image in each frame is extracted, which is the first image. A total of N first images are extracted from the N frames of images, which are used as the input for feature extraction in this embodiment.

[0094] Existing methods for identifying fatigued driving either only consider the driver's face, especially the eye area, for feature extraction, or they select the entire frame image for feature extraction. Compared to methods that only consider the driver's face, especially the eye area, the scheme in this embodiment expands the input spatial range, which helps to obtain global macroscopic features. Furthermore, compared to schemes that extract features from the entire image, this embodiment only considers the area of ​​the image containing the driver, thus eliminating background interference.

[0095] Step 202: Perform spatiotemporal feature extraction on the N frames of the first image to obtain the first image features.

[0096] The first N frames of images are subjected to temporal and spatial feature extraction, denoted as the first image features, with a feature dimension of N*C*H*W*D. In this embodiment, a 3D CNN model is used to extract the spatiotemporal features from the input N frames of the first image.

[0097] Step 203: Determine the temporal features of the driver's state based on the N frames of the first image.

[0098] The driver's state temporal features include facial features arranged in chronological order and related to a preset action group. Specifically, features that reflect the driver's fatigue level can be selected, and the values ​​of the corresponding features are determined in the first N frames of the image. Since the driver's state temporal features need to reflect temporal characteristics, they include feature values ​​arranged in chronological order, reflecting state information that changes over time.

[0099] In this embodiment, the process of determining the temporal characteristics of the driver's state may specifically include:

[0100] 1) For each frame in the N frames of images, the value of a preset feature quantity is determined by the object detection network;

[0101] Among them, the preset feature quantities are determined based on preset action groups. The preset feature quantities include three categories: the position of preset key areas of the driver's face that are related to the driver's fatigue level, the head posture angle, and the target action feature quantities determined by the position of the preset key areas;

[0102] Specifically, in this embodiment, common actions during driving are broken down and defined, such as opening eyes, closing eyes, squinting, yawning, speaking with mouth open, looking down, looking up, etc., to form a preset action group. Corresponding to the preset action group, key areas in the preset feature quantities are set, including the eye area, the tip of the nose and the mouth area. The target action feature quantities determined by the position of the preset key areas include the degree of eye opening and closing and the degree of mouth opening and closing.

[0103] In this embodiment, when obtaining the values ​​of the aforementioned feature quantities, for each frame of the N frames of the first image, the position of the key region and the head pose angle can be predicted through an object detection network (e.g., a 2D CNN network). Specifically, this can include the position of the eye region (specifically, the position coordinates of 6 different eye regions), the position coordinates of the nose tip and the mouth region, as well as the head pose angle, such as left and right side tilt angle, up and down pitch angle, and forward and backward rotation angle. Next, the degree of eye opening and mouth opening can be calculated from the position of the key regions. In this way, the preset feature quantity values ​​for each first image can be obtained.

[0104] 2) Based on the preset feature values ​​of N frames of images, calculate the value change information of each feature value in different frames of images;

[0105] To demonstrate the intended action, in addition to acquiring the feature values ​​in the first image of each frame, it is also necessary to acquire the change information of each feature value across multiple frames. Specifically, this could be the motion change of each feature value across different frames, such as the relative change compared to the previous frame and the cumulative change compared to the first frame in the target video. This allows for the determination of the changes in key points in the eye region, which is beneficial for recognizing eye-opening and eye-closing actions, as well as determining the changes in the degree of eye opening and closing and the degree of mouth opening, thus facilitating the recognition of actions such as rapid blinking and yawning.

[0106] 3) The values ​​and changes of each feature in the same frame are spliced ​​together, and the splicing results of N frames are arranged in chronological order to form an N*M-dimensional temporal feature of driver status.

[0107] For the same frame of image, the regions and value changes of all features are stitched together as a row. The stitching results of N frames are arranged in chronological order to form an N*M dimensional temporal feature of the driver's state, where M is the sum of the number of preset feature quantities and their value changes.

[0108] The above can be used to determine the temporal characteristics of the driver's state, including detailed characteristics that directly reflect the driver's state and characterize the degree of driver fatigue.

[0109] The processes in steps 202 and 203 can be executed in parallel or in any order.

[0110] Step 204: Encode the first image features and the driver state temporal features respectively, and fuse the encoding results to obtain the fused features.

[0111] As shown above, the first image features and the driver's state temporal features were obtained through steps 202 and 203 respectively. In this step, after encoding the two types of features, a fusion process is performed to obtain the fused features.

[0112] The method for fusing the encoding results of the two types of features can be designed according to actual needs. The simplest way is to directly concatenate the encoding results of the two types of features. However, since the encoding results of the two types of features are quite different, the training effect of the classification model is relatively limited when the fused features obtained by direct concatenation are used to train the classification model. Based on this, this embodiment provides an exemplary fusion method that can better fuse the encoding results of the two types of features.

[0113] The encoding and fusion processing in this embodiment is as follows: Figure 3 As shown, it can specifically include:

[0114] 1) Encode the location of the first image features and the temporal features of the driver's state;

[0115] Positional encoding can reveal the feature differences between multiple frames of images.

[0116] 2) Perform self-attention encoding on the positional encoding results of the first image features and the positional encoding results of the driver state temporal features to obtain the first encoding feature and the second encoding feature;

[0117] The encoding result obtained by performing self-attention encoding on the first image features is called the first encoding feature, and the encoding result obtained by performing self-attention encoding on the driver state temporal features is called the second encoding feature.

[0118] 3) Perform mutual attention encoding on the first image features and the driver state temporal features to obtain joint encoded features;

[0119] In mutual attention encoding, the driver's state temporal features can be used as the query key, and the first image features can be used as the query key and value key to calculate the weighting coefficient. Since the first image features are richer than the driver's state temporal features, the feature fusion effect of this mutual attention encoding method is more ideal. Of course, in mutual attention encoding, the first image features can also be used as the query key, and the driver's state temporal features can be used as the query key and value key to calculate the weighting coefficient.

[0120] 4) The joint coding feature, the first coding feature, and the second coding feature are concatenated to obtain the fused feature;

[0121] The fusion method described in this embodiment is equivalent to encoding the two types of features through mutual attention, and then adding the self-attention features of each type of feature as "residuals" to the mutual attention encoding result. This avoids only considering the highly similar features between the two types of features during the mutual attention calculation process, while ignoring the unique features of each type of feature. Of course, in practical applications, depending on the requirements, the joint encoded features obtained by encoding the two types of features through mutual attention can also be directly used as the fusion features for subsequent classification model processing.

[0122] Step 205: Decode the fused features to obtain the classification result of whether it is fatigue driving.

[0123] Since the fusion features include spatiotemporal features of the images as well as newly introduced temporal features of the driver's state, the fatigue driving classification based on this can combine more image and driver fatigue level features, effectively improving the classification accuracy of fatigue driving.

[0124] Furthermore, in this embodiment, the classification model can be further improved based on the characteristics of fatigued driving, building upon the existing binary classification model. Specifically, during the training of the classification model, the timing of actions constituting the fatigue event can be further determined during decoding. The model predicts the first action group occurring sequentially in N frames, representing the driver's fatigue level. This first action group is compared with the actual action group to determine their similarity, resulting in a first loss function. This first loss function is then used as the basis for updating the classification model parameters. In other words, in addition to category prediction, the model also predicts the actual action timing combinations occurring in the target video. During training, the actual action timing combinations (referred to as the actual action group) are used as the ground truth, and the similarity between the predicted first action group and the actual action group is calculated, participating in the parameter update of the classification model. Therefore, the classification model in this embodiment is a dual-task prediction model, performing both binary classification prediction (whether it is fatigued driving) and prediction of action timing combinations.

[0125] More specifically, in this embodiment, the decoding processing of fused features and the model parameter update processing during the training of the classification model may include:

[0126] 1) Complete Task 1 - Category Prediction: Predict the category of the target video, i.e., fatigue or non-fatigue. Compare the predicted value with the true category value of the target video and calculate the classification loss, called the second loss function, denoted as Loss_cls;

[0127] 2) Complete Task 2 - Action Temporal Combination Prediction: Predict the first action group that actually occurs in N frames of images and is arranged in chronological order. Compare the predicted first action group with the real action group and calculate the loss function, called the first loss function, denoted as Loss_seq. Since the number of actions in each target video segment is not fixed, and the number of actions in the action group predicted by the classification model is not necessarily equal to the number of actions in the real action group, optionally, the first loss function in this embodiment adopts a loss function with alignment function, such as the CTC loss function or its variant function, so that the number of actions contained in the two can be supplemented before comparing the first action group and the real action group, and then the similarity between the two can be calculated.

[0128] 3) Calculate the loss function for training the classification model based on the first and second loss functions: The total loss function Loss_total during the classification model training process can be calculated based on the first and second loss functions. The specific calculation method can be set according to requirements. In this embodiment, Loss_total = Loss_cls + λLoss_seq, where λ is a balancing coefficient used to balance the order-of-magnitude difference between the two loss functions, ensuring that they are roughly the same and do not excessively favor the prediction results of one branch. Specifically, a pre-training can be performed before the formal training process begins, and λ can be set according to the order-of-magnitude relationship between the two loss functions. During the formal training process, λ remains unchanged.

[0129] 4) Update the classification model parameters based on the loss function of the classification model.

[0130] The above describes the decoding process and model parameter update process in the classification model training of this embodiment. By decomposing, defining, and temporally combining and predicting the actions in the N frames of images, the classification model is forced to focus on the correspondence between each sub-segment and each sub-action in the N frames of images and output them in the order of occurrence. This enables the feature encoder and decoder to have temporal modeling capabilities and micro-action recognition capabilities, helping the model to better understand the correspondence between various actions at the micro level and behavioral categories at the macro level, and to find the most important actions for a certain category of behavior (such as closing eyes, yawning, and slightly nodding, which are important actions when a driver is fatigued). This helps to improve the performance of the category prediction branch of Task 1.

[0131] After generating the classification model through the training process, when actually using the classification model to identify fatigued driving, the category output by the category prediction branch of Task 1 is used as the output of the classification model, that is, as the judgment result of whether the current target video belongs to fatigued driving. When it is determined to be fatigued driving, the driver can be further reminded.

[0132] At this point, Figure 2The fatigue driving recognition method flow of this specific embodiment of the application is now complete. Unlike other fatigue driving recognition methods that only use image features as model input and are limited by single-modality constraints, the fatigue driving recognition method in this application abstracts features that characterize the driver's fatigue level (such as key areas of the eyes and mouth, driver's head posture angle, and the degree of eye and mouth opening and closing, etc.) and constructs a sequence in chronological order as additional input, enriching the input information and supplementing details that are easily overlooked in image features. Simultaneously, the features of the two modalities are fused using self-attention, mutual attention encoding, and residual connections. Furthermore, unlike other fatigue driving recognition methods that classify the recognition results into binary or multiple behavioral categories, this embodiment defines several fatigue-related actions during the training process of the classification model and predicts the temporal combinations of these actions. This allows the classification model to better understand the subtle differences in the order of factor actions or sub-action types between various action combinations, strengthening the model's temporal modeling ability and micro-motion capture ability, thereby improving the accuracy of fatigue driving prediction.

[0133] The above describes a specific implementation of the fatigue driving identification method in this application. This application also provides a fatigue driving identification device, which can be used to implement the above-described fatigue driving identification method. Figure 4 This is a schematic diagram of the basic structure of the fatigue driving detection device in this application, as shown below. Figure 4 As shown, the device includes an image region extraction module and a classification processing module.

[0134] The image region extraction unit is used to acquire N frames of images arranged sequentially and at equal intervals in the target video, and extract the region of interest, including the driver, from the N frames to obtain the first N frames; where N is a natural number greater than 1.

[0135] The classification processing module is used to input the first N frames of images into a pre-trained classification model, and after processing by the classification model, determine whether the target video represents the classification result of fatigued driving.

[0136] The classification processing module includes a first image feature extraction unit, an encoding unit, and a decoding unit;

[0137] The first image feature extraction unit is used to jointly extract spatiotemporal features from N frames of the first image to obtain the first image features;

[0138] The encoding unit is used for encoding processing based on the first image features;

[0139] The decoding unit is used to decode the output of the encoding unit and output the classification result; the classification result is related to the preset action group used to characterize the driver's fatigue level.

[0140] Optionally, the classification processing module may further include a driver state temporal feature extraction unit, used to determine the driver state temporal features based on N frames of the first image; wherein, the driver state temporal features may include facial features related to a preset action group and arranged in chronological order;

[0141] The encoding unit can be further used to encode the first image features and the driver state temporal features respectively, and then fuse the encoding results to obtain the fused features for output.

[0142] Optionally, the device can further train a module to train a classification model; and during the training of the classification model, when performing decoding processing, it can further predict the first action group that occurs sequentially in N frames of images to characterize the driver's fatigue level, determine the first loss function based on the comparison between the first action group and the actual action group, and use the first loss function as the basis for updating the classification model parameters; wherein, the actions included in the first action group are derived from a preset action group.

[0143] Optionally, in the driver state temporal feature extraction unit, the processing of determining the driver state temporal features based on the N frames of the first image may specifically include:

[0144] For each frame in the first N frames of the image, the value of a preset feature quantity is determined by an object detection network;

[0145] Based on the preset feature values ​​of the first N frames, calculate the change in the value of each feature value across different frames.

[0146] The values ​​and changes of each feature in the same frame are stitched together, and the stitching results of N frames are arranged in chronological order to form an N*M-dimensional temporal feature of the driver's state.

[0147] Among them, the preset feature quantities may include: the position of the preset key area of ​​the driver's face related to the driver's fatigue level, the head posture angle, and the target motion feature quantity determined by the position of the preset key area, where M is the sum of the number of preset feature quantities and their value changes.

[0148] Optionally, the preset key areas may include: the eye area, the mouth area, and the tip of the nose;

[0149] Target motion features may include the degree of eye opening and closing and the degree of mouth opening and closing.

[0150] Optionally, in the coding unit, the processing of fusing the coding results to obtain fused features may specifically include:

[0151] The encoding results of the first image features and the encoding results of the driver state temporal features are respectively subjected to self-attention encoding to obtain the first encoding feature and the second encoding feature;

[0152] The encoding results of the first image features and the encoding results of the driver's state temporal features are subjected to mutual attention encoding to obtain joint encoded features;

[0153] The joint coding feature, the first coding feature, and the second coding feature are concatenated to obtain the fused feature.

[0154] Optionally, in the coding unit, when performing mutual attention coding, the coding result of the driver state temporal features can be used as the query key, and the coding result of the first image features can be used as the query key and value key to calculate the weighting coefficient.

[0155] Optionally, when training the classification model in the training module, after determining the first loss function, the fusion result of the first loss function and the second loss function can be used as the loss function of the classification model; wherein, the second loss function is a loss function determined based on the classification information of whether it is fatigue driving.

[0156] Optionally, the first loss function can be a loss function with alignment capabilities.

[0157] This application also provides a computer-readable storage medium that stores instructions, which, when executed by a processor, can perform the steps in the fatigue driving identification method described above. In practical applications, the computer-readable medium may be included in the devices / apparatus / systems of the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium stores instructions, which, when executed by a processor, can perform the steps in the fatigue driving identification method described above.

[0158] According to the embodiments disclosed in this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, but not intended to limit the scope of protection of this application. In the embodiments disclosed in this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0159] Figure 5 An electronic device is also provided for this application. For example... Figure 5As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0160] The electronic device may include a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 502 is executed, a method for identifying fatigued driving can be implemented.

[0161] Specifically, in practical applications, this electronic device may also include components such as a power supply 503 and an input / output unit 504. Those skilled in the art will understand that... Figure 5 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0162] The processor 501 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, it performs various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0163] Memory 502 can be used to store software programs and modules, i.e., the aforementioned computer-readable storage medium. Processor 501 executes various functional applications and data processing by running the software programs and modules stored in memory 502. Memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, application programs required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 502 may also include a memory controller to provide processor 501 with access to memory 502.

[0164] The electronic device also includes a power supply 503 that supplies power to the various components. This power supply can be logically connected to the processor 501 via a power management system, enabling functions such as charging, discharging, and power consumption management. The power supply 503 may also include one or more DC or AC power supplies, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components.

[0165] The electronic device may also include an input / output unit 504, which can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 504 can also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying fatigued driving, characterized in that, include: Obtain N frames of images arranged sequentially and at equal intervals from the target video, and extract the region of interest, including the driver, from the N frames to obtain the first N frames; wherein, N is a natural number greater than 1. The first N frames of images are input into a pre-trained classification model. After processing by the classification model, it is determined whether the target video represents the classification result of fatigued driving. The classification model processing includes: jointly extracting spatiotemporal features from the N frames of the first image to obtain first image features; determining the driver's state temporal features based on the N frames of the first image; encoding the first image features and the driver's state temporal features respectively; fusing the encoding results to obtain fused features; decoding the fused features; and outputting the classification result, which is related to a preset action group used to characterize the driver's fatigue level. The method for associating the classification result with the preset action group is as follows: The classification model further includes extracting driver state temporal features from the N frames of the first image to determine the classification result; wherein the driver state temporal features include facial features related to the preset action group and arranged in chronological order.

2. The method according to claim 1, characterized in that, The method further includes: During the training of the classification model, the decoding process further predicts the first set of actions that occur sequentially in the N frames of images to characterize the driver's fatigue level. A first loss function is determined based on the comparison between the first set of actions and the actual set of actions, and the first loss function is used as the basis for updating the parameters of the classification model.

3. The method according to claim 1 or 2, characterized in that, Determining the temporal features of the driver's state based on the N frames of the first image includes: For each frame in the N frames of the first image, the value of a preset feature quantity is determined by a target detection network; Based on the preset feature values ​​of the N frames of the first image, calculate the change in the value of each feature value in different frame images; The values ​​of each feature quantity and the changes in those values ​​in the same frame image are concatenated together, and the concatenation results of N frames are arranged in chronological order to form the N * M dimensional temporal features of the driver's state. The preset feature quantities include: the position of the driver's face and the preset key area related to the driver's fatigue level, the head posture angle, and the target action feature quantity determined by the position of the preset key area, where M is the sum of the number of preset feature quantities and their value changes.

4. The method according to claim 1 or 2, characterized in that, The process of fusing the encoding results to obtain the fused features includes: The encoding results of the first image features and the encoding results of the driver state temporal features are respectively subjected to self-attention encoding to obtain the first encoding feature and the second encoding feature; The encoding results of the first image features and the encoding results of the driver state temporal features are subjected to mutual attention encoding to obtain joint encoded features; The fused feature is obtained by concatenating the joint coding feature, the first coding feature, and the second coding feature.

5. The method according to claim 4, characterized in that, When performing the mutual attention encoding, the encoding result of the driver state temporal features is used as the query key, and the encoding result of the first image features is used as the query key and value key to calculate the weighting coefficient.

6. The method according to claim 2, characterized in that, The method further includes: when training the classification model, after determining the first loss function, using the fusion result of the first loss function and the second loss function as the loss function of the classification model; wherein, the second loss function is a loss function determined based on classification information of whether it is fatigued driving.

7. The method according to claim 2, characterized in that, The first loss function is a loss function with alignment functionality.

8. A fatigue driving detection device, characterized in that, include: Image region extraction module and classification processing module; The image region extraction unit is used to acquire N frames of images arranged sequentially and at equal intervals in the target video, and extract the region of interest, including the driver, from the N frames of images to obtain N first frames; wherein, N is a natural number greater than 1. The classification processing module is used to input the N frames of the first image into a pre-trained classification model, and after processing by the classification model, determine whether the target video represents the classification result of fatigue driving. The classification processing module includes a first image feature extraction unit, a driver state temporal feature extraction unit, an encoding unit, and a decoding unit. The first image feature extraction unit is used to jointly extract spatiotemporal features from the N frames of the first image to obtain the first image features; The driver state temporal feature extraction unit is used to determine the driver state temporal features based on the N frames of the first image; wherein, the driver state temporal features include facial features related to a preset action group and arranged in chronological order; The encoding unit is used to encode the first image features and the driver state temporal features respectively, and then fuse the encoding results to obtain fused features for output. The decoding unit is used to decode the output of the encoding unit and output the classification result; the classification result is related to a preset action group used to characterize the driver's fatigue level.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instruction is executed by the processor, it can realize the fatigue driving identification method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes at least a computer-readable storage medium and a processor; The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the fatigue driving identification method according to any one of claims 1 to 7.

Images