A method and system for detecting fatigue driving

Abstract

This invention relates to a fatigue driving detection method and system. The method involves obtaining the distance range of the center of the human eye frame marked in the training samples within a video frame. Each frame of the captured driver video is divided into multiple image blocks. The positional encodings of the image blocks are fused and input into an encoder to obtain the encoder's output feature vector. The feature vectors are filtered based on the clustering results of the feature vectors, the position of the corresponding image block in the frame, and the distance range. The feature matrix composed of the filtered feature vectors is multiplied by the transpose of the matrix composed of k object query vectors from the decoder to obtain a feature map of the filtered feature vectors. The filtered feature vectors are sorted based on the feature map, and the object query vectors and the first k sorted feature vectors are used as the input to the decoder. The decoder's output is input to a detection head to obtain the eye closure status in the frame, and fatigue driving is determined based on the number of closed-eye frames detected per unit time.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method and system for detecting driver fatigue. Background Technology

[0002] According to a report by the World Health Organization (WHO), fatigued driving causes a significant number of deaths and injuries each year. Especially during long drives, drivers' attention is easily diminished, reaction time is significantly slowed, and judgment and decision-making abilities are severely impaired. Fatigue driving not only causes drivers to misjudge road conditions in a short period of time, but also directly increases the probability of traffic accidents. Compared to other traffic violations such as drunk driving and speeding, fatigued driving is more insidious, as drivers may continue driving without realizing they are fatigued, making it far more dangerous.

[0003] Fatigue driving monitoring is mainly divided into local and remote monitoring. Local monitoring uses the in-vehicle system to identify fatigue driving, while remote monitoring transmits video to a remote server for monitoring. Local monitoring can significantly reduce data transmission time, ensuring the system can complete fatigue state identification quickly, thereby reducing missed warning opportunities due to data delays. In an in-vehicle environment, computing resources are relatively limited; therefore, improving detection speed while maintaining accuracy and utilizing limited in-vehicle resources is crucial. Summary of the Invention

[0004] To improve monitoring speed, a fatigue driving detection method is provided in a first aspect of the present invention, the method comprising:

[0005] The distance range of the center of the human eye frame labeled in the training sample in the video frame is obtained. Each frame of the captured driver video is divided into multiple image blocks. The position encoding of the image blocks is fused and then input into the encoder to obtain the feature vector output by the encoder. The feature vector is clustered. The feature vector is filtered according to the clustering result, the position of the image block corresponding to the feature vector in the frame and the distance range.

[0006] The feature map of the filtered feature vectors is obtained by multiplying the feature matrix composed of the filtered feature vectors and the transpose of the matrix composed of the k object query vectors of the decoder; the filtered feature vectors are sorted based on the feature map, and the object query vectors and the first k sorted feature vectors are used as the input of the decoder.

[0007] The decoder output is input to the detection head to obtain the eye closure status in the frame, and the number of closed-eye frames detected per unit time is used to determine whether the driver is fatigued.

[0008] Preferably, the step of filtering the feature vector based on the clustering results, the position of the image patch corresponding to the feature vector in the frame, and the distance range specifically involves:

[0009] Calculate the center of the image patch corresponding to the feature vector, and calculate the center of each cluster in the clustering result;

[0010] The similarity between feature vectors in a cluster is adjusted based on the center distance between the image patches corresponding to the two feature vectors and the distance range.

[0011] Calculate the average adjusted similarity among all feature vectors in the cluster, and filter out the feature vectors that are far from the cluster center from the two feature vectors whose adjusted similarity is greater than the average value.

[0012] Preferably, adjusting the similarity between feature vectors in a cluster based on the center distance between the image patches corresponding to the two feature vectors and the distance range specifically involves:

[0013] If the center distance between the image patches corresponding to the two feature vectors is within the specified distance range, then the adjustment factor is set to 0.

[0014] If the center distance between the image blocks corresponding to two feature vectors is not within the specified distance range, then the farthest distance between the centers of the two image blocks in the video frame is obtained, the similarity between the two feature vectors in the cluster is calculated, and the ratio of the distance between the centers of the image blocks corresponding to the two feature vectors to the farthest distance is calculated; 1 minus the ratio is used as an adjustment factor for the similarity.

[0015] The product of similarity and adjustment factor is used as the similarity adjustment result.

[0016] Preferably, the step of sorting the filtered feature vectors based on the feature map specifically involves:

[0017] Calculate the average value of each row in the feature map after removing the maximum value, and calculate the difference between the maximum value and the average value;

[0018] Sort the rows of the feature map in descending order of the difference; or,

[0019] The data in each row of the feature map are divided into two categories according to their size. The absolute difference of the average value of each category and the difference in the number of data in each category are calculated. The rows of the feature map are sorted in descending order according to the product of the absolute difference and the difference.

[0020] Preferably, the step of determining whether driving is due to fatigue based on the number of closed-eye frames detected per unit time specifically involves:

[0021] If the number of closed-eye frames detected per unit time exceeds the threshold, a warning for fatigued driving will be issued; otherwise, no warning will be issued.

[0022] A second aspect of the present invention provides a fatigue driving detection system, the system comprising:

[0023] The filtering module is used to obtain the distance range of the center of the human eye frame marked in the training sample in the video frame. Each frame of the captured driver video is divided into multiple image blocks. The position encoding of the image blocks is fused and then input into the encoder to obtain the feature vector output by the encoder. The feature vector is clustered and filtered according to the clustering result, the position of the image block corresponding to the feature vector in the frame, and the distance range.

[0024] The decoding module is used to multiply the feature matrix composed of filtered feature vectors and the transpose of the matrix composed of k object query vectors of the decoder to obtain the feature map of filtered feature vectors; based on the feature map, the filtered feature vectors are sorted, and the object query vectors and the first k sorted feature vectors are used as the input of the decoder.

[0025] The detection module is used to input the output of the decoder into the detection head to obtain the eye closure status in the frame, and to determine whether the driver is fatigued based on the number of closed-eye frames detected per unit time.

[0026] Preferably, the step of filtering the feature vector based on the clustering results, the position of the image patch corresponding to the feature vector in the frame, and the distance range specifically involves:

[0027] Calculate the center of the image patch corresponding to the feature vector, and calculate the center of each cluster in the clustering result;

[0028] The similarity between feature vectors in a cluster is adjusted based on the center distance between the image patches corresponding to the two feature vectors and the distance range.

[0029] Calculate the average adjusted similarity among all feature vectors in the cluster, and filter out the feature vectors that are far from the cluster center from the two feature vectors whose adjusted similarity is greater than the average value.

[0030] Preferably, adjusting the similarity between feature vectors in a cluster based on the center distance between the image patches corresponding to the two feature vectors and the distance range specifically involves:

[0031] If the center distance between the image patches corresponding to the two feature vectors is within the specified distance range, then the adjustment factor is set to 0.

[0032] If the center distance between the image blocks corresponding to two feature vectors is not within the specified distance range, then the farthest distance between the centers of the two image blocks in the video frame is obtained, the similarity between the two feature vectors in the cluster is calculated, and the ratio of the distance between the centers of the image blocks corresponding to the two feature vectors to the farthest distance is calculated; 1 minus the ratio is used as an adjustment factor for the similarity.

[0033] The product of similarity and adjustment factor is used as the similarity adjustment result.

[0034] Preferably, the step of sorting the filtered feature vectors based on the feature map specifically involves:

[0035] Calculate the average value of each row in the feature map after removing the maximum value, and calculate the difference between the maximum value and the average value;

[0036] Sort the rows of the feature map in descending order of the difference; or,

[0037] The data in each row of the feature map are divided into two categories according to their size. The absolute difference of the average value of each category and the difference in the number of data in each category are calculated. The rows of the feature map are sorted in descending order according to the product of the absolute difference and the difference.

[0038] Preferably, the step of determining whether driving is due to fatigue based on the number of closed-eye frames detected per unit time specifically involves:

[0039] If the number of closed-eye frames detected per unit time exceeds the threshold, a warning for fatigued driving will be issued; otherwise, no warning will be issued.

[0040] In a third aspect of the invention, a computer program is provided that, when executed by a processor, implements the method described in the first aspect of the invention.

[0041] To reduce the computational load of the decoder in fatigue driving monitoring, this invention filters the feature vectors output by the encoder based on clustering and the distance between the corresponding image blocks in the video frame, removing feature vectors that are very close and similar. Then, a feature map of the feature vectors is obtained based on the relationship between the feature vectors and object query vectors. The k object query vectors and the k feature vectors selected using the feature map are then used as input to the decoder. This process reduces the number of feature vectors input to the decoder, decreasing the computational load and thus improving monitoring speed. Attached Figure Description

[0042] Figure 1 This is a flowchart of Example 1;

[0043] Figure 2 This is a schematic diagram of the encoder;

[0044] Figure 3 This is a schematic diagram of the decoder;

[0045] Figure 4 This is a schematic diagram of the recognition results. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of one or more embodiments of this specification, and not all embodiments. Based on the embodiments in one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0047] It should be noted that, unless otherwise specified, one or more embodiments and features described in this specification can be combined with each other. One or more embodiments of this specification will now be described in detail with reference to the accompanying drawings and examples.

[0048] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0049] Example 1: This example provides a method for detecting driver fatigue, such as... Figure 1 As shown, the method includes:

[0050] S1, obtain the distance range of the center of the human eye frame marked in the training sample in the video frame, divide each frame of the captured driver video into multiple image blocks, fuse the position encoding of the image blocks and input them into the encoder to obtain the feature vector output by the encoder, cluster the feature vector, and filter the feature vector according to the clustering result, the position of the image block corresponding to the feature vector in the frame and the distance range.

[0051] The training samples are labeled with human eyes, indicating whether they are open or closed. Since the distance between a person's two eyes is generally fixed and varies little from person to person, especially in video images, the distance between a person's two eyes in a video frame can be obtained from the center of the labeled eye boxes in the training samples. Here, the distance between the eye sockets refers to the distance between the centers of the labeled eye boxes of a person within the same frame.

[0052] Each frame of the video is divided into multiple small blocks, which are called image blocks. For each image block, after flattening, its positional encoding within the frame is added. Then, these image blocks containing positional information are fed into the encoder. The encoder preferably uses a DETR model, but a ViT model encoder can also be used. The encoder outputs multiple feature vectors. In one embodiment, the number and dimension of the input and output features of the encoder are the same. Each input image block corresponds to one feature vector output by the encoder, such as... Figure 2 As shown.

[0053] After obtaining the feature vectors, clustering is performed on them. Clustering methods include, but are not limited to, K-means clustering, hierarchical clustering, and DBSCAN clustering. After clustering, similar feature vectors will be clustered together. Feature vectors that are close in distance and similar often represent the same object in decoding. To reduce the amount of data to be decoded, feature vectors are filtered out. Feature vectors that belong to the same cluster, have a similarity greater than a preset value, and whose corresponding database positions in the frame are less than a certain value are filtered out.

[0054] In another embodiment, filtering the feature vector based on the clustering result, the position of the image patch corresponding to the feature vector in the frame, and the distance range specifically involves:

[0055] Calculate the center of the image patch corresponding to the feature vector, and calculate the center of each cluster in the clustering result;

[0056] The similarity between feature vectors in a cluster is adjusted based on the center distance between the image patches corresponding to the two feature vectors and the distance range.

[0057] Calculate the average adjusted similarity among all feature vectors in the cluster, and filter out the feature vectors that are far from the cluster center from the two feature vectors whose adjusted similarity is greater than the average value.

[0058] For each feature vector, the center position of its corresponding image patch is calculated. The center position of the image patch can be represented by the coordinates of the image patch in the video frame. The center position of each cluster is calculated. For each pair of feature vectors within the same cluster, the distance between the centers of their corresponding image patches is calculated, and the similarity between these two feature vector pairs is also calculated. The similarity between feature vectors in the cluster is adjusted based on the center distance between the image patches corresponding to the two feature vectors and the distance range. Furthermore, the average of the adjusted similarity for all feature vector pairs in the cluster is calculated. If the adjusted similarity is greater than the average, the feature vector farthest from the cluster center in this pair is filtered out. This filters out some very close and similar feature vectors, reducing the amount of data the decoder needs to compute.

[0059] In yet another embodiment, adjusting the similarity between feature vectors in a cluster based on the center distance of the image patches corresponding to the two feature vectors and the distance range specifically involves:

[0060] If the center distance between the image patches corresponding to the two feature vectors is within the specified distance range, then the adjustment factor is set to 0.

[0061] If the center distance between the image blocks corresponding to two feature vectors is not within the specified distance range, then the farthest distance between the centers of the two image blocks in the video frame is obtained, the similarity between the two feature vectors in the cluster is calculated, and the ratio of the distance between the centers of the image blocks corresponding to the two feature vectors to the farthest distance is calculated; 1 minus the ratio is used as an adjustment factor for the similarity.

[0062] The product of similarity and adjustment factor is used as the similarity adjustment result.

[0063] For any two feature vectors in a cluster, the distance between the center positions of their corresponding image patches is first calculated. If the distance is within a preset distance range, the adjustment factor is set to 0, indicating that no similarity adjustment is needed between the two feature vectors. The distance range can avoid filtering feature vectors of the two eye parts.

[0064] If the distance between the center of the image blocks corresponding to two feature vectors is not within the preset distance range, the farthest distance between the centers of all image blocks in the video frame is obtained. Based on the actual distance between the center positions of the image blocks corresponding to these two feature vectors, the ratio of this distance to the farthest distance is calculated. Subtracting this ratio from 1 yields a similarity adjustment factor, which is used to reduce the similarity value of feature vectors that are far apart between image blocks.

[0065] S2, multiply the feature matrix composed of the filtered feature vectors and the transpose of the matrix composed of the k object query vectors of the decoder to obtain the feature map of the filtered feature vectors; sort the filtered feature vectors based on the feature map, and use the object query vectors and the first k sorted feature vectors as the input of the decoder.

[0066] After filtering, for the retained feature vectors, the feature vectors are concatenated row-wise to obtain a feature matrix. The feature matrix is ​​then multiplied by the transpose of the matrix composed of the k object query vectors from the decoder to obtain the feature map. The feature vectors have the same dimension as the object query vectors. The k object query vectors are concatenated row-wise, transposed, and then multiplied by the feature matrix. For example, if the feature matrix is ​​d×5, there are d filtered feature vectors. The matrix composed of the k object query vectors is k×5, which becomes 5×k after transposition. The result of multiplying the feature matrix and the transpose of the matrix composed of the k object query vectors is d×k, meaning the feature map size is d×k. Each row in the feature map represents a vector, and the elements in each row represent the similarity or relevance between the feature matrix and the object query vectors. For example, the third element in the second row of the feature map represents the similarity or relevance between the second filtered feature vector and the third filtered object query vector. This principle is the same as the principle used in the attention mechanism to calculate the attention score based on the query matrix Q and the key matrix. To prevent overfitting, in one embodiment, the feature map of the filtered feature vectors is obtained by multiplying the feature matrix composed of the filtered feature vectors and the transpose of the matrix composed of the k object query vectors of the decoder. Then, the feature matrix composed of the filtered feature vectors is multiplied by a trainable weight matrix, and the matrix composed of the k object query vectors of the decoder is also multiplied by another trainable weight matrix. Finally, the feature matrix multiplied by the trainable weight matrix is ​​multiplied by the transpose of the matrix composed of the k object query vectors multiplied by the trainable weight matrix. Here, the object query vector is also called an Object Query; when there are multiple object query vectors, they are called Object Queries, such as... Figure 3 As shown. Unless otherwise specified, in this invention, the object query vector and the Object Query in the DETR model have the same meaning and function. In DETR, each Object Query can be viewed as a representation of a target, which is compared with each position of the prediction result to find the best matching prediction result. Here, k is a positive integer greater than 2.

[0067] If a feature vector and an object query vector are used to identify a bounding box in subsequent eye state determination, the similarity of this feature vector to other object query matrices will generally be different. The relationship between the feature vector and different object query vectors can be seen through the feature map. Based on this, the filtered feature vectors are further filtered based on the feature map. Specifically, in one embodiment, the average value of each row in the feature map after removing the maximum value is calculated, and the difference between the maximum value and the average value is calculated; the rows of the feature map are then sorted in descending order of the difference.

[0068] For each row in the feature map, calculate the average value after removing the maximum value, and then calculate the difference between the maximum value and the average value of the row. The larger the difference, the more distinct the feature vector is for different object query vectors. If one feature value is large while others are small, this feature vector is likely the feature vector corresponding to the target box, i.e., the eye box. Conversely, if the differences are relatively uniform, this feature vector is similar for all object query vectors, and therefore may not have a corresponding target box. Then, sort the rows of the feature map in descending order of the differences. For example, if the difference in the first row of the feature map is 1 and the difference in the second row is 4, then the second row is placed before the first row.

[0069] In another embodiment, the data in each row of the feature map are divided into two categories according to size, the absolute difference of the average value of each category and the difference in the number of data in each category are calculated, and the rows of the feature map are sorted in descending order according to the product of the absolute difference and the difference.

[0070] Since there are multiple target query vectors, the feature map may contain more than one large value. Therefore, the data in each row is divided into two categories based on size, for example, the feature... Figure 1 If a row has 5 elements: 1.2, 1, 4, 1.9, and 4.3, then 4 and 4.3 will be grouped into one class, and 1.2, 1, and 1.9 into another. The absolute difference of the mean values ​​for each class is calculated. In the example above, the absolute difference is 2.78, the difference in the number of elements is 1, and the final product is 2.78. The product of other rows in the feature map can be calculated in the same way, and then the rows of the feature map are sorted according to the product.

[0071] The first k feature vectors corresponding to the sorted rows are selected, and these k feature vectors, along with the k object query vectors, are input into the decoder. Figure 3 As shown. The invention does not limit the specific structure of the decoder, but preferably uses the decoder in the DETR model, or the Transformer Decoder.

[0072] S3 inputs the decoder output to the detection head to obtain the eye closure status in the frame, and determines whether the driver is fatigued based on the number of closed-eye frames detected per unit time.

[0073] The human eye state corresponding to each output vector of the decoder is obtained by passing the vector output by the decoder through FFN and other methods. Figure 4 The monitoring results for open and closed eyes are shown. The PERCLOS fatigue detection criteria are used to determine whether the driver is driving while fatigued. Specifically, if the number of closed-eye frames detected per unit time exceeds a threshold, fatigue driving is indicated; otherwise, no indication is given.

[0074] Example 2: This example provides a fatigue driving detection system, the system comprising:

[0075] The filtering module is used to obtain the distance range of the center of the human eye frame marked in the training sample in the video frame. Each frame of the captured driver video is divided into multiple image blocks. The position encoding of the image blocks is fused and then input into the encoder to obtain the feature vector output by the encoder. The feature vector is clustered and filtered according to the clustering result, the position of the image block corresponding to the feature vector in the frame, and the distance range.

[0076] The decoding module is used to multiply the feature matrix composed of filtered feature vectors and the transpose of the matrix composed of k object query vectors of the decoder to obtain the feature map of filtered feature vectors; based on the feature map, the filtered feature vectors are sorted, and the object query vectors and the first k sorted feature vectors are used as the input of the decoder.

[0077] The detection module is used to input the output of the decoder into the detection head to obtain the eye closure status in the frame, and to determine whether the driver is fatigued based on the number of closed-eye frames detected per unit time.

[0078] Preferably, the step of filtering the feature vector based on the clustering results, the position of the image patch corresponding to the feature vector in the frame, and the distance range specifically involves:

[0079] Calculate the center of the image patch corresponding to the feature vector, and calculate the center of each cluster in the clustering result;

[0080] The similarity between feature vectors in a cluster is adjusted based on the center distance between the image patches corresponding to the two feature vectors and the distance range.

[0081] Calculate the average adjusted similarity among all feature vectors in the cluster, and filter out the feature vectors that are far from the cluster center from the two feature vectors whose adjusted similarity is greater than the average value.

[0082] Preferably, adjusting the similarity between feature vectors in a cluster based on the center distance between the image patches corresponding to the two feature vectors and the distance range specifically involves:

[0083] If the center distance between the image patches corresponding to the two feature vectors is within the specified distance range, then the adjustment factor is set to 0.

[0084] If the center distance between the image blocks corresponding to two feature vectors is not within the specified distance range, then the farthest distance between the centers of the two image blocks in the video frame is obtained, the similarity between the two feature vectors in the cluster is calculated, and the ratio of the distance between the centers of the image blocks corresponding to the two feature vectors to the farthest distance is calculated; 1 minus the ratio is used as an adjustment factor for the similarity.

[0085] The product of similarity and adjustment factor is used as the similarity adjustment result.

[0086] Preferably, the step of sorting the filtered feature vectors based on the feature map specifically involves:

[0087] Calculate the average value of each row in the feature map after removing the maximum value, and calculate the difference between the maximum value and the average value;

[0088] Sort the rows of the feature map in descending order of the difference; or,

[0089] The data in each row of the feature map are divided into two categories according to their size. The absolute difference of the average value of each category and the difference in the number of data in each category are calculated. The rows of the feature map are sorted in descending order according to the product of the absolute difference and the difference.

[0090] Preferably, the step of determining whether driving is due to fatigue based on the number of closed-eye frames detected per unit time specifically involves:

[0091] If the number of closed-eye frames detected per unit time exceeds the threshold, a warning for fatigued driving will be issued; otherwise, no warning will be issued.

[0092] Example 3: This example provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in Example 1.

[0093] Example 4: This example provides a computer program that, when executed by a processor, implements the method described in Example 1.

[0094] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for the device, playback end, image acquisition end, executable program code, and storage medium are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0095] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0096] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting driver fatigue, characterized in that, The method includes: The distance range of the center of the human eye frame labeled in the training sample in the video frame is obtained. Each frame of the captured driver video is divided into multiple image blocks. The position encoding of the image blocks is fused and then input into the encoder to obtain the feature vector output by the encoder. The feature vector is clustered. The feature vector is filtered according to the clustering result of the feature vector, the position of the image block corresponding to the feature vector in the frame, and the distance range. The feature map of the filtered feature vectors is obtained by multiplying the feature matrix composed of the filtered feature vectors and the transpose of the matrix composed of the k object query vectors of the decoder; the filtered feature vectors are sorted based on the feature map, and the object query vectors and the first k sorted feature vectors are used as the input of the decoder. The decoder output is input to the detection head to obtain the eye closure status in the frame, and the number of closed eye frames detected per unit time is used to determine whether the driver is driving while fatigued. The filtering of feature vectors based on the clustering results of feature vectors, the position of the image patch corresponding to the feature vector in the frame, and the distance range is specifically as follows: Calculate the center of the image patch corresponding to the feature vector, and calculate the center of each cluster in the clustering result; The similarity between feature vectors in a cluster is adjusted based on the center distance between the image patches corresponding to the two feature vectors and the distance range. Calculate the average adjusted similarity among all feature vectors in the cluster, and filter out the feature vectors that are far from the cluster center from the two feature vectors whose adjusted similarity is greater than the average value; The adjustment of the similarity between feature vectors in a cluster based on the center distance of the image patches corresponding to the two feature vectors and the distance range is specifically as follows: If the center distance between the image patches corresponding to the two feature vectors is within the specified distance range, then the adjustment factor is set to 0. If the center distance between the image blocks corresponding to two feature vectors is not within the specified distance range, then the farthest distance between the centers of the two image blocks in the video frame is obtained, the similarity between the two feature vectors in the cluster is calculated, and the ratio of the distance between the centers of the image blocks corresponding to the two feature vectors to the farthest distance is calculated; 1 minus the ratio is used as an adjustment factor for the similarity. The product of similarity and adjustment factor is used as the similarity adjustment result.

2. The method as described in claim 1, characterized in that, The sorting of the filtered feature vectors based on the feature map is specifically as follows: Calculate the average value of each row in the feature map after removing the maximum value, and calculate the difference between the maximum value and the average value; Sort the rows of the feature map in descending order of the difference; or, The data in each row of the feature map are divided into two categories according to their size. The absolute difference of the average value of each category and the difference in the number of data in each category are calculated. The rows of the feature map are sorted in descending order according to the product of the absolute difference and the difference.

3. The method as described in claim 1, characterized in that, The method of determining whether driving is due to fatigue based on the number of closed-eye frames detected per unit time is as follows: If the number of closed-eye frames detected per unit time exceeds the threshold, a warning for fatigued driving will be issued; otherwise, no warning will be issued.

4. A fatigue driving detection system, characterized in that, The system includes: The filtering module is used to obtain the distance range of the center of the human eye frame marked in the training sample in the video frame. Each frame of the captured driver video is divided into multiple image blocks. The position encoding of the image blocks is fused and input into the encoder to obtain the feature vector output by the encoder. The feature vector is clustered. The feature vector is filtered according to the clustering result of the feature vector, the position of the image block corresponding to the feature vector in the frame, and the distance range. The decoding module is used to multiply the feature matrix composed of filtered feature vectors and the transpose of the matrix composed of k object query vectors of the decoder to obtain the feature map of filtered feature vectors; based on the feature map, the filtered feature vectors are sorted, and the object query vectors and the first k sorted feature vectors are used as the input of the decoder. The detection module is used to input the output of the decoder to the detection head to obtain the eye closure status in the frame, and to determine whether the driver is fatigued based on the number of closed eye frames detected per unit time. The filtering of feature vectors based on the clustering results of feature vectors, the position of the image patch corresponding to the feature vector in the frame, and the distance range is specifically as follows: Calculate the center of the image patch corresponding to the feature vector, and calculate the center of each cluster in the clustering result; The similarity between feature vectors in a cluster is adjusted based on the center distance between the image patches corresponding to the two feature vectors and the distance range. Calculate the average adjusted similarity among all feature vectors in the cluster, and filter out the feature vectors that are far from the cluster center from the two feature vectors whose adjusted similarity is greater than the average value; The adjustment of the similarity between feature vectors in a cluster based on the center distance of the image patches corresponding to the two feature vectors and the distance range is specifically as follows: If the center distance between the image patches corresponding to the two feature vectors is within the specified distance range, then the adjustment factor is set to 0. If the center distance between the image blocks corresponding to two feature vectors is not within the specified distance range, then the farthest distance between the centers of the two image blocks in the video frame is obtained, the similarity between the two feature vectors in the cluster is calculated, and the ratio of the distance between the centers of the image blocks corresponding to the two feature vectors to the farthest distance is calculated; 1 minus the ratio is used as an adjustment factor for the similarity. The product of similarity and adjustment factor is used as the similarity adjustment result.

5. The system as described in claim 4, characterized in that, The sorting of the filtered feature vectors based on the feature map is specifically as follows: Calculate the average value of each row in the feature map after removing the maximum value, and calculate the difference between the maximum value and the average value; Sort the rows of the feature map in descending order of the difference; or, The data in each row of the feature map are divided into two categories according to their size. The absolute difference of the average value of each category and the difference in the number of data in each category are calculated. The rows of the feature map are sorted in descending order according to the product of the absolute difference and the difference.

Images