Large model cue word multi-scale spatial-temporal feature fusion fatigue driving detection method

Abstract

The invention belongs to the technical field of fatigue driving detection, and particularly relates to a large model cue word multi-scale spatio-temporal feature fusion fatigue driving detection method, which comprises the following steps of: generating a corresponding cue word by utilizing a pre-trained large model, extracting a semantic embedding vector of the cue word, performing multi-scale sampling on a video frame input into a mixed expert model, and extracting a semantic embedding vector of the cue word; the method comprises the following steps of: cascading cue word features with video sequence channels of various scales, fusing multi-scale features to generate a micro-expression space-time motion feature map, inputting the micro-expression space-time motion feature map into a lightweight Transform network, combining a dynamically updated comprehensive learning vector, outputting a fatigue probability through a classifier, and judging whether fatigue driving exists or not based on threshold early warning. And finally, training the hybrid expert model by using a cross entropy loss function in combination with an optimizer. According to the method, subtle changes of facial expressions of the driver can be comprehensively captured, space-time motion features are efficiently modeled through guidance of large model cues, and an efficient and accurate solution is provided for fatigue driving detection of the vehicle driver.

Description

Technical Field

[0001] The present invention belongs to the technical field of fatigue driving detection, and in particular relates to a fatigue driving detection method integrating multi-scale spatiotemporal features of large model prompt words. Background Art

[0002] Vehicles are one of the most important means of transportation in the world today, and fatigue driving has a huge impact on the safe driving of vehicles and the order of public transportation. According to statistics, fatigue driving can cause a significant decrease in the driver's reaction speed and serious impairment of judgment. The economic losses and personal injuries caused by fatigue driving traffic accidents in my country are relatively high every year.

[0003] Automatic detection of fatigue driving using facial expression recognition is one of the important artificial intelligence fatigue driving detection methods. However, existing methods mainly extract temporal features or single-scale spatial features through traditional methods, such as the existing patent with announcement number CN108053615B, the existing patent with announcement number CN118887810B, the existing patent with announcement number CN117789181B and the existing patent application with publication number CN118097636A. These solutions are difficult to fully capture the subtle changes in the driver's facial expressions, and lack an efficient spatiotemporal motion feature modeling mechanism, resulting in low detection accuracy. Summary of the invention

[0004] The present invention provides a large model prompt word multi-scale spatiotemporal feature fusion fatigue driving detection method, which guides multi-scale comprehensive spatiotemporal feature learning and analysis through a large model prompt word to achieve higher detection accuracy. The method comprises the following steps: S1. Define the expression movement units related to fatigue according to the facial action coding system, use the pre-trained large model to generate the prompt words corresponding to the expression movement units, extract the semantic embedding vector of the prompt words, and generate the prompt word feature matrix P ; S2. Hybrid expert model design and processing: S2.1. Video frame sequence for input hybrid expert model S Perform multi-scale sampling to generate L Video sequences of different scales S l ; The prompt word feature matrix P With each scale video sequence S l Perform channel dimension cascading to achieve the fusion of semantic embedding vector and motion feature, and generate a fused multi-scale feature sequence PS l ; S2.2. For each fused feature sequence PS l, extract multi-scale spatial features F l and multi-scale spatiotemporal features G l ; S2.3. Spatiotemporal features of different scales G l Upsample to a uniform spatial size, cascade and integrate through convolutional layers to generate micro-expression spatiotemporal motion feature maps M ; S2.4, Use lightweight Transformer Network-based spatiotemporal motion feature map of micro-expressions M The spatiotemporal motion characteristics are comprehensively analyzed and the initialized comprehensive learning vector Y Updated to Y’ ; S3, based on Y’ The fatigue state probability is output by the fully connected layer classifier, and combined with the preset threshold, it is determined whether the driver is in a fatigue state; S4. Use the cross entropy loss function combined with the optimizer to train the hybrid expert model.

[0005] In a specific embodiment, in step S1, the expression movement unit includes at least one of eyelid closure, yawning, and head tilt, and the pre-trained large model is CLIP or GPT .

[0006] In a specific embodiment, in step S1, the prompt word feature matrix P The shape is [ N , C P ],in N is the number of prompt words, C P The number of feature channels for each cue word.

[0007] In a specific embodiment, in step S2.1, the input video frame sequence S Perform multi-scale sampling including sampling at least three scales of 1, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size; generate L Video sequences of different scales S l middle, L ≥3, video sequence at each scale S l The shape is [ T , H l , W l , C S ],T is the number of frames, H l , W l is the space size, C S is the original number of channels of the input video frame.

[0008] In a specific embodiment, in step S2.2, using space Transformer Extracting multi-scale spatial features F l , multi-scale spatial features F l The shape is [ T , H l , W l , C F ], usage time Transformer Extracting multi-scale spatiotemporal features G l , multi-scale spatiotemporal features G l The shape is [ T , H l , W l , C G ].

[0009] In a specific embodiment, the specific operation of step S2.3 is: first, the spatiotemporal features of each scale are G l Perform bilinear interpolation upsampling to the maximum scale space size, then cascade the upsampled features in the time dimension to generate cascade features, and finally compress the number of cascade feature channels through a 3×3 convolutional layer to obtain the micro-expression spatiotemporal motion feature map M .

[0010] In a specific embodiment, in step S2.4, the comprehensive learning vector Y The initial shape is [ C M ], lightweight Transformer Output of the network Y’ Retention and Y Same channel dimensions.

[0011] In a specific embodiment, in step S2.5, the preset threshold is 0.5-0.8, and a warning signal is triggered when the fatigue probability output by the classifier exceeds the threshold.

[0012] In a specific embodiment, the specific operation of step S4 is: a) Input video frame sequence S Perform data enhancement and generate training samples; b) Input the training samples into the mixed expert model to generate a comprehensive learning vector Y’ ; c) Calculate the cross entropy loss between the classifier output and the true label; d) Based on the calculated cross entropy loss, AdamW The optimizer back-propagates to update the network parameters of the hybrid expert model and combines the weight decay strategy to prevent overfitting; e) Repeat steps a)-d) until the hybrid expert model converges.

[0013] In a specific embodiment, in step a), the data enhancement includes at least one of random cropping, horizontal flipping, and color jittering; in step d), the learning rate is dynamically adjusted using a cosine annealing learning rate scheduling method; in step d), using Dropout And weight decay techniques to further prevent overfitting.

[0014] The present invention has at least the following beneficial effects: 1. Through the learning mechanism of the cue words of the expression movement unit, combined with the semantic understanding ability of the pre-trained large model, the micro-expression movement features related to fatigue are accurately extracted, solving the problem that traditional methods are not sensitive enough to subtle changes in expression.

[0015] 2. Design the hybrid expert model as multi-scale sampling and coordinate space-time Transformer The cascade structure can simultaneously extract local details and global context information, enhance the adaptability to facial expressions at different resolutions, and overcome the limitations of single-scale feature modeling.

[0016] 3. By combining the semantic embedding vector of the prompt word with the multi-scale video sequence sampled by the hybrid expert model S l Cascade, realize the fusion of semantic embedding vector and motion feature, enhance spatiotemporal features, and combine lightweight Transformer The network's dynamic update of the comprehensive learning vector improves the representation ability of fatigue-related features and solves the problem of spatiotemporal motion feature separation in traditional methods.

[0017] 4. The hybrid expert model adopts a staged feature fusion strategy of upsampling and convolution integration, and adopts lightweight Transformer The network design can preserve key information while reducing computational redundancy.

[0018] In summary, the method of the present invention guides the hybrid expert model based on time, multi-scale space and micro-expression motion units to perform fatigue driving detection through pre-training large-model expression motion unit prompt word learning. It can comprehensively capture the subtle changes in the driver's facial expressions and efficiently model spatiotemporal features, providing an efficient and accurate solution for vehicle driver fatigue driving detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The flowchart of the method of the present invention is shown in FIG. DETAILED DESCRIPTION

[0020] See also Figure 1 The present invention provides a large-model prompt word multi-scale spatiotemporal feature fusion fatigue driving detection method, which specifically includes the following steps: S1, according to the facial action coding system FACS Define the expression movement unit related to fatigue, where the expression movement unit includes at least one of eyelid closure, yawning, and head tilt. Use the pre-trained large model to generate prompt words corresponding to the expression movement unit, such as eye closure, yawning, head drooping, etc., and extract the semantic embedding vector of the prompt word to generate the prompt word feature matrix P , so that the above prompt words are converted into numerical vector representations suitable for machine learning model processing. Among them, the pre-trained large model can be used CLIP , you can also use GPT . Prompt word feature matrix P The shape is [ N , C P ],in N is the number of prompt words, C P The number of feature channels for each cue word.

[0021] S2, hybrid expert model processing: S2.1. Video frame sequence for input hybrid expert model S Multi-scale sampling to capture video frame sequences S Detailed information at different resolutions, generated L Video sequences of different scales S l Among them, the video frame sequence S The preprocessed video frame includes existing basic preprocessing operations such as video frame size standardization, normalization, and color space conversion. The multi-scale sampling includes sampling at least three scales of 1, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size. L ≥3, video sequence at each scale S lThe shape is [ T , H l , W l , C S ], T is the number of frames, H l , W l is the height and width, C S is the original number of channels of the input video frame. P With each scale video sequence S l Perform channel dimension cascading so that at each scale, the previously obtained semantic embedding vector is fused with the motion features of the video frame sequence to generate a fused multi-scale feature sequence PS l ,Doing so allows the hybrid expert model to take into account specific signs of fatigue motion when analyzing each frame.

[0022] S2.2. For each fused feature sequence PS l , use space Transformer Extracting multi-scale spatial features F l , multi-scale spatial features F l The shape is [ T , H l , W l , C F ] in order to capture local patterns in the image and the relationships between them, and use time Transformer Extracting multi-scale spatiotemporal features G l , multi-scale spatiotemporal features G l The shape is [ T , H l , W l , C G ] in order to extract dynamic change information in the time dimension, which is crucial for identifying subtle changes in expression over time.

[0023] S2.3. First, the spatiotemporal characteristics of each scale G lPerform bilinear interpolation upsampling to a uniform spatial size, for example, upsampling to the spatial size of the maximum scale, and then cascade the upsampled features in the time dimension to generate cascade features. Compress the number of cascade feature channels through a 3×3 convolutional layer to integrate information from different scales and obtain the micro-expression spatiotemporal motion feature map. M .

[0024] S2.4. Micro-expression spatiotemporal motion feature map M Input Lightweight Transformer network and introduce a randomly initialized comprehensive learning vector into the input Y , comprehensive learning vector Y The initial shape is [ C M ], lightweight Transformer Network-based spatiotemporal motion feature map of micro-expressions M Comprehensively analyze the spatiotemporal features of Y for Y’ ,and Y’ Retention and Y The same channel dimension. This step is done by a lightweight Transformer The network conducts in-depth learning of the fused features and updates the comprehensive learning vector Y , which strengthens the hybrid expert model’s understanding of the characteristics of temporal, spatial, and expression motion units.

[0025] S3, based on Y’ The fatigue state probability is output by the fully connected layer classifier, and combined with the preset threshold, it is determined whether the driver is in a fatigue state. For example, the preset threshold is 0.5-0.8, and when the fatigue probability output by the classifier exceeds the threshold, a warning signal is triggered, and it is determined that the driver is in a fatigue state.

[0026] S4. Use the cross entropy loss function combined with the optimizer to train the hybrid expert model. The specific operations are as follows: a) Input video frame sequence S Perform data enhancement to generate training samples, where the data enhancement includes at least one of random cropping, horizontal flipping, and color jittering to improve the generalization ability of the model.

[0027] b) Input the training samples into the mixed expert model to generate a comprehensive learning vector Y’ .

[0028] c) Calculate the cross entropy loss between the classifier’s output and the true label.

[0029] d) Based on the calculated cross entropy loss, AdamWThe optimizer back-propagates to update the network parameters of the hybrid expert model and combines the weight decay strategy to prevent overfitting. In this step, the cosine annealing learning rate scheduling method is used to dynamically adjust the learning rate of the hybrid expert model to improve training stability, and use Dropout And weight decay techniques to further prevent overfitting.

[0030] e) Repeat steps a)-d) until the hybrid expert model converges.

[0031] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A large-model prompt word multi-scale spatiotemporal feature fusion fatigue driving detection method, characterized in that: The steps include: S1. Define the expression movement units related to fatigue according to the facial action coding system, use the pre-trained large model to generate the prompt words corresponding to the expression movement units, extract the semantic embedding vector of the prompt words, and generate the prompt word feature matrix P ; S2. Hybrid expert model design and processing: S2.

1. Video frame sequence for input hybrid expert model S Perform multi-scale sampling to generate L Video sequences of different scales S l ; The prompt word feature matrix P With each scale video sequence S l Perform channel dimension cascading to achieve the fusion of semantic embedding vector and motion feature, and generate a fused multi-scale feature sequence PS l ; S2.

2. For each fused feature sequence PS l , extract multi-scale spatial features F l and multi-scale spatiotemporal features G l ; S2.

3. Spatiotemporal features of different scales G l Upsample to a uniform spatial size, cascade and integrate through convolutional layers to generate micro-expression spatiotemporal motion feature maps M ; S2.4, Use lightweight Transformer Network-based spatiotemporal motion feature map of micro-expressions M The spatiotemporal motion characteristics are comprehensively analyzed and the initialized comprehensive learning vector Y Updated to Y’ ; S3, based on Y’ The fatigue state probability is output by the fully connected layer classifier, and combined with the preset threshold, it is determined whether the driver is in a fatigue state; S4. Use the cross entropy loss function combined with the optimizer to train the hybrid expert model.

2. The fatigue driving detection method based on large model prompt word multi-scale spatiotemporal feature fusion according to claim 1 is characterized in that: In step S1, the expression movement unit includes at least one of eyelid closure, yawning, and head tilt, and the pre-trained large model is CLIP or GPT .

3. The method for detecting fatigue driving by integrating multi-scale spatiotemporal features of large model prompt words according to claim 1 is characterized in that: In step S1, the prompt word feature matrix P The shape is [ N , C P ],in N is the number of prompt words, C P The number of feature channels for each cue word.

4. The method for detecting fatigue driving by integrating multi-scale spatiotemporal features of large model prompt words according to claim 2 is characterized in that: In step S2.1, the input video frame sequence S The multi-scale sampling includes sampling at least three scales of 1, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size; generate L Video sequences of different scales S l middle, L ≥3, video sequence at each scale S l The shape is [ T , H l , W l , C S ], T is the number of frames, H l , W l is the space size, C S is the original number of channels of the input video frame.

5. The method for detecting fatigue driving by integrating multi-scale spatiotemporal features of large model prompt words according to claim 4 is characterized in that: In step S2.2, use the space Transformer Extracting multi-scale spatial features F l , multi-scale spatial features F l The shape is [ T , H l , W l , C F ], usage time Transformer Extracting multi-scale spatiotemporal features G l , multi-scale spatiotemporal features G l The shape is [ T , H l , W l , C G ].

6. The method for detecting fatigue driving by integrating multi-scale spatiotemporal features of large model prompt words according to claim 5 is characterized in that: The specific operation of step S2.3 is: first, the spatiotemporal features of each scale G l Perform bilinear interpolation upsampling to the maximum scale space size, then cascade the upsampled features in the time dimension to generate cascade features, and finally compress the number of cascade feature channels through a 3×3 convolutional layer to obtain the micro-expression spatiotemporal motion feature map M .

7. The method for detecting fatigue driving by integrating multi-scale spatiotemporal features of large model prompt words according to claim 6 is characterized in that: In step S2.4, the comprehensive learning vector Y The initial shape is [ C M ], lightweight Transformer Output of the network Y’ Retention and Y Same channel dimensions.

8. The method for detecting fatigue driving by integrating multi-scale spatiotemporal features of large model prompt words according to claim 7 is characterized in that: In step S2.5, the preset threshold is 0.5-0.8, and a warning signal is triggered when the fatigue probability output by the classifier exceeds the threshold.

9. The method for detecting fatigue driving by integrating multi-scale spatiotemporal features of large model prompt words according to claim 7 is characterized in that: The specific operations of step S4 are: a) Input video frame sequence S Perform data enhancement and generate training samples; b) Input the training samples into the mixed expert model to generate a comprehensive learning vector Y’ ; c) Calculate the cross entropy loss between the classifier output and the true label; d) Based on the calculated cross entropy loss, AdamW The optimizer back-propagates to update the network parameters of the hybrid expert model and combines the weight decay strategy to prevent overfitting; e) Repeat steps a)-d) until the hybrid expert model converges.

10. The method for detecting fatigue driving by integrating multi-scale spatiotemporal features of large model prompt words according to claim 9 is characterized in that: In step a), the data enhancement includes at least one of random cropping, horizontal flipping, and color jittering; in step d), the learning rate is dynamically adjusted using a cosine annealing learning rate scheduling method ... Dropout And weight decay techniques to further prevent overfitting.

Images