Driver distraction detection method based on ResPrompt and double cross attention

Abstract

The invention discloses a driver distraction detection method based on ResPrompt and double cross attention, and the method comprises the steps: inputting a front / side visual angle image of a driver distraction behavior into a linear embedding module and a ResPrompt module, carrying out the partitioning of the image through the linear embedding module, generating a feature vector, and generating a prompt vector of the image through the ResPrompt module; performing multi-scale feature extraction on the feature vector of the front / side view angle image and the most matched prompt vector by using a Swindow-Transfomer module to generate each view angle feature; and fusing the double-view-angle features by using a double-cross attention module, inputting the double-view-angle features into a classifier fusing center loss, and performing behavior detection on the features to generate a final detection result. According to the method, the accuracy and robustness of detection can be effectively improved, and reliable technical support is provided for safe operation of semi-automatic and automatic driving vehicles.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving and driver behavior detection, and in particular to a driver distraction behavior detection method based on ResPrompt and dual cross attention fusion dual-view features. Background Art

[0002] In semi-autonomous and autonomous driving, the driver's distracted behavior directly affects the safety of the vehicle. However, existing distraction detection methods have deficiencies in detection accuracy, adaptability, and robustness, and it is difficult to fully ensure the safety of vehicles and passengers. Traditional detection methods usually rely on single-view data or simple behavioral analysis methods, such as capturing the driver's facial expressions, eye movements, and body movements through cameras. When the driver's movements are subtle or the face is partially obscured, these methods are difficult to provide sufficient information, resulting in inaccurate detection results. In addition, traditional methods are prone to missed detections or misjudgments when identifying complex distracting behaviors (such as atypical limb movements or long-term fatigue states).

[0003] Existing detection methods usually need to be retrained when faced with new driving behaviors or scenarios, which not only increases time and computing costs, but also leads to catastrophic forgetting, that is, forgetting existing content when learning new knowledge, which seriously affects the stability and adaptability of the method. This limitation hinders the effective application of detection methods in complex driving situations and also brings high training costs.

[0004] Although multi-view data (such as front and side views) has advantages in improving the comprehensiveness of distraction detection, the existing methods have poor multi-view data fusion effects and often fail to fully integrate information from different views, resulting in the loss of important features or information redundancy, which weakens the accurate judgment of the driver's state. In addition, when dealing with frequent changes in driver behavior, traditional methods often make misjudgments or respond late, lack sufficient robustness, and may pose a potential threat to driving safety.

[0005] In summary, the current driver distraction behavior detection methods urgently need to be improved in terms of detection accuracy, adaptability, and multi-view data fusion in order to cope with complex and changing driving environments. Summary of the invention

[0006] In view of the shortcomings in the prior art, the present invention provides a driver distraction detection method based on ResPrompt and double cross attention, which effectively improves the accuracy and robustness of detection, and provides reliable technical support for the safe operation of semi-automatic and autonomous driving vehicles.

[0007] The present invention achieves the above technical objectives through the following technical means.

[0008] A driver distraction detection method based on ResPrompt and double cross attention:

[0009] Inputting the front / side view images of the driver's distraction behavior into the linear embedding module and the ResPrompt module, wherein the linear embedding module divides the image into blocks and generates a feature vector, and the ResPrompt module generates a prompt vector of the image;

[0010] The Swin-Transformer module is used to extract multi-scale features from the feature vectors of the front / side view images and the most matching hint vectors to generate features for each view.

[0011] The dual-view features are fused using a double cross attention module and input into a classifier with fusion center loss to perform behavior detection on the features and generate the final detection results.

[0012] Furthermore, the linear embedding module divides the image into blocks and generates a feature vector, specifically: the front / side view image is divided into N fixed-size image blocks and L prompt information p, each image block is flattened and mapped to a high-dimensional space through a linear layer to form an embedding vector, and the position information and prompt information are added to the embedding vector to form a feature vector z0:

[0013]

[0014] Among them, x Class is a learnable global feature vector in the detection task, p0,p1,…,p L-1 Indicates L prompt information, represents the embedding vector, E Pos Encode the position.

[0015] Furthermore, the ResPrompt module generates a prompt vector for the image, specifically: the front / side view image is feature extracted through the ResNet18 model to generate a feature vector F for matching the prompt vector t ; Each prompt vector A in the prompt pool j With a learnable key k j Associated, the query function q is introduced, and the feature vector F t Encoded to the same dimension as the key; Calculate the query vector q(F t ) and key k j The similarity between them is selected from the prompt pool and the query vector q(F t )The best matching hint vector.

[0016] Furthermore, the query vector q(F t ) and the key matching process is:

[0017]

[0018] in, represents the query vector q(F t ) selects a subset of keys, is the scoring function between the matching query and the key, calculated using the cosine distance, and the calculation formula is:

[0019]

[0020] Furthermore, the multi-scale feature extraction is specifically as follows: through linear transformation, the best matching prompt vector is adjusted to the same dimension as the image feature vector z0, and the query matrix Q′=[P′; z0] is spliced, where P′ is the prompt vector after linear transformation; the query matrix Q′ after splicing by window multi-head self-attention processing is obtained to obtain the intermediate feature matrix After further processing by a multi-layer perceptron, the shifted window multi-head self-attention further fuses the best matching cue vector and image feature vector by redistributing features among different windows, and finally the multi-layer perceptron is used again for enhancement.

[0021] Furthermore, when using the dual cross attention module to fuse the dual-view features, the query matrix is ​​exchanged between the two views, specifically: the query matrix Q of the frontal view is front The bond matrix K with the side view side Sum value matrix V side To interact, the query matrix Q of the side view side The bond matrix K with the front view front Sum value matrix V front to interact.

[0022] Furthermore, the calculation process in each criss-cross attention module is as follows:

[0023]

[0024] Among them, d front Denotes the dimensions of the query matrix and key matrix for the frontal view, d side Denotes the dimensions of the query matrix and key matrix of the side view, B front and B side is the relative position deviation calculated for each view.

[0025] Furthermore, the center loss is calculated as follows:

[0026]

[0027] Where B represents the image batch size, is the feature vector obtained by fusing the dual-view images of the i-th driver distraction behavior through the dual cross-attention module, y i is the category label of the dual-view image of the i-th driver distraction behavior, Indicates the value corresponding to the category label y in the kth iteration i The category center vector of .

[0028] Furthermore, the updating process of each category center vector is as follows:

[0029]

[0030] Among them, δ(y i =j) is the indicator function, when y i =j takes the value of 1, otherwise it takes the value of 0; ρ∈(0,1) is the learning rate of the category center vector update process; and denotes the center vector of the j-th category in the t-th and t+1-th iterations, respectively, c j Represents the center vector of the j-th category.

[0031] Furthermore, the center loss L c and the cross entropy loss L CE Combined to form the final total loss function of the classifier: L = L CE +αL C , where α is the weight parameter.

[0032] The advantages of the present invention compared with the prior art are:

[0033] (1) Stronger feature extraction capability, avoiding catastrophic forgetting, and improving model adaptability: The present invention can efficiently extract the multi-level behavioral features of the driver through the ResPrompt module, which not only improves the detection accuracy, but also enhances the ability to understand complex behavioral patterns, especially when the driver's movements are subtle or there are occlusions, and can still maintain high accuracy. In addition, the ResPrompt module is combined with the continuous learning algorithm of the prompt pool, so that the distraction detection model does not need to be frequently retrained when facing new driving behaviors or scenarios, effectively avoiding the problem of catastrophic forgetting.

[0034] (2) Deep fusion of multiple perspectives to capture subtle behavioral changes: The distraction detection model of the present invention adopts an innovative solution of fusing dual-perspective data from the front view and the side view, and extracts multi-scale features through the Swin-Transformer module. On this basis, the dual cross-attention module is introduced to achieve deep fusion of data from different perspectives, thereby more accurately capturing subtle changes in driver behavior and improving the robustness of the distraction detection model in complex scenarios.

[0035] (3) Optimize classification performance and reduce the risk of misclassification: The present invention adjusts the loss function of the classifier and introduces the center loss function based on the cross entropy loss function to combine the two. In the feature space, the center loss forces the feature vectors of similar behaviors to cluster at the center of their respective categories, thereby effectively reducing the overlap between different categories. This design not only improves the accuracy of classification, but also reduces the risk of misclassification of new behavior types, and enhances the stability and reliability of the distraction detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The flowchart of driver distraction detection based on ResPrompt and double cross attention according to the present invention;

[0037] Figure 2 A structural diagram of the driver distraction detection based on ResPrompt and double cross attention according to the present invention;

[0038] Figure 3 A specific flow chart of prompt vector selection and key value matching in the ResPrompt module of the present invention;

[0039] Figure 4 This is the internal structure diagram of the Swin-Transfomer module of the present invention;

[0040] Figure 5 This is a structural diagram of the dual cross attention module described in the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0042] like Figure 1 , 2 As shown, the present invention provides a driver distraction detection method based on ResPrompt and double cross attention, comprising the following steps:

[0043] Step 1: The behavior detection model inputs the dual-view image of the driver's distracted behavior;

[0044] The dual-view (front and side) images of driver distraction behavior include normal driving and nine different distracted driving actions, such as using a mobile phone, adjusting the central control, and eating and drinking, providing a variety of behavior categories for the distraction detection model. The dataset of dual-view images of driver distraction behavior contains 147,148 image frames, each with a resolution of 512×512, and the data comes from 44 participants. The diversity and large scale of this data provide rich information for model training, which helps to improve the generalization ability of the distraction detection model and make it more robust when facing different drivers.

[0045] The behavior detection model runs in a computer and includes a linear embedding module, a ResPrompt module, a Swin-Transfomer module, a double cross attention module and a classifier.

[0046] Step 2: Use the linear embedding module to divide the dual-view image of driver distraction behavior into blocks and generate features, and use the ResPrompt module to generate the prompt vector of the image.

[0047] The dual-view images of driver distraction behavior are input into two independent branches for synchronous processing, each of which contains a linear embedding module and a ResPrompt module.

[0048] (1) In each branch, a linear embedding module is used to divide each front / side view image into multiple small blocks, each of which is 32×32 pixels in size. The image is divided into N fixed-size image patches and L prompt information p, which can capture the local details of the driver's behavior. For example, the driver's eyes, hand movements and other behavioral features can be retained and amplified by this method. H and W are the height and width of the image, respectively, and C is the number of channels. Assuming that the size of each image patch is P×P×C (P represents the length and width of the image patch), the image is divided into blocks, as the effective input sequence length of the Swin-Transformer module; all image blocks x on each image p It is expressed as:

[0049]

[0050] Each image block Flatten to form a one-dimensional vector:

[0051]

[0052] Each flattened image block is mapped to a high-dimensional space through a linear layer to form an embedding vector. Assuming the embedding dimension is D, the embedding vector is expressed as:

[0053]

[0054] In order to preserve the position information of the image block, position encoding is added Through position encoding, the behavior detection model can identify the driver's behavior characteristics and changes at different positions, which is helpful for understanding the driver's overall posture and behavior pattern. These position encodings are learnable, and position information and prompt information are added to the embedding vector to form a feature vector z0:

[0055]

[0056] Among them, x Class is a learnable global feature vector in the detection task.

[0057] (2) The front / side view image Enter the ResPrompt module, which includes the ResNet18 model and the prompt pool. Figure 3 The figure shows the specific process of prompt vector selection and key value matching in the ResPrompt module. The input image is extracted through the ResNet18 model to generate a feature vector F for matching the prompt vector. t .

[0058] Define the prompt pool as:

[0059] A={A1,A2,…,A j ,…,A L′}

[0060] Where L′ is the total number of hint vectors, and each hint vector With the eigenvector F t Same embedding size D′ and tag length L p .

[0061] The cue vectors in the cue pool contain contextual information related to the driving scenario, such as cues of specific driver actions or distracting behaviors, helping the behavior detection model dynamically adapt to different tasks.

[0062] In order to dynamically select the most suitable hint vector in the detection task, the present invention designs a query strategy based on key-value pairs. j With a learnable key k j Associated to form the following key-value pairs:

[0063] {(k1,A1),(k2,A2),…,(k j ,A j ),…,(k L′ ,A L′ )}

[0064] The set of all keys is represented by Each key With fixed dimension, D k Represents the dimension of the hint vector.

[0065] Introducing a query function The feature vector F t Encode to the same dimension as the key:

[0066] q(F t )=F t [0,:]

[0067] Among them, F t It is the feature vector extracted by the ResNet18 model, and [0,:] represents the index operation.

[0068] Then, the query vector q(F t ) and key k j The similarity between them is selected from the prompt pool and the query vector q(F t ) The matching process is as follows:

[0069]

[0070] Among them, K Ft represents the query vector q(F t ) selects a subset of keys, containing the N′ most matching keys, indirectly the feature vector F t Provide the best matching hint vector; is the scoring function between the matching query and the key, calculated using the cosine distance, and the calculation formula is:

[0071]

[0072] Through this similarity calculation, the cue pool is able to dynamically select the best matching cue vectors for different driving scenarios and distraction behaviors.

[0073] Step 3: Use the Swin-Transformer module to perform multi-scale feature extraction on the feature vector z0 of the front / side view image and the most matching hint vector.

[0074] In the feature extraction stage, the Swin-Transformer module is used to perform multi-scale fusion of the image feature vector z0 and the best matching cue vector. The best matching cue vector is embedded into the query matrix by splicing and directly participates in the self-attention calculation, thereby providing contextual guidance for the image feature vector z0 in the multi-scale feature extraction process, enabling the distraction detection model to more effectively capture the key behavioral features in driving scenarios. Specifically:

[0075] Through linear transformation, the best matching hint vector is adjusted to the same dimension as the image feature vector z0 and concatenated into a query matrix Q′=[P′;z0], where P′ is the hint vector after linear transformation. In this way, the hint vector is introduced into the attention calculation as part of the query.

[0076] The internal structure of the Swin-Transfomer module is as follows Figure 4 As shown, it includes two main components: window multi-head self-attention (W-MSA) and shifted window multi-head self-attention (SW-MSA).

[0077] The concatenated query matrix Q′ is processed using the window multi-head self-attention (W-MSA) component. The specific formula is as follows:

[0078]

[0079] Among them, LN means layer normalization.

[0080] In the W-MSA component, the cue vector is used as part of the query matrix, so that the attention weight distribution is affected by contextual information, helping the distraction detection model to focus on important features in the driving scenario.

[0081] After W-MSA calculation, the intermediate feature matrix After further processing by a multi-layer perceptron, the formula is:

[0082]

[0083] This step enhances the nonlinear expression ability of features and makes the image feature vector and prompt vector richer.

[0084] Subsequently, the shifted window multi-head self-attention (SW-MSA) component further fuses the best matching cue vector and image feature vector by redistributing features among different windows. The specific formula is as follows:

[0085]

[0086] Through SW-MSA, the distraction detection model exchanges information of image feature vectors and prompt vectors in different windows to achieve global feature sharing and further improve the understanding of driver behavior.

[0087] Finally, the multi-layer perceptron is used again to enhance the features:

[0088]

[0089] The features output in this step combine the image feature vector and the prompt vector, providing accurate behavior representation for subsequent classifier detection tasks.

[0090] Step 4: Use the dual cross attention module to fuse the dual-view features.

[0091] The structure diagram of the double cross attention module is as follows Figure 5 shown.

[0092] Each viewing feature generated by the Swin-Transformer module is recorded as and Where l represents the number of layers of the Swin-Transformer module. The query (Q), key (K) and value (V) of the two perspectives are calculated separately and are denoted as Q front , K front 、V front and Q side , K side 、V side In order to achieve dual-view attention fusion, the query matrix is ​​exchanged between the two views. Specifically, the query matrix Q of the front view is front The bond matrix K with the side view side Sum value matrix V side Interact and vice versa. The calculation process in each cross-attention module is as follows:

[0093]

[0094] Among them, d front Denotes the dimensions of the query matrix and key matrix for the frontal view, d side Denotes the dimensions of the query matrix and key matrix of the side view, B front and B side is the relative position deviation calculated for each view.

[0095] The attention results output by each cross-attention module are recorded as and Through this design, the attention learning of the front view is guided by the image feature vector of the side view, while the attention learning of the side view is guided by the image feature vector of the front view, so that the image feature vectors of the two views can interact with each other, significantly improving the performance of the safety officer task recognition. and As the input of the classifier, it is used to identify different safety officer task states.

[0096] Through the dual cross-attention module, one view absorbs the contextual feature vector of the other view, making up for the information missing from a single view. This module is particularly suitable for complex scenarios, such as when the driver is partially blocked, the system can still accurately identify the driver's state through information from another view.

[0097] Step 5: Use the classifier with fusion center loss to perform behavior detection on the features and generate the final detection results.

[0098] In the classifier, if the distraction detection model is optimized only by minimizing the classification loss, what is obtained is only a classification hyperplane that divides the input samples into different categories. It is difficult to obtain features with high discrimination, which will limit the performance of classification detection. Therefore, in order to improve the accuracy of classification detection, the center loss constraint L is introduced in the classifier. c ,The center loss enhances the discriminative ability of the same behavior features by ,encouraging the feature vectors of similar behavior samples to cluster near their category ,center vector.

[0099] A center vector is randomly initialized for each type of driver distraction behavior dual-view image. During the initialization process, the number of center vectors is the same as the number of driver distraction behavior dual-view image categories in the dataset, and the dimension of each center vector is consistent with the feature vector extracted in the Swin-Transformer module. For each driver distraction behavior dual-view image i in a batch, the Euclidean distance between the target vector and the corresponding category center vector is calculated, and the sum of the squares of these distances is used as the center loss. The center loss is calculated as follows:

[0100]

[0101] Where B represents the batch size, is the feature vector obtained by fusing the dual-view images of the i-th driver distraction behavior through the dual cross-attention module, y i is the category label of the dual-view image of the i-th driver distraction behavior, Indicates the value corresponding to the category label y in the kth iteration i The category center vector of .

[0102] In order to accelerate the convergence of the target vector to the corresponding category center vector, each category center vector is continuously updated during the distraction detection model training process; the update process of each category center vector is as follows:

[0103]

[0104] Among them, δ(y i =j) is the indicator function, when y i =j takes the value of 1, otherwise it takes the value of 0; ρ∈(0,1) is the learning rate of the category center vector update process; and denotes the center vector of the j-th category in the t-th and t+1-th iterations, respectively, c j Represents the center vector of the j-th category.

[0105] In the safety officer task detection, there may be a large feature overlap between different behaviors. After introducing the center loss, by constraining the feature vectors to be more closely clustered near the center of their corresponding categories, the feature overlap can be effectively reduced, thereby improving the model's recognition accuracy for different behaviors.

[0106] Secondly, in order to optimize the classification effect of the distraction detection model, the center loss L c and the cross entropy loss L CE Combined to form the final total loss function:

[0107] L=L CE +αL C

[0108] Among them, L CE is the cross entropy loss, which is used to measure the gap between the predicted results and the true labels in the classification task; L C is the center loss, which is used to encourage the feature vectors of similar samples to be more closely clustered together and reduce intra-class differences; α is a weight parameter used to adjust the balance between cross entropy loss and center loss.

[0109] The optimized classifier can generate more accurate behavior detection results and ultimately output the driver's distracted behavior detection results.

[0110] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A driver distraction detection method based on ResPrompt and double cross attention, characterized by: Inputting the front / side view images of the driver's distraction behavior into the linear embedding module and the ResPrompt module, wherein the linear embedding module divides the image into blocks and generates a feature vector, and the ResPrompt module generates a prompt vector of the image; The Swin-Transformer module is used to extract multi-scale features from the feature vectors of the front / side view images and the most matching hint vectors to generate features for each view. The dual-view features are fused using a double cross attention module and input into a classifier with fusion center loss to perform behavior detection on the features and generate the final detection results.

2. The driver distraction detection method according to claim 1, characterized in that: The linear embedding module divides the image into blocks and generates a feature vector. Specifically, the front / side view image is divided into N fixed-size image blocks and L prompt information p, each image block is flattened and mapped to a high-dimensional space through a linear layer to form an embedding vector, and the position information and prompt information are added to the embedding vector to form a feature vector z0: Among them, x Class is a learnable global feature vector in the detection task, p0, p1, ..., p L-1 Indicates L prompt information, represents the embedding vector, E Pos Encode the position.

3. The driver distraction detection method according to claim 1, characterized in that: The ResPrompt module generates a prompt vector for an image. Specifically, the front / side view image is feature extracted through the ResNet18 model to generate a feature vector F for matching the prompt vector. t ; Each prompt vector A in the prompt pool j With a learnable key k j Associated, the query function q is introduced, and the feature vector F t Encoded to the same dimension as the key; Calculate the query vector q(F t ) and key k j The similarity between them is selected from the prompt pool and the query vector q(F t )The best matching hint vector.

4. The driver distraction detection method according to claim 3, characterized in that: The query vector q(F t ) and the key matching process is: in, represents the query vector q(F t ) selects a subset of keys, is the scoring function between the matching query and the key, calculated using the cosine distance, and the calculation formula is:

5. The driver distraction detection method according to claim 3, characterized in that: The multi-scale feature extraction is specifically as follows: through linear transformation, the best matching prompt vector is adjusted to the same dimension as the image feature vector z0, and the query matrix Q′=[P′; z0] is spliced, where P′ is the prompt vector after linear transformation; the query matrix Q′ after window multi-head self-attention processing is spliced ​​to obtain the intermediate feature matrix After further processing by a multi-layer perceptron, the shifted window multi-head self-attention further fuses the best matching cue vector and image feature vector by redistributing features among different windows, and finally the multi-layer perceptron is used again for enhancement.

6. The driver distraction detection method according to claim 1, characterized in that: When using the dual cross attention module to fuse the dual-view features, the query matrix is ​​exchanged between the two views, specifically: the query matrix Q of the front view front The bond matrix K with the side view side Sum value matrix V side To interact, the query matrix Q of the side view side The bond matrix K with the front view front Sum value matrix V front to interact.

7. The driver distraction detection method according to claim 6, characterized in that: The calculation process in each criss-cross attention module is as follows: Among them, d front Denotes the dimensions of the query matrix and key matrix for the frontal view, d side Denotes the dimensions of the query matrix and key matrix of the side view, B front and B side is the relative position deviation calculated for each view.

8. The driver distraction detection method according to claim 1, characterized in that: The center loss is calculated as follows: Where B represents the image batch size, is the feature vector obtained by fusing the dual-view images of the i-th driver distraction behavior through the dual cross-attention module, y i is the category label of the dual-view image of the i-th driver distraction behavior, Indicates the value corresponding to the category label y in the kth iteration i The category center vector of .

9. The driver distraction detection method according to claim 8, characterized in that: The updating process of each category center vector is as follows: Among them, δ(y i =j) is the indicator function, when y i =j takes the value of 1, otherwise it takes the value of 0; ρ∈(0,1) is the learning rate of the category center vector update process; and denotes the center vector of the j-th category in the t-th and t+1-th iterations, respectively, c j Represents the center vector of the j-th category.

10. The driver distraction detection method according to claim 9, characterized in that: The center loss L c and the cross entropy loss L CE Combined to form the final total loss function of the classifier: L = L CE +αL C , where α is the weight parameter.

Images