Driver behavior recognition method based on double-branch self-attention mechanism

Abstract

The invention belongs to the technical field of computer vision and deep learning, particularly discloses a driver behavior recognition method based on a double-branch self-attention mechanism, and effectively solves the problem of misrecognition caused by individual behavior differences of drivers in a driver behavior recognition algorithm. The method comprises the following steps: (1) using a public data set, dividing the public data set into a training set, a verification set and a test set according to drivers, and preprocessing data; (2) selecting a network model, removing a final classification layer, and only retaining a convolution layer; (3) constructing a driver behavior recognition model based on a double-branch self-attention mechanism; (4) freezing parameters of the enhanced branch, and training the basic branch; (5) freezing parameters of a basic branch, and training an enhanced branch by using a small sample data set; and (6) classifying the driver behavior recognition model constructed in the step (3) through a linear layer. Accuracy and reliability of practical application of the driver behavior recognition model are improved, and training performance requirements are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning technology, and in particular relates to a driver behavior recognition method based on a dual-branch self-attention mechanism, which is suitable for real-time monitoring of driver status in an autonomous driving system. Background Art

[0002] Research on driver behavior recognition can be divided into physiological data-based, external data-based, and visual data-based methods based on the data they are based on. Among them, (1) physiological data-based driver behavior recognition methods infer the driver's behavior status by monitoring the driver's physiological indicators. Typical physiological data include electroencephalogram, electrocardiogram, electrooculogram, etc. These data reflect the driver's physiological state, such as attention, emotion, fatigue, etc. Although physiological data-based methods have certain potential, their promotion and application are subject to certain constraints due to many limitations such as equipment cost, individual differences, and environmental factors. (2) External data-based driver behavior recognition methods use vehicle-built-in sensors or external devices to collect driving behavior-related data, such as steering wheel angle, vehicle speed, acceleration, GPS location, on-board diagnostic information, etc. These data can provide rich information about driving behavior. However, external data-based methods are affected by factors such as vehicle type, driving environment, and driver habits, and their accuracy and generalization ability are limited. (3) Visual data-based driver behavior recognition methods infer the driver's driving behavior by analyzing information such as the driver's facial features, posture, and in-vehicle environment. This approach not only extracts rich discriminative information but is also robust to device cost and individual differences. Powered by deep learning, methods based on visual data have made significant progress, achieving high accuracy and generalization capabilities. The introduction of lightweight models, in particular, has enabled this approach to be effectively implemented on resource-constrained embedded devices.

[0003] However, because driving behavior is influenced by numerous factors, such as traffic conditions, weather conditions, and the driver's facial expressions, constructing a comprehensive dataset that covers all situations is extremely difficult. Furthermore, driving habits vary significantly among drivers, further complicating the dataset creation process. Consequently, the robustness of trained models for driver behavior recognition cannot be guaranteed, posing challenges to accuracy and reliability in practical applications. Summary of the Invention

[0004] The purpose of the present invention is to provide a driver behavior recognition method based on a dual-branch self-attention mechanism, which effectively solves the problem of incorrect recognition caused by individual driver behavior differences in the driver behavior recognition algorithm.

[0005] To solve the above technical problems, the present invention adopts a technical solution: a driver behavior recognition method based on a dual-branch self-attention mechanism, comprising the following steps:

[0006] S1. Use the public datasets SFD3 and AUCv2, divide the dataset into training set, validation set, and test set according to the driver, take one image from each driver in the test set to construct a small sample dataset, and preprocess the data, including normalization, scaling, and random rotation.

[0007] S2. Select the network model, remove the last classification layer, and only retain the convolution layer to extract the feature information of the driver image.

[0008] S3. Construct the overall structure of the driver behavior recognition model based on the dual-branch self-attention mechanism. Connect the output of the network model in step S2 to the input of DBViT to construct two branches: a basic branch and an enhanced branch. The two branches are feature-fused through DBViT, and a long-range dependency relationship between the features of the two branches is established to construct global features. Finally, the final driver behavior classification is completed through a linear layer.

[0009] S4. Freeze the parameters of the enhancement branch, use the training set divided in step S1 to train the basic branch, and use the validation set to evaluate the training results.

[0010] S5. Freeze the parameters of the base branch and use the small sample data set divided in step S1 to train the enhanced branch.

[0011] S6. After two steps of training in steps S4 and S5, the base branch retains the parameters pre-trained based on the original dataset and focuses on extracting and constructing local feature representations of common driving behaviors. The enhancement branch is fine-tuned for new driver data to capture individual-specific behavior patterns. Finally, the driver behavior recognition model based on the dual-branch self-attention mechanism constructed in step S3 is classified through a linear layer.

[0012] The structure of DBViT described in step S3 is: DBSSA and FFN are stacked together to form a Transformer-DB encoder block; the input of each Transformer-DB encoder block is passed through DBSSA to capture global information and calculate the mutual attention of the two branches, and then passed through FFN for feature transformation, and finally the output features are passed to the next layer.

[0013] DBSSA calculates the context scores of the two branches by attention calculation on the feature maps input by the two branches, then obtains the attention scores by element-by-element multiplication, and applies the attention scores to the backbone.

[0014] Furthermore, in step S1, the dataset is divided according to the driver division method: test set: three drivers' behavior images are manually screened from each dataset to form an independent test set, ensuring that the data of the selected drivers are excluded from the initial training of the model, and a small sample dataset is constructed by taking one image for each driver; training set and validation set: the remaining images in the dataset are divided into training set and validation set in a ratio of 8:2.

[0015] Furthermore, in step S2, the network model is a model with feature extraction capability, including ResNet18 and MobileNet.

[0016] Furthermore, DBSSA calculates the context scores c of the two branches by paying attention to the feature maps input by the two branches. VS and c VT , the calculation steps are:

[0017] (1) For the basic branch, the input x1 uses weights and W V1 ∈R d×d The linear layer is calculated to get and

[0018] Among them, W I1 represents the weight of the linear layer I1, W K1 represents the weight of the linear layer K1, W V1 represents the weight of the linear layer V1, x I1 represents the output of the linear layer I1, x K1 represents the output of the linear layer K1, x V1 Represents the output of the linear layer V1; Represents a two-dimensional space with a size of d×d; Represents a one-dimensional space with a size of d; k×d represents the size of the feature map input to DBSSA.

[0019] Apply the Softmax function to x I1 To generate the context score of the base branch The context score c of the base branch S1 The calculation formula is:

[0020] c S1 =Softmax(x1W I1 );

[0021] Among them, Softmax represents the Softmax function operation.

[0022] The context score c of the base branch S1 and xK1 The context vector of the base branch is obtained by element-by-element multiplication and summation

[0023] The context vector c of the base branch VS The calculation formula is:

[0024] c VS =∑(c S1 *x1W K1 );

[0025] Among them, * represents element-by-element multiplication operation, ∑ represents summation operation, W K1 Represents the weight of the linear layer K1.

[0026] (2) For the enhanced branch, the input x2 uses weights and The linear layer is calculated to get and Among them, W I2 represents the weight of the linear layer I2, W K2 represents the weight of the linear layer K2, x I2 represents the output of the linear layer I2, x K2 Represents the output of the linear layer K2.

[0027] Apply the Softmax function to x I2 To generate the context score c of the enhanced branch S2 ∈R k , the context score c of the enhanced branch S2 The calculation formula is:

[0028] c S2 =Softmax(x2W I2 );

[0029] c S2 、x K2 and the trainable parameter vector The context vector of the enhanced branch is obtained by element-by-element multiplication and summation operations The context vector c of the boosted branch VT The calculation formula is:

[0030] c VT =∑(c S2 *x2W K2 *γ);

[0031] Source domain feature x V1 Input into the activation function ReLU, and then with c VS and c VT After element-by-element multiplication, the input weight is The linear layer gets the final output

[0032] The calculation formula of the two-branch separable self-attention is:

[0033] y=(c VS *c VT *ReLU(x1W V1 ))W OD ;

[0034] Among them, W V1 and W OD Represent the linear layer V1 and linear layer O respectively D The weight of .

[0035] Furthermore, the basic branch inherits the local feature extraction capability of the backbone network and generates high-resolution local feature maps; the enhanced branch constructs a lightweight feature extraction branch through multi-scale convolution to generate auxiliary feature maps.

[0036] The base branch and the enhanced branch match the feature map sizes through dynamic adaptive pooling, and use DBViT to capture the long-range dependencies between the two and construct global features.

[0037] Compared with the prior art, the beneficial technical effects of the present invention are: (1) After the present invention uses a data set to train the basic branch, only a small number of image samples are needed to train the enhanced branch to make the model accuracy reach an applicable level, thereby improving the accuracy and reliability of the actual application of the driver behavior recognition model, reducing the training performance requirements, and also reducing the number of images required for actual application.

[0038] (2) The base branch of the present invention inherits the local feature extraction capability of the backbone network to generate high-resolution local feature maps. The enhancement branch constructs a lightweight feature extraction branch through multi-scale convolution to generate auxiliary feature maps. The base and enhancement branches match the feature map sizes through dynamic adaptive pooling, and DBViT is used to capture the long-range dependencies between the two and construct global features, greatly improving the reliability of the model in real-world situations.

[0039] (3) The DBViT proposed in this paper can be embedded in different network models and is suitable for different image recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of the overall structure of the driver behavior recognition model based on the dual-branch self-attention mechanism in Example 1.

[0041] Figure 2 This is a schematic diagram of the DBViT structure.

[0042] Figure 3 It is a schematic diagram of the structure of DBSSA. DETAILED DESCRIPTION

[0043] Example 1: A driver behavior recognition method based on a dual-branch self-attention mechanism, comprising the following steps:

[0044] S1. Use the public datasets SFD3 and AUCv2, divide the dataset into training set, validation set, and test set according to the driver, take one image from each driver in the test set to construct a small sample dataset, and preprocess the data, including normalization, scaling, and random rotation.

[0045] In this example, the datasets were divided by driver: (1) Test set: Three driver behavior images were manually selected from each dataset to form an independent test set. This ensured that the selected drivers' data were excluded from the initial model training, and a small sample dataset was constructed by taking one image from each driver. (2) Training and validation sets: The remaining images in the dataset were divided into training and validation sets in an 8:2 ratio. The number of images in the SFD3 dataset after the division is shown in Table 1, and the number of images in the AUCv2 dataset after the division is shown in Table 2.

[0046] Table 1 Number of images after SFD3 dataset division

[0047] category Category Name training set Validation set Test set c0 Safe driving 1772 442 275 c1 Texting with right hand 1614 403 250 c2 Right-hand call 1639 409 272 c3 Texting with left hand 1678 419 250 c4 Left-handed call 1657 414 255 c5 Operate the radio 1652 412 248 c6 Drinking water 1656 414 255 c7 Reach behind 1405 351 246 c8 Doing your hair or makeup 1374 343 194 c9 Talk to passengers 1516 378 235 total —— 15963 3985 2480

[0048] Table 2 Number of images after AUCv2 dataset division

[0049] category Category Name training set Validation set Test set c0 Safe driving 2019 504 183 c1 Texting with right hand 1044 261 133 c2 Right-hand call 690 172 114 c3 Texting with left hand 596 148 100 c4 Left-handed call 760 190 90 c5 Operate the radio 603 150 90 c6 Drinking water 587 146 63 c7 Reach behind 553 138 63 c8 Doing your hair or makeup 559 139 66 c9 Talk to passengers 1104 275 138 total —— 8515 2123 1040

[0050] S2. Select the network model ResNet18 with feature extraction capabilities, remove the last classification layer, and only retain the convolution layer to extract the feature information of the driver image.

[0051] S3. Construct the overall structure of the driver behavior recognition model based on the dual-branch self-attention mechanism, such as Figure 1 As shown in the figure, the output of ResNet18 is connected to the input of DBViT to construct two branches: a basic branch and an enhanced branch. The two branches are feature fused through DBViT, and a long-range dependency relationship between the features of the two branches is established to construct a global feature. Finally, the final driver behavior classification is completed through a linear layer (Linear). The driver behavior classification categories are: c0-safe driving, c1-texting with the right hand, c2-talking with the right hand, c3-texting with the left hand, c4-talking with the left hand, c5-operating the radio, c6-drinking water, c7-reaching to the back, c8-fixing hair or makeup, and c9-talking with passengers.

[0052] S4. Freeze the parameters of the enhancement branch, use the training set divided in step S1 to train the basic branch, and use the validation set to evaluate the training results.

[0053] S5. Freeze the parameters of the base branch and use the small sample dataset divided in step S1 to train the enhanced branch of DBViT.

[0054] S6. After two steps of training in steps S4 and S5, the base branch retains the parameters pre-trained based on the original dataset and focuses on extracting and constructing local feature representations of common driving behaviors. The enhancement branch is fine-tuned for new driver data to capture individual-specific behavior patterns. Finally, the driver behavior recognition model based on the dual-branch self-attention mechanism constructed in step S3 is classified through a linear layer.

[0055] like Figure 2 As shown in Figure 3, the DBViT architecture described in step S3 is: DBSSA and FFN are stacked together to form a Transformer-DB encoder block. FFN is the original Transformer architecture. The input of each Transformer-DB encoder block is processed by DBSSA to capture global information and calculate the mutual attention between the two branches. The input is then transformed by the FFN, and the output features are passed to the next layer.

[0056] DBSSA calculates the context scores of the two branches by attention calculation on the feature maps input by the two branches, then obtains the attention scores by element-by-element multiplication, and applies the attention scores to the backbone.

[0057] like Figure 3 As shown, DBSSA calculates the context scores c of the two branches by paying attention to the feature maps input by the two branches. VS and c VT , the calculation steps are:

[0058] (1) For the basic branch, the input x1 uses weights and W V1 ∈R d×d The linear layer is calculated to get and

[0059] Among them, W I1 represents the weight of the linear layer I1, W K1 represents the weight of the linear layer K1, W V1 represents the weight of the linear layer V1, x I1 represents the output of the linear layer I1, x K1 represents the output of the linear layer K1, x V1 Represents the output of the linear layer V1; Representative 2

[0060] dimensional space, size is d×d; Represents a one-dimensional space with a size of d; k×d represents the size of the feature map input to DBSSA.

[0061] Apply the Softmax function to x I1 To generate the context score of the base branch The context score c of the base branch S1 The calculation formula is:

[0062] c S1 =Softmax(x1W I1 );

[0063] Among them, Softmax represents the Softmax function operation.

[0064] The context score c of the base branch S1 and x K1 The context vector of the base branch is obtained by element-by-element multiplication and summation

[0065] The context vector c of the base branch VS The calculation formula is:

[0066] c VS =∑(c S1 *x1W K1 );

[0067] Among them, * represents element-by-element multiplication operation, ∑ represents summation operation, W K1 Represents the weight of the linear layer K1.

[0068] (2) For the enhanced branch, the input x2 uses weights and The linear layer is calculated to get and Among them, W I2 represents the weight of the linear layer I2, W K2 represents the weight of the linear layer K2, x I2 represents the output of the linear layer I2, x K2 Represents the output of the linear layer K2.

[0069] Apply the Softmax function to x I2 To generate the context score c of the enhanced branch S2 ∈R k , the context score c of the enhanced branch S2 The calculation formula is:

[0070] c S2 =Softmax(x2W I2 );

[0071] c S2 、x K2 and the trainable parameter vector The context vector of the enhanced branch is obtained by element-by-element multiplication and summation operations The context vector c of the boosted branch VT The calculation formula is:

[0072] c VT =∑(c S2 *x2W K2 *γ);

[0073] Source domain feature x V1 Input into the activation function ReLU, and then with c VS and c VT After element-by-element multiplication, the input weight is The linear layer gets the final output

[0074] The calculation formula of the two-branch separable self-attention is:

[0075] y=(c VS *c VT *ReLU(x1W V1 ))W Od ;

[0076] Among them, W v1 and W OD Represent the linear layer V1 and linear layer O respectively D The weight of .

[0077] In this embodiment, the base branch inherits the local feature extraction capabilities of the backbone network (i.e., the ResNet18 network model described in step S2) to generate high-resolution local feature maps. The enhancement branch constructs a lightweight feature extraction branch through multi-scale convolution to generate auxiliary feature maps. The base and enhancement branches use dynamic adaptive pooling to match the feature map sizes, and DBViT is used to capture long-range dependencies between the two and construct global features.

[0078] Test Example 1: On the AUCv2 dataset, we selected five representative state-of-the-art models to verify the effectiveness of DBViT: ResNet50, DenseNet121, MobileNet v3, MobileViT, and MobileViT v2. ResNet50, DenseNet121, and MobileNet v3 are CNN models, while MobileViT and MobileViT v2 are hybrid models combining CNN and Transformer. DBViT was embedded into each model to form the corresponding DBViT model. The test results are shown in Table 3.

[0079] Table 3 Experimental results of each model in AUCv2 dataset

[0080] Model Accuracy (baseline) Accuracy (fine-tuning) Operational Amount Parameter quantity ResNet50 52.02％ 81.92％ 5396.51M 23.52M ResNet50-DBViT 49.62％ 94.25％ 5815.5M 24.02M DenseNet121 55.77％ 84.13％ 3782.51M 6.96M DenseNet121-DBViT 51.83％ 94.44％ 4197.45M 7.4M MobileNet v3_small 33.65％ 72.98％ 79.95M 1.52M MobileNet v3-DBViT 59.33％ 93.33％ 486.77M 1.25M MobileViT_xx_small 56.73％ 84.13％ 354.29M 0.95M MobileViT-DBViT 58.77％ 95.06％ 539.17M 1.3M MobileViT v2_0.5 52.31％ 86.06％ 489.99M 1.11M MobileViT v2-DBViT 59.71％ 96.31％ 685.55M 1.49M

[0081] In the AUCv2 dataset, DBViT exhibits differentiated performance across different models: fine-tuned models embedded with DBViT generally outperform the original models (with an average improvement of approximately 12.83%). For example, MobileNet v3-DBViT's fine-tuning accuracy increases from 72.98% to 93.33%, and ResNet50-DBViT's accuracy increases from 81.92% to 94.25%. However, direct inference accuracy after pre-training shows significant divergence, with lightweight models showing significant improvements (MobileNet v3-DBViT's accuracy increases by 25.68% compared to the base model, and MobileViT v2-DBViT's accuracy increases by 7.4%). However, the baseline accuracy of ResNet50 and DenseNet121 decreases by 2.4% and 3.94%, respectively, suggesting that DBViT may interfere with the pre-trained features of traditional CNN architectures.

[0082] In terms of computational efficiency, similar to the SFD3 dataset, except for MobileNet v3, the computational load of other models generally increased by about 30%. However, the number of parameters did not change much (MobileNet parameters decreased by 17.8%, while the MobileViT series increased by about 37%).

[0083] In summary, in the AUCv2 dataset, embedding DBViT can still significantly improve fine-tuning accuracy at a slightly increased computational cost, increasing the model's reliability in real-world driving scenarios. However, we must be wary of the problem that lightweight models (such as MobileNet) may lose their original deployment advantages due to excessive computational overhead.

[0084] Test Example 2: To further verify the impact of embedding DBViT on the computational cost of the original model, we tested the real-time performance of the model embedded with DBViT. We performed continuous inference on 10,000 images on an Intel Core i5 13400 CPU and recorded the average inference time for a single image. The results are shown in Table 4.

[0085] Table 4 Average inference speed of each model using CPU

[0086] Model Average inference speed (ms) ResNet50 89.02 ResNet50-DBViT 105.13 DenseNet121 91.35 DenseNet121-DBViT 113.82 MobileNet v3_small 16.86 MobileNet v3-DBViT 30.38 MobileViT_xx_small 38.36 MobileViT-DBViT 47.16 MobileViT v2_0.5 42.53 MobileViT v2-DBViT 52.21

[0087] Table 4 shows that DBViT slows down CPU inference in all models, with the efficiency loss being particularly significant for lightweight models. MobileNet v3_small saw an 80.2% increase in latency, and the MobileViT series saw a 23-25% increase (e.g., MobileViT_xx_small increased from 38.36ms to 47.16ms). Traditional models like ResNet50 and DenseNet121 experienced relatively low latency increases.

[0088] Although DBViT slightly increases the computational cost of the model, it significantly improves the model's reliability in real-world scenarios by establishing long-range dependencies between the two branches of the model. In summary, DBViT can significantly improve real-world reliability at an acceptable computational cost, achieving a balance between computational effort and accuracy.

[0089] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A driver behavior recognition method based on a dual-branch self-attention mechanism is characterized by: The following steps are involved: S1. Using the public datasets SFD3 and AUCv2, the datasets are divided into training, validation, and test sets according to the driver. From the test set, one image is taken for each driver to construct a small sample dataset. The data is then preprocessed, including normalization, scaling, and random rotation. S2. Select the network model, remove the last classification layer, and only keep the convolution layer to extract the feature information of the driver image; S3. Construct the overall structure of the driver behavior recognition model based on the dual-branch self-attention mechanism. Connect the output of the network model in step S2 to the input of DBViT to construct two branches: a basic branch and an enhanced branch. The two branches are fused with features through DBViT, and a long-range dependency relationship between the features of the two branches is established to construct a global feature. Finally, the final driver behavior classification is completed through a linear layer. S4. Freeze the parameters of the enhancement branch, train the basic branch using the training set divided in step S1, and evaluate the training results using the validation set; S5. Freeze the parameters of the base branch and use the small sample data set divided in step S1 to train the enhanced branch; S6. After two steps of training in steps S4 and S5, the base branch retains the parameters pre-trained based on the original dataset and focuses on extracting and building local feature representations of general driving behaviors. The enhancement branch is fine-tuned for new driver data to capture individual-specific behavior patterns. Finally, the driver behavior recognition model based on the dual-branch self-attention mechanism constructed in step S3 is classified through a linear layer. The structure of DBViT described in step S3 is as follows: DBSSA and FFN are stacked together to form a Transformer-DB encoder block. The input of each Transformer-DB encoder block is processed by DBSSA to capture global information and calculate the mutual attention between the two branches. After that, the feature is transformed by FFN, and the output features are passed to the next layer. DBSSA calculates the context scores of the two branches by attention calculation on the feature maps input by the two branches, then obtains the attention scores by element-by-element multiplication, and applies the attention scores to the backbone.

2. The driver behavior recognition method based on the dual-branch self-attention mechanism according to claim 1 is characterized in that: In step S1, the data set is divided according to the driver division method: Test set: We manually selected behavioral images of three drivers from each dataset to form an independent test set. This ensured that the selected drivers' data were excluded from the initial model training. We also constructed a small sample dataset by taking one image from each driver. Training set and validation set: The remaining images in the dataset are divided into training set and validation set in a ratio of 8:

2.

3. The driver behavior recognition method based on the dual-branch self-attention mechanism according to claim 2 is characterized in that: In step S2, the network model is a model with feature extraction capability, including ResNet18 and MobileNet.

4. The driver behavior recognition method based on the dual-branch self-attention mechanism according to claim 3 is characterized in that: DBSSA calculates the context scores c of the two branches by paying attention to the feature maps input by the two branches. VS and c VT , the calculation steps are: For the basic branch, the input x1 uses weights and W V1 ∈R d×d The linear layer is calculated to get and Among them, W I1 represents the weight of the linear layer I1, W K1 represents the weight of the linear layer K1, W V1 represents the weight of the linear layer V1, x I1 represents the output of the linear layer I1, x K1 represents the output of the linear layer K1, x V1 Represents the output of the linear layer V1; Represents a two-dimensional space with a size of d×d; Represents a one-dimensional space with a size of d; k×d represents the size of the feature map of the input DBSSA; Apply the Softmax function to x I1 To generate the context score of the base branch The context score c of the base branch S1 The calculation formula is: c S1 =Softmax(x1W I1 ); Among them, Softmax represents the Softmax function operation; The context score c of the base branch S1 and x K1 The context vector of the base branch is obtained by element-by-element multiplication and summation The context vector c of the base branch VS The calculation formula is: c VS ∑(c S1 *x1W K1 ): Among them, * represents element-by-element multiplication operation, ∑ represents summation operation, W K1 Represents the weight of the linear layer K1; For the enhancement branch, the input x2 uses weights and The linear layer is calculated to get and Among them, W I2 represents the weight of the linear layer I2, W K2 represents the weight of the linear layer K2, x I2 represents the output of the linear layer I2, x K2 Represents the output of the linear layer K2; Apply the Softmax function to x I2 To generate the context score c of the enhanced branch S2 ∈R k , the context score c of the enhanced branch S2 The calculation formula is: c S2 =Softmax(x2W I2 ); c S2 、x K2 and the trainable parameter vector The context vector of the enhanced branch is obtained by element-by-element multiplication and summation operations The context vector c of the boosted branch VT The calculation formula is: c VT =∑(c S2 *x2W K2 *c); Source domain feature x V1 Input into the activation function ReLU, and then with c VS and c VT After element-by-element multiplication, the input weight is The linear layer gets the final output The calculation formula of the two-branch separable self-attention is: y=(c VS *c VT *ReLU(x1W V1 ))W OD 4 Among them, W V1 and W OD Represent the linear layer V1 and linear layer O respectively D The weight of .

5. The driver behavior recognition method based on the dual-branch self-attention mechanism according to claim 4 is characterized in that: The basic branch inherits the local feature extraction capability of the backbone network and generates high-resolution local feature maps; the enhanced branch constructs a lightweight feature extraction branch through multi-scale convolution to generate auxiliary feature maps; The base branch and the enhanced branch match the feature map sizes through dynamic adaptive pooling, and use DBViT to capture the long-range dependencies between the two and construct global features.

Images