Driver behavior recognition method based on double-branch parallel bidirectional interaction neural network

Abstract

The application discloses a driver behavior recognition method based on a double-branch parallel bidirectional interaction neural network, introduces ViT into the network to encode global information, and improves detection accuracy to a certain extent; the network is composed of two parallel branches; the first branch is based on a lightweight CNN structure, and the second branch is based on a ViT structure. Considering the asymmetry between the CNNBranch and the ViTBranch features, a bidirectional feature interaction module BiFIM is used to interact different dimension features between the two branches to simultaneously focus on local features and global features, and finally, the features of the two branches are fused and the driver behavior is detected. The application is carried out on a self-built multi-view driver dataset, the verification set accuracy reaches 97.18%, the parameter quantity is 38.22MB, and the MAdds is 271.20M; the lightweight BiViTNet improves the accuracy of driver distraction behavior recognition, can assist the driver to drive safely, and reduces the occurrence of traffic accidents.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing detection, and particularly relates to a driver behavior recognition method based on a bidirectional interaction neural network based on vision transformer (BiViTNet). BACKGROUND

[0002] Distraction driving is the primary cause of traffic accidents, and it is particularly important to detect and identify the behavior of drivers during driving. The real-time and accuracy of identification are one of the elements of auxiliary driving safety, and correct identification can better assist drivers to drive safely and reduce traffic accidents to a certain extent. Therefore, driver distraction behavior detection has broad research prospects in the fields of computer vision and autonomous driving.

[0003] Although driver distraction behavior recognition is a general problem of human action recognition, due to the small and complex environment in the car and the subtle differences in the posture of the driver when distracted, the current methods for human action recognition are not suitable for driver distraction recognition.

[0004] Driver distraction behavior detection has received extensive attention, and researchers have proposed a variety of methods in the past few years. According to the different data collection methods, it can be divided into contact sensor recognition method and non-contact sensor recognition method. The contact sensor recognition method requires the driver to wear special equipment to collect the physiological information and limb movement state of the driver, which may interfere with the normal driving behavior of the driver. The non-contact sensor recognition method mainly collects data through cameras, pressure sensors, etc. to analyze the driving state of the driver, which has good deployment convenience. The application adopts the non-contact sensor recognition method, which collects driving data through a monocular camera.

[0005] According to the algorithm adopted, it can be divided into traditional machine learning based method and deep learning based method. In previous studies, many researchers detected the driver distraction behavior through traditional machine learning method, Yan et al. detected the driver distraction behavior through pyramid direction gradient histogram and random forest classifier; Craye et al. detected the driver distraction behavior through hidden Markov model and Adaboost classifier; Almahasneh et al. detected the driver distraction behavior through singular value boundary method and classifier. However, the traditional machine learning method usually needs to design an artificial feature extraction algorithm, such as gradient histogram, texture edge feature, etc. Since such artificial features cannot adapt to a large number of scenes, the traditional machine learning often has good performance in a single scene, but it is difficult to migrate to more scenes.

[0006] In recent years, due to the great success of convolutional neural network in various fields, many researchers also use it in the driver distraction behavior detection task, Olabiyi et al. combined camera images and vehicle motion information, and used a bidirectional recurrent neural network to realize driver behavior prediction; Peng et al. first extracted the semantic information of a single moment from the collected driving data through VGG-19 network, and then detected the driver distraction behavior through LSTM (Long Short Term Memory) network; Li proposed a lightweight network OLCMNet combining Depth-wise and Point-wise convolution to detect the driver distraction behavior.

[0007] Although the above methods can basically detect the driver distraction behavior, there are still some problems: 1) the data set adopted is generally from the experimental condition, which has certain error compared with the data in real driving scene, so the data should be as close to the real driving environment as possible; 2) the parameter quantity and calculation quantity of most networks are large, which is difficult to meet the real-time detection requirement; 3) the detection accuracy is not high, which is a common problem. Therefore, the present application proposes a lightweight neural network BiViTNet which can accurately identify various driver distraction behaviors. SUMMARY

[0008] The present application provides a driver behavior recognition method based on a double-branch parallel bidirectional interaction neural network, which realizes the interaction of local and global visual representation information, quickly and accurately detects the driver distraction behavior, assists the driver to drive safely, and reduces the occurrence of traffic accidents.

[0009] Technical scheme: The present application provides a driver behavior recognition method based on a double-branch parallel bidirectional interaction neural network, which includes the following steps:

[0010] (1) Make a data set and preprocess the data, including normalization and scaling;

[0011] (2) Construct a BiViTNet network, which adopts a double-branch parallel structure, the first branch adopts a convolution structure, focusing on local features, including a backbone module, a residual transition module and a local perception module LSM; the second branch adopts a ViT structure to encode global information; and the different dimension features are interacted between the two branches through a bidirectional feature interaction module BiFIM;

[0012] (3) The preprocessed training image is sent into the BiViTNet network model for training;

[0013] (4) The preprocessed test image is sent into the trained BiViTNet network model for testing, and the performance of the model is evaluated.

[0014] Further, the backbone module in step (2) adopts 3x3 convolution combined with maximum pooling, which scales the image size to 1 / 4 of the original size and expands the channel number from 3 to 64.

[0015] Further, the residual transition module in step (2) adopts depth separable convolution, including channel-wise convolution and point-wise convolution;

[0016] The parameter quantity P and the calculation quantity F of the standard convolution are as follows: s s

[0017] P s =K 2 ×M×N (1)

[0018] F s =H out ×W out ×M×K 2 ×N (2)

[0019] The parameter quantity P and the calculation quantity F of the depth separable convolution are as follows: DP DP

[0020] P DP =K 2 ×M+M×N (3)

[0021] F DP =K 2 ×H out ×W out ×M+H out ×W out ×M×N (4) ​​​​

[0022] Therefore, the ratio P of the parameter amount and the ratio F of the calculation amount of the standard convolution and the depth separable convolution are as follows:

[0023]

[0024]

[0025] wherein H out and W out are the height and width of the output feature map respectively, M is the channel number of the input feature map, K is the size of the convolution kernel, and N is the number of the convolution kernel and is also the channel number of the output feature map; it can be seen from equations (5) and (6) that the parameter amount and the calculation amount of the depth separable convolution are significantly reduced compared with the standard convolution.

[0026] Further, the local perception module in step (1) first uses a 1x1 convolution to map the feature map after the maximum pooling from a low-dimensional space to a high-dimensional space to extract more information; in order to obtain local features while reducing the parameter amount and the calculation amount, the LSM also uses a depth separable convolution, and a residual structure is also used to prevent the problem of gradient explosion caused by a too deep network.

[0027] Further, the ViT structure in step (2) contains 3 ViT blocks, and different receptive field range feature information is obtained by controlling the image block size input at each stage and the patch size divided, the last of each ViT block does not do classification, but is connected in series to the next ViT block through a bidirectional feature interaction module BiFIM, and different levels of semantic information are obtained through the combination of BiFIM module and ViT; finally, the ViT Branch and the CNN Branch are fused and then classified to maximize the preservation of local features and global representation, and to realize the interaction of local and global visual representation information.

[0028] Further, the bidirectional feature interaction module BiFIM in step (2) includes a feature alignment module FAM and a sampling module; the FAM module is used to convert between two feature dimensions and match the channels through a 1x1 convolution; at the same time, up-sampling and down-sampling are performed to facilitate the fusion of the features of the CNN Branch and the ViT Branch; the BiFIM performs feature dimension conversion on the one hand to realize the series connection of the ViT blocks, and on the other hand establishes bidirectional feature interaction between the CNN Branch and the ViT Branch.

[0029] Beneficial effects: compared with the prior art, the beneficial effects of the present application: 1, the present application proposes a double-branch parallel lightweight neural network, which combines local features based on CNN and global features based on ViT to enhance representation learning; 2, the double-branch parallel structure is adopted, and CNN Branch and ViT Branch can maximize the preservation of local features and global representation; 3, in order to make the network more suitable for actual, the data set of the present application comes from the multi-view monitoring video in the vehicle of the traffic supervision platform, on the one hand, this is the real driving environment, on the other hand, multi-view monitoring is more comprehensive compared with the data collected by a single camera. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The flowchart of the present application is shown in the figure;

[0031] Figure 2 The structure diagram of Stem module is shown in the figure;

[0032] Figure 3 The structure diagram of Transition module is shown in the figure;

[0033] Figure 4 The structure diagram of LSM is shown in the figure;

[0034] Figure 5 The structure diagram of ViT is shown in the figure;

[0035] Figure 6 The structure diagram of ViT Branch is shown in the figure;

[0036] Figure 7 The structure diagram of BiFIM is shown in the figure;

[0037] Figure 8 The data set diagram is shown in the figure;

[0038] Figure 9 The multi-view monitoring diagram is shown in the figure;

[0039] Figure 10 The data change diagram in training is shown in the figure. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below in combination with the drawings.

[0041] Some variables are involved in the present embodiment, and each variable will be described as follows, as shown in Table 1.

[0042] Table 1 variable description table

[0043]

[0044]

[0045] The application provides a driver behavior recognition method based on a double-branch parallel bidirectional interaction neural network, and the specific implementation process is as follows: first, a data set is made, and the data is preprocessed, including normalization and scaling. The data set made is divided into a training set and a test set. A BiViTNet (Bidirectional Interaction Neural Network Based on Vision Transformer) network model is constructed, and the preprocessed training set is sent into the BiViTNet network model for training; the preprocessed test set is sent into the trained BiViTNet network model for testing, and the performance of the model is evaluated.

[0046] The overall structure of the BiViTNet proposed by the application is shown in Figure 1 The BiViTNet network adopts a double-branch parallel structure, the first branch adopts a convolution structure (CNN Branch), focuses on local features, and includes a stem module (Stem), a residual transition module (Transition) and a local sensing module (LSM (Local Sensing Module)); the second branch adopts a ViT structure (ViT Branch), the ViT Branch is stacked by a ViT (Vision Transformer), and encodes global information; and the bidirectional feature interaction module (BiFIM (Bidirectional Feature Interaction Module)) is used to interact different dimensional features between the two branches. Finally, the features of the two branches are fused, and when the features are fused, an add function is used to increase the amount of feature information in each dimension, which is helpful for the final classification.

[0047] A typical feature of CNN is that the obtained features have locality, which is an inductive bias based on the assumption that adjacent pixels have greater correlation; and the Transformer is based on the interaction of global information, so there is a certain degree of difference between the CNN and the Transformer in learning methods and feature properties. Therefore, the combination of the CNN and the Transformer can integrate local and global visual representation information, which is helpful to improve the learning and expression ability of the whole network to features.

[0048] The CNN Branch is based on a feature pyramid architecture. First, the input is processed by a Stem module, which is mainly used to extract the initial local features of the image. Since the CNN Branch focuses on local features of the image, it does not use large convolution kernels to expand the receptive field as other networks do, which reduces the parameter quantity to some extent. Therefore, the Stem module uses 3x3 convolution combined with max pooling to scale the image size to 1 / 4 of the original size, and the channel number is expanded from 3 to 64 in order to make up for the information loss in this process. The feature map after the Stem module is S. The structure of the Stem module is shown in FIG. 8. Figure 2

[0049] After the image is processed by the Stem module, it enters the Transition module. In order to ensure real-time performance and keep the parameter quantity and computational cost as low as possible, the Transition module uses depthwise separable convolution, including depthwise convolution and pointwise convolution. When performing convolution in the depthwise direction, only one channel of the feature map is convolved by one convolution kernel. However, the number of channels of the feature map before and after depthwise convolution is the same, and it cannot be expanded in the channel dimension. Moreover, this operation is independent of each channel of the input, and cannot effectively utilize the information of different channels at the same spatial position. Therefore, pointwise convolution is needed to integrate the features output by depthwise convolution. On the one hand, pointwise convolution can freely change the number of output channels, and on the other hand, it can also fuse the features output by depthwise convolution in the channel dimension.

[0050] The parameter quantity (P s ) and computational cost (FLOPs, F s ) of standard convolution are as follows:

[0051] P s =K 2 ×M×N (1)

[0052] F s =H out ×W out ×M×K 2 ×N (2)

[0053] The parameter quantity (P DP ) and computational cost (FLOPs, F DP ) of depthwise separable convolution are as follows:

[0054] P DP =K 2 ×M+M×N (3)​

[0055] F DP = K 2 × H out × W out × M + H out × W out × M × N (4)

[0056] Therefore, the ratio of the parameter amount (P) and the ratio of the calculation amount (F) of the standard convolution and the depth separable convolution are as follows:

[0057]

[0058]

[0059] where H out and W out are the height and width of the output feature map respectively, N is the number of channels of the input feature map, K is the size of the convolution kernel, and N is the number of convolution kernels, which is also the number of channels of the output feature map. It can be seen from equations (5) and (6) that the parameter amount and the calculation amount of the depth separable convolution are significantly reduced compared with the standard convolution.

[0060] The Transition module also adopts a residual structure, which can ensure that long-distance information is not lost, and also enables feature interaction between different levels. The feature map S first passes through a Pointwise convolution with a step of 1 and a convolution kernel size of 1x1, then is normalized by the BatchNorm2d function, and then uses the SiLU activation function to increase the nonlinearity factor; then, it passes through a Depthwise convolution with a step of 1, padding of 1, and a convolution kernel size of 3x3, then is normalized by the BatchNorm2d function, and then uses the SiLU activation function to increase the nonlinearity factor; then, it passes through a Pointwise convolution with a step of 1 and a convolution kernel size of 1x1, then is normalized by the BatchNorm2d function to obtain the feature map T1; finally, S and T1 are fused by the add method to obtain T, and in this process, the number of channels of the feature map is kept at 64. The structure of the Transition module is shown in Figure 3 .

[0061] The local sensing module LSM in the CNN Branch is mainly used to efficiently and cheaply extract local features of an image. The feature map T is first subjected to a Pointwise convolution with a step size of 1 and a convolution kernel size of 1x1, and then subjected to a BatchNorm2d function for normalization to obtain L1; then, the feature map T is subjected to a Depthwise convolution with a step size of 1, padding of 1 and a convolution kernel size of 3x3, and then subjected to a BatchNorm2d function for normalization, and then subjected to a SiLU activation function to increase the nonlinearity factor; then, the feature map T is subjected to a Pointwise convolution with a step size of 1 and a convolution kernel size of 1x1, and then subjected to a BatchNorm2d function for normalization to obtain L2; finally, L1 and L2 are fused by an add operation to obtain L.

[0062] The structure of the LSM is shown in Figure 4 In the LSM, no convolution kernel larger than 3x3 is used, and no wider network structure is used, but a simple single-branch structure is used, because the CNN Branch is only used to obtain local features of an image, and only 1x1 convolution and 3x3 convolution are used to complete this task, so there is no need to expand the receptive field range by using a larger convolution kernel or a wider network structure. The global information that cannot be captured by the CNN Branch is compensated by the ViT Branch and the bidirectional feature interaction module.

[0063] Because the CNN Branch uses 1x1 and 3x3 convolution kernels to reduce the number of parameters, the receptive field is small, so it can only focus on local information, so the ViT Branch needs to establish global dependence to compensate for the shortcomings of the CNN Branch.

[0064] The structure of the ViT block is shown in Figure 5 The ViT divides the picture into small blocks, and then converts the small blocks into embeddings as the input of the Transformer, and the processing method is the same as the token in NLP. The key of the ViT is the Transformer block, which is more global dependence than the CNN structure.

[0065] As shown in Figure 6As shown, the ViT Branch includes 3 ViT blocks, and different receptive field range feature information is obtained by controlling the image block size and the divided patch size of each stage input, and the last of each ViT block does not perform classification, but is connected to the next ViT block through the bidirectional feature interaction module BiFIM, and different levels of semantic information are obtained through the combination of BiFIM module and ViT. Finally, the ViT Branch and the CNN Branch are fused and then classified, which can maximize the retention of local features and global representation, and realize the interaction of local and global visual representation information, and better detect the driver distraction behavior.

[0066] The BiFIM (Bidirectional Feature Interaction Module) is used for matching different features of the two branches between the CNN Branch and the ViT Branch, realizing the bidirectional fusion between the local features and the global representation. The BiFIM includes two parts, namely the feature alignment module FAM (Feature Alignment Module) and the sampling module. Since the feature dimensions after the transformation and the feature dimensions after the convolution are not matched, the FAM module is used to convert the two kinds of feature dimensions and realize the channel matching through 1x1 convolution; meanwhile, up-sampling and down-sampling are also needed to facilitate the fusion of the features of the CNN Branch and the ViT Branch. Therefore, the BiFIM realizes the connection of the ViT blocks on the one hand by converting the feature dimensions, and on the other hand, establishes the bidirectional feature interaction between the CNN Branch and the ViT Branch. The structure of the BiFIM is as shown in Figure 7 .

[0067] The data used in the present application is from the multi-view monitoring video in the vehicle provided by the traffic supervision platform, and there are a total of 780 videos. The production of the data set first cuts out the short video with distraction behavior in the original monitoring video, then cuts the short video into pictures, and finally selects the representative images as the data set. The data set used in the present application is divided into five categories, and a total of 31694 pictures are included, wherein the training set and the verification set are divided according to the ratio of 7:3. Figure 8 The schematic diagram of the driver distraction behavior data set.

[0068] Although the multi-view monitoring improves the richness of the data and is more comprehensive than the data collected by a single camera, it also brings great challenges to the detection. Taking the multi-view monitoring of C2 making a phone call as an example, Figure 9 .

[0069] In order to accelerate the training process of the network, the data set is preprocessed. First, the size of the image is uniformly converted to 224x224, and then data augmentation is performed by using scaling, cropping, flipping and increasing / decreasing contrast.

[0070] In order to evaluate the performance of BiViTNet, the present application uses the following four indicators to evaluate the network, which are accuracy (Accuracy, A), precision (Precision, P), recall (Recall, R) and F1 score (F1 Score, F1). Among them, the accuracy represents the ratio of the number of correctly predicted samples to the total number of samples, and the higher the accuracy, the better the classifier; the precision is the proportion of correctly predicted samples in the predicted positive samples; the recall is the proportion of correctly predicted positive samples in the total actual positive samples. The definitions of the four evaluation indicators are as follows:

[0071]

[0072]

[0073]

[0074]

[0075] Among them, TP (true positive) is the number of correctly classified positive samples; TN (true negative) is the number of correctly classified negative samples; FP (false positive) is the number of negative samples that are incorrectly classified as positive samples; FN (false negative) is the number of positive samples that are incorrectly classified as negative samples. The definitions of TP, TN, FP and FN are shown in Table 2.

[0076] Table 2 Confusion matrix

[0077]

[0078] The deep learning framework used by the present application is pytorch 1.10, the operating system is Centos 8.0, the GPU is TeslaV100, and the memory size is 16G. The momentum of the model is set to 0.9, the training batch size is 16, the training round is 64, the initial learning rate is 0.001, the learning rate is changed to 0.5 every 10 rounds, and the random gradient descent method (SGD) is used to train the network.

[0079] The accuracy (acc) and loss of the training set and validation set of the BiViTNet network during the training process are as follows: Figure 10The figure shows that the network begins to converge gradually after 50 epochs of training, and the training loss is gradually decreasing. Neither the accuracy nor the loss is overfitting during the training process. In addition, the curve does not show noise during the training process. The overall performance is smooth during the entire rising and falling process, except for the first 3-4 epochs. This also shows that the BiViTNet model built by the application has good robustness and is more suitable for driver distraction detection tasks.

[0080] To show the effect of BiViTNet in more detail, here, the performance of C0 normal driving, C1 looking around, C2 talking on the phone, C3 playing mobile phones, and C4 smoking in the four evaluation indexes of Accuracy, Precision, Recall and F1 Score is analyzed in detail. The data of each index is shown in Table 3.

[0081] Table 3 verifies the evaluation indexes of BiViTNet

[0082] Class Accuracy Precision Recall F1 Score C0 96.87 95.65 95.80 95.73 C1 92.83 91.83 90.61 90.70 C2 99.62 99.34 99.58 99.46 C3 98.73 97.94 97.72 97.83 C4 97.85 97.55 98.37 97.96 Average 97.18 96.46 96.42 96.34

[0083] From Table 3, it can be seen that the average accuracy, average precision, average recall and F1 Score of the method are 97.18%, 96.46%, 96.42% and 96.34% respectively. Among them, the C2 class representing talking on the phone has the highest accuracy, precision, average recall and F1 Score. However, the accuracy of C0 representing normal driving and C1 representing looking around is lower than that of other classes. Especially for C1 class, the accuracy is only 92.83%, and the recall is only 90.61%. This may be due to the small difference in body posture between looking around and normal driving, only a slight twist of the head and almost no difference in other parts of the body. Most of them can be distinguished through the changes of the face, which is very easy to be confused with normal driving, so the model is difficult to distinguish between normal driving and looking around.

[0084] On the test set of the self-built multi-view driver dataset, the same training strategy is adopted to compare the application with some common network models and methods. The experimental results of BiViTNet network compared with classic CNN network and classic ViT network are shown in Table 4 and Table 5 respectively.

[0085] Table 4 comparison with CNN series network

[0086] Model name Image size Model size (MB) Parameters (MB) M Adds Accuracy (%) VGG 224 2 ]] 754.67 537.24 15.48G 90.21 ResNet-50 224 2 ]] 264.07 94.07 3.86G 90.85 DenseNet 224 2 ]] 208.97 27.84 2.83G 92.55 MobileNetV2 256 2 ]] 149.27 8.92 391.22M 95.47 CNN Branch 224 2 ]] 21.48 0.90 249.94M 96.78 BiViTNet 224 2 ]] 208.82 38.22 271.20M 97.18

[0087] Table 5 comparison with ViT series network

[0088] Model name Image size Model size (MB) Parameters (MB) M Adds (M) Accuracy (%) ViT 224 2 ]] 504.07 342.49 85.62 95.39 CrossViT 256 2 ]]> 319.21 218.09 54.52 95.76 ViT Branch 224 2 ]] 187.43 37.29 17.19 96.74 BiViTNet 224 2 ]] 208.82 38.22 271.20 97.18

[0089] As can be seen from Table 4, compared with the CNN series network, the BiViTNet model proposed in the application has certain advantages. Compared with the MobileNetV2 model with the highest performance in the CNN series network, the CNN Branch significantly reduces the parameter quantity and model size. The accuracy of the method proposed in the application reaches 97.18%, and the calculation amount is obviously reduced compared with the VGG, ResNet-50 and DenseNet models. The MAdds (Multiply-Accumulate Operations) is even reduced by an order of magnitude.

[0090] As can be seen from Table 5, compared with the classic ViT model and the CrossViT model, the BiViTNet model proposed in the application exhibits better experimental results. The verification accuracy of the ViT Branch reaches 96.74%, the verification accuracy of the whole model is higher, and the model size and parameter quantity are obviously reduced, and the comprehensive performance is better.

[0091] The experimental results of the two branches are compared as shown in Table 6.

[0092] Table 6 Comparison of two branches

[0093] Branch name Params (M) M Adds (M) Accuracy (%) ViT Branch 37.29 17.19 96.74 CNN Branch 0.90 249.94 96.78 BiViTNet 38.22 271.20 97.18

[0094] As can be seen from Table 6, the two branches proposed in the application have high accuracy, but since the experiment is carried out without any pre-training, and the size of the data set is not particularly large, the accuracy of the ViT Branch is slightly inferior to that of the CNN Branch. The ViT Branch pays more attention to global representation, and the accuracy is 96.74%. The CNN Branch pays more attention to local features, and the accuracy is 96.78%. Since the BiFIM can enhance the global perception ability of local features and the local details of global representation, after the connection between the two branches is established through the BiFIM, the accuracy of the model is higher than that of any single branch. The BiViTNet model proposed in the application achieves good experimental results, and the accuracy reaches 97.18%.

Claims

1. A driver behavior recognition method based on a dual-branch parallel bidirectional interactive neural network, characterized in that, Includes the following steps: (1) Create a dataset and preprocess the data, including normalization and scaling; (2) Construct a BiViTNet network. The BiViTNet network adopts a dual-branch parallel structure. The first branch adopts a convolutional structure and focuses on local features, including the backbone module, residual transition module and local perception module LSM. The second branch adopts a ViT structure to encode global information. The two branches interact with features of different dimensions through the bidirectional feature interaction module BiFIM. (3) The preprocessed training images are fed into the BiViTNet network model for training; (4) The preprocessed test images are fed into the trained BiViTNet network model for testing, and the performance of the model is evaluated. The residual transition module described in step (2) employs depthwise separable convolution, including channel-wise convolution and pointwise convolution; The number of parameters in standard convolution and computational complexity as follows: ； The number of parameters of depthwise separable convolution and computational complexity as follows: ； Therefore, the ratio of the number of parameters P to the computational cost F between standard convolution and depthwise separable convolution are as follows: ； in, and These are the height and width of the output feature map, respectively. It is the number of channels in the input feature map. It is the kernel size. The number of convolution kernels is also the number of channels in the output feature map; it can be seen from equations (5) and (6) that depthwise separable convolution has significantly reduced parameters and computational cost compared to standard convolution; The local sensing module described in step (1) first uses Convolution maps the feature map after max pooling from a low-dimensional space to a high-dimensional space to extract more information. In order to reduce the number of parameters and computation while obtaining local features, LSM also uses depthwise separable convolution and uses residual structure to prevent the gradient explosion problem caused by the network being too deep. Step (2) The Bi-directional Feature Interaction Module (BiFIM) includes a feature alignment module (FAM) and a sampling module. The FAM module converts between two feature dimensions and performs channel matching through 1×1 convolution. At the same time, upsampling and downsampling are performed to facilitate the fusion of features of the CNNBranch and ViT Branch. On the one hand, BiFIM performs feature dimension conversion to realize the concatenation of ViT blocks, and on the other hand, it establishes bi-directional feature interaction between the CNN Branch and the ViT Branch.

2. The driver behavior recognition method based on a dual-branch parallel bidirectional interactive neural network according to claim 1, characterized in that, The backbone modules mentioned in step (2) all adopt The convolutional operation, combined with max pooling, scales the image size to 1 / 4 of its original size and increases the number of channels from 3 to 64.

3. The driver behavior recognition method based on a dual-branch parallel bidirectional interactive neural network according to claim 1, characterized in that, The ViT structure described in step (2) consists of three ViT blocks. By controlling the size of the input image blocks and the size of the divided patches at each stage, feature information of different receptive field ranges is obtained. Each ViT block is not classified at the end, but is connected to the next ViT block in series through the bidirectional feature interaction module BiFIM. The semantic information of different levels is obtained by combining the BiFIM module with ViT. Finally, the ViT Branch and CNN Branch are fused for feature fusion before classification, so as to retain local features and global representations to the maximum extent and realize the interaction between local and global visual representation information.

Images