Driver behavior recognition method based on multi-trunk adaptive pruning and gate fusion network

Abstract

The present application belongs to the field of intelligent transportation and intelligent driving, and specifically relates to a driver behavior recognition method based on a multi-trunk adaptive pruning and gating fusion network. The method comprises: constructing a multi-scene driver behavior dataset; establishing a multi-trunk feature extraction network composed of ConvNeXt-t, ResNeXt50 and ResNet50; designing an adaptive pruning method based on the differences in the trunk structure to structurally compress different trunk networks; constructing a gating multi-trunk feature fusion module to dynamically allocate weights to different trunk features; adopting a progressive fine-tuning strategy to realize phased optimization training of the network; and finally outputting the driver behavior recognition result through a classifier. The present application effectively reduces the model complexity while ensuring the recognition accuracy, improves the generalization ability and real-time inference performance in complex driving scenarios, and achieves an identification accuracy of 92.61% and a real-time detection speed of 27.5 FPS on the SAA13 dataset, and can be applied to intelligent transportation, vehicle driving monitoring and advanced driving assistance systems.

Description

Technical Field

[0001] This invention belongs to the research fields of intelligent transportation, intelligent driving and computer vision, and specifically relates to a driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network. Background Technology

[0002] With the rapid development of intelligent transportation systems and advanced driver assistance systems (ADAS), driver behavior recognition technology has gradually become an important research direction for ensuring road traffic safety. A large number of traffic accidents are closely related to driver distraction, such as making phone calls, sending text messages, talking to passengers, and operating the center console, all of which significantly reduce a driver's perception of the road environment. Therefore, achieving high-precision, real-time driver behavior recognition in complex driving environments is of great significance for reducing traffic accident rates and improving the safety of intelligent driving systems.

[0003] Most existing driver behavior recognition methods are based on deep learning models. While these methods can improve recognition accuracy to some extent, they still suffer from the following problems: First, a single backbone network struggles to simultaneously capture both local detail features and global semantic features, resulting in insufficient model generalization ability in complex driving scenarios. Second, multi-backbone fusion models typically introduce a large number of parameters and redundant computations, making it difficult to meet the real-time requirements of in-vehicle edge devices. Furthermore, traditional feature fusion methods often employ simple concatenation or weighting, lacking the ability to dynamically model the differences in contribution from different backbones, easily leading to information redundancy and insufficient feature utilization. Therefore, designing a driver behavior recognition method that balances high accuracy, lightweight design, and stable training capabilities is of significant research importance. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of the prior art, this invention proposes a driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network, which can realize high-precision real-time recognition of driver behavior in complex driving scenarios and reduce model complexity and computational overhead.

[0005] Technical Solution: To achieve the above objectives, this invention provides a driver behavior recognition method based on a multi-trunk adaptive pruning and gating fusion network, comprising the following steps:

[0006] S1: Construct an image dataset of driver behavior containing complex driving scenarios;

[0007] S2: Construct a multi-backbone driver behavior recognition network consisting of ConvNeXt-t, ResNeXt50, and ResNet50;

[0008] S3: Research on the backbone-awaited adaptive pruning (BAAP) method based on differences in backbone structure.

[0009] S4: Perform structured compression on different backbone networks to generate a lightweight backbone feature extraction network;

[0010] S5: Research on the gated multi-backbone feature fusion mechanism G-MBFF.

[0011] S6: Map the deep features extracted by different backbone networks to a unified feature space and perform dynamic weighted fusion;

[0012] S7: Researching a progressive fine-tuning strategy (PFS) for stabilizing network training.

[0013] S8: The driver behavior prediction results are output through a classifier, thereby constructing a complete MBAPF (multi-backbone adaptive pruning and fusion) driver behavior recognition framework. Further, the specific method for constructing the standard image dataset of driver behavior in step S1 is as follows:

[0014] Furthermore, the specific method for constructing the driver behavior image dataset in step S1 is as follows:

[0015] S1-1: Integrate five publicly available driver behavior datasets—State Farm, ASU, AUCV2, 100-Driver, and 3MDAD—and construct the SAA13 comprehensive driver behavior dataset;

[0016] S1-2: An interval sampling strategy is used for the original video frames to reduce redundant information between adjacent frames;

[0017] S1-3: The dataset covers ten categories of driving behaviors, including safe driving, texting with the left hand, texting with the right hand, making a phone call with the left hand and making a phone call with the right hand, operating the center console, drinking water or eating, leaning back, fixing hair or applying makeup, and talking to passengers.

[0018] S1-4: The driver is divided into training, validation and test sets respectively to improve the generalization ability of the model.

[0019] Furthermore, the specific method for constructing the multi-backbone driver behavior recognition network in step S2 is as follows: three residual structure networks, ConvNeXt-t, ResNeXt50 and ResNet50, are used as sub-backbone networks, and ImageNet pre-trained weights are used to initialize the parameters of each backbone, thereby realizing the joint extraction of different structural features.

[0020] Furthermore, the specific method for studying the backbone-aware adaptive pruning method BAAP in step S3 is as follows: The contribution value of convolutional channels is calculated using an importance evaluation method based on the L1 norm of the convolutional channels, and corresponding pruning strategies are designed according to different backbone structure characteristics; ConvNeXt-t mainly prunes point convolutional layers, ResNeXt50 uses a group-level pruning method, and ResNet50 prunes the intermediate channels of the bottleneck layer; low-contribution channels are removed according to a preset pruning rate, thereby reducing the number of network parameters and computational complexity.

[0021] Furthermore, the specific content of step S4 is as follows: different pruning rates are set for ConvNeXt-t, ResNeXt50 and ResNet50 respectively, wherein the pruning rate of ConvNeXt-t is 0.20, the pruning rate of ResNeXt50 is 0.50 and the pruning rate of ResNet50 is 0.40; the pruned network effectively reduces the model parameter scale while maintaining the recognition ability.

[0022] Furthermore, the specific content of studying the gated multi-backbone feature fusion mechanism G-MBFF in step S5 is as follows: Deep features f from the outputs of the ConvNeXt-t, ResNeXt50, and ResNet50 backbone networks are extracted respectively. c f x with f r The system projects features of different dimensions onto a unified feature space using a linear mapping; then, the features are concatenated and input into a gating module, where a sigmoid activation function is used to generate corresponding dynamic weights g. c g x With g r Finally, dynamic weights are used to weight and fuse different backbone features to form the final fused feature.

[0023] The gating module adopts a Linear-ReLU-Linear structure and outputs dynamic weights corresponding to each backbone through the Sigmoid activation function, thereby realizing adaptive feature allocation under different driving scenarios.

[0024] Furthermore, the specific content of the progressive fine-tuning strategy PFS in step S7 is as follows: In the first stage, all backbone networks are frozen, and only the classifier is trained; in the second stage, the last convolutional layers of each backbone network are unfrozen and the classifier is jointly optimized; in the third stage, the entire network is unfrozen and overall fine-tuning is performed, thereby achieving stable staged training.

[0025] Furthermore, the specific process of step S8 is as follows: after the fused features are processed by a fully connected classifier, the probability values ​​corresponding to each category are output, and a normalized probability distribution is generated using the Softmax activation function, wherein the category corresponding to the highest probability is taken as the final driver behavior recognition result.

[0026] The MBAPF framework of this invention achieves a recognition accuracy of 92.61% and a real-time inference speed of 27.5 FPS on the SAA13 comprehensive driver behavior dataset. Compared with the traditional multi-backbone fusion model, it effectively reduces the number of model parameters and improves the real-time deployment capability.

[0027] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: (1) The BAAP method proposed in the present invention can implement differentiated pruning according to the characteristics of different backbone structures, and maintain high recognition accuracy while reducing the number of model parameters; (2) The G-MBFF gating fusion mechanism proposed in the present invention can dynamically model the feature contributions between different backbone networks and improve the feature expression ability in complex driving scenarios; (3) The progressive fine-tuning strategy PFS proposed in the present invention can stabilize the training process of multi-backbone fusion networks and improve the model optimization effect; (4) The present invention achieves high recognition accuracy and real-time inference performance in complex real driving environments and has good engineering deployment value. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall network structure of MBAPF proposed in this invention;

[0029] Figure 2 A comparison diagram of the original and pruned structures of ConvNeXt-t modules;

[0030] Figure 3 A comparison diagram of the original ResNeXt50 module and the module structure after pruning;

[0031] Figure 4 A comparison diagram of the original ResNet50 module and the module structure after pruning;

[0032] Figure 5 This is a diagram showing the confusion matrix results of MBAPF on the SAA13 dataset. Detailed Implementation

[0033] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0034] like Figure 1 As shown, this invention proposes a driver behavior recognition method based on a multi-trunk adaptive pruning and gating fusion network, which is based on a framework consisting of a multi-trunk feature extraction module, a trunk perception adaptive pruning module, a gating multi-trunk feature fusion module, and a progressive fine-tuning training module; the specific steps are as follows:

[0035] S1: Construct an image dataset of driver behavior containing complex driving scenarios, using the following method:

[0036] S1-1: Integrate five publicly available driver behavior datasets, namely State Farm, ASU, AUCV2, 100-Driver and 3MDAD, and construct the SAA13 comprehensive driver behavior dataset;

[0037] S1-2: An interval sampling strategy is used for the original video frames to reduce redundant information between adjacent frames;

[0038] S1-3: The dataset covers ten categories of driving behaviors: safe driving (C0), texting with the left hand (C1), texting with the right hand (C2), making a phone call with the left hand (C3), making a phone call with the right hand (C4), operating the center console (C5), drinking water or eating (C6), leaning back (C7), fixing hair or applying makeup (C8), and talking to passengers (C9).

[0039] S1-4: A driver-independent partitioning method is adopted, dividing different drivers into training, validation, and test sets respectively to improve the model's generalization ability. Table 1 shows the detailed partitioning results of the SAA13 dataset.

[0040] Table 1. Detailed information about the SAA13 dataset.

[0041]

[0042] S2: Construct a multi-backbone driver behavior recognition network composed of ConvNeXt-t, ResNeXt50 and ResNet50. The specific method is as follows: use three residual structure networks, ConvNeXt-t, ResNeXt50 and ResNet50, as sub-backbone networks, and use ImageNet pre-trained weights to initialize the parameters of each backbone, so as to achieve joint extraction of different structural features.

[0043] S3: Research on the trunk-aware adaptive pruning method BAAP based on trunk structure differences. The specific method is as follows:

[0044] S3-1: The contribution value of the convolutional channel is calculated using an importance evaluation method based on the L1 norm of the convolutional channel.

[0045] S3-2: Based on the different backbone structure characteristics, corresponding pruning strategies are designed. Among them, ConvNeXt-t mainly prunes the point convolutional layers, and its structure is as follows: Figure 2 As shown, ρ represents the degree of pruning. ResNeXt50 uses a group-level pruning method, and its structure is as follows: Figure 3 As shown, ResNet50 prunes the intermediate channels of the bottleneck layer, and its structure is as follows. Figure 4 As shown;

[0046] S3-3: Remove low-contribution channels according to the preset pruning rate, thereby reducing the number of network parameters and computational complexity.

[0047] S4: Structured compression is performed on different backbone networks to generate lightweight backbone feature extraction networks. The specific content is as follows: different pruning rates are set for ConvNeXt-t, ResNeXt50 and ResNet50 respectively, where the pruning rate of ConvNeXt-t is 0.20, the pruning rate of ResNeXt50 is 0.50 and the pruning rate of ResNet50 is 0.40; the pruned network effectively reduces the model parameter size while maintaining the recognition ability.

[0048] S5: Research on Gated Multi-Branch Feature Fusion Mechanism (G-MBFF), the specific method is as follows:

[0049] S5-1: Extract deep features f from the outputs of different backbone networks respectively. c f x with f r ;

[0050] S5-2: Projecting features of different dimensions to a unified feature space through linear mapping;

[0051] S5-3: The projected features are stitched together and input into the gating module to generate dynamic weights g. c g x With g r ;

[0052] S5-4: Use dynamic weights to weight and fuse different backbone features to form the final fused feature.

[0053] S6: Map the deep features extracted from different backbone networks to a unified feature space and perform dynamic weighted fusion. The gating module adopts a Linear-ReLU-Linear structure and outputs the dynamic weights corresponding to each backbone through the Sigmoid activation function, thereby realizing adaptive feature allocation under different driving scenarios.

[0054] S7: Research on the progressive fine-tuning strategy PFS for stabilizing network training, the specific method is as follows:

[0055] S7-1: In the first stage, all backbone networks are frozen, and only the classifier is trained;

[0056] S7-2: The second stage involves unfreezing the last convolutional layers of each backbone network and jointly optimizing the classifier;

[0057] S7-3: The third stage unfreezes the entire network and performs overall fine-tuning to achieve stable, progressive optimization training.

[0058] S8: The driver behavior prediction results are output through the classifier, thereby constructing a complete MBAPF driver behavior recognition framework. The specific process is as follows: after the fused features are passed through a fully connected classifier, the probability values ​​corresponding to each category are output, and the Softmax activation function is used to generate a normalized probability distribution. The category with the highest probability is taken as the final driver behavior recognition result.

[0059] During the experiment, this invention compressed each backbone network with different pruning rates and statistically analyzed the performance changes of the models under different pruning rates. Table 2 shows the performance results of each backbone network under different pruning rates. As can be seen from Table 2, appropriate pruning can not only reduce model complexity but also improve model recognition performance to a certain extent. Among them, ResNeXt50 and ResNet50 showed improved accuracy under moderate pruning rates, indicating that the pruning process effectively removed redundant feature representations.

[0060] Table 2 Performance results of each backbone network under different pruning rates

[0061]

[0062] The overall ablation results of MBAPF are summarized in Table 3, which shows the progressive performance improvements of different components. Starting from the pruning fusion baseline, introducing PFS improved accuracy by 1.05% without introducing any additional cost. Furthermore, integrating the G-MBFF mechanism into the model achieved a state-of-the-art accuracy of 92.61%, indicating that modeling the relationships between heterogeneous feature sources can more effectively fuse features, thus fully demonstrating the complementary advantages of multiple backbone networks.

[0063] Table 3 Overall Ablation Experiment Results

[0064]

[0065] To verify the comprehensive performance of the proposed MBAPF framework in complex driving scenarios, this embodiment compares the present invention with existing mainstream driver behavior recognition methods. Table 4 shows the performance comparison results between the present invention and existing methods. As can be seen from Table 4, the present invention achieves higher driver behavior recognition accuracy while ensuring real-time inference capabilities. Furthermore, the present invention performs a visual analysis of the recognition results for each category of driving behavior, and the confusion matrix results are shown below. Figure 5 As shown. By Figure 5 As can be seen, this invention achieves high recognition accuracy for most driving behaviors, with the "reaching behind" category showing the highest accuracy. However, the "texting-left" and "texting-right" categories are relatively more difficult to recognize due to occlusion and small target issues. In summary, the MBAPF driver behavior recognition method proposed in this invention effectively balances recognition accuracy, model lightweighting, and real-time deployment performance, demonstrating significant engineering application value.

[0066] Table 4 Performance comparison with existing methods

[0067]

[0068] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A driver behavior recognition method based on a multi-trunk adaptive pruning and gating fusion network, characterized in that: Includes the following steps: S1: Construct an image dataset of driver behavior containing complex driving scenarios; S2: Construct a multi-backbone driver behavior recognition network composed of ConvNeXt-t, ResNeXt50 and ResNet50; S3: Research on the trunk sensing adaptive pruning method BAAP based on trunk structure differences; S4: Perform structured compression on different backbone networks to generate a lightweight backbone feature extraction network; S5: Research on Gated Multi-Branch Feature Fusion Mechanism G-MBFF; S6: Map the deep features extracted by different backbone networks to a unified feature space and perform dynamic weighted fusion; S7: Research on the progressive fine-tuning strategy PFS for stable network training; S8: The driver behavior prediction results are output through the classifier, thereby constructing a complete MBAPF driver behavior recognition framework.

2. The driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network according to claim 1, characterized in that: The specific method for step S1 is as follows: S1-1: Integrate five publicly available driver behavior datasets, namely State Farm, ASU, AUCV2, 100-Driver and 3MDAD, and construct the SAA13 comprehensive driver behavior dataset; S1-2: An interval sampling strategy is used for the original video frames to reduce redundant information between adjacent frames; S1-3: The dataset covers ten categories of driving behaviors, including safe driving, texting with the left hand, texting with the right hand, making a phone call with the left hand and making a phone call with the right hand, operating the center console, drinking water or eating, leaning back, fixing hair or applying makeup, and talking to passengers. S1-4: Using an independent driver segmentation method, different drivers are separately assigned to the training set, validation set, and test set.

3. The driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network according to claim 1, characterized in that: The specific method of step S2 is as follows: three residual structure networks, ConvNeXt-t, ResNeXt50 and ResNet50, are used as sub-backbone networks, and ImageNet pre-trained weights are used to initialize the parameters of each backbone to achieve joint extraction of different structural features.

4. The driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network according to claim 1, characterized in that: The specific method for step S3 is as follows: S3-1: The contribution value of the convolutional channel is calculated using an importance evaluation method based on the L1 norm of the convolutional channel. S3-2: Design corresponding pruning strategies based on different backbone structure characteristics. ConvNeXt-t prunes the point convolutional layers, ResNeXt50 adopts a group-level pruning method, and ResNet50 prunes the middle channels of the bottleneck layer. S3-3: Remove low-contribution channels according to the preset pruning rate to reduce the number of network parameters and computational complexity.

5. The driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network according to claim 1, characterized in that: The specific content of step S4 is as follows: different pruning rates are set for ConvNeXt-t, ResNeXt50 and ResNet50 respectively, wherein the pruning rate of ConvNeXt-t is 0.20, the pruning rate of ResNeXt50 is 0.50 and the pruning rate of ResNet50 is 0.

40.

6. The driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network according to claim 1, characterized in that: The specific method for step S5 is as follows: S5-1: Extract the deep features f from the outputs of the ConvNeXt-t, ResNeXt50, and ResNet50 backbone networks respectively. c f x with f r ; S5-2: Projecting features of different dimensions to a unified feature space through linear mapping; S5-3: Concatenate the projected features and input them into the gating module to generate the corresponding dynamic weights g. c g x With g r ; S5-4: Use dynamic weights to weight and fuse different backbone features to form the final fused feature.

7. The driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network according to claim 6, characterized in that: The gating module adopts a Linear-ReLU-Linear structure and outputs dynamic weights corresponding to each backbone through the Sigmoid activation function to achieve adaptive feature allocation under different driving scenarios.

8. The driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network according to claim 1, characterized in that: The specific method for step S7 is as follows: S7-1: In the first stage, all backbone networks are frozen, and only the classifier is trained; S7-2: The second stage involves unfreezing the last convolutional layers of each backbone network and jointly optimizing the classifier; S7-3: The third stage unfreezes the entire network and performs overall fine-tuning to achieve stable, progressive optimization training.

9. The driver behavior recognition method based on multi-trunk adaptive pruning and gating fusion network according to claim 1, characterized in that: The specific process of step S8 is as follows: after the fused features are processed by a fully connected classifier, the probability values ​​corresponding to each category are output, and a normalized probability distribution is generated by using the Softmax activation function. The category corresponding to the highest probability is taken as the final driver behavior recognition result.

Images