Driver behavior recognition method based on compressed video for vehicle-mounted equipment

Abstract

This paper provides a method for recognizing driver behavior from compressed video for in-vehicle devices. First, ffmpeg is used to directly process the MPEG-4 compressed video data stream captured by the camera, effectively utilizing the motion vectors and residual errors in the P-frame. Secondly, a multi-branch spatiotemporal convolutional network model is designed. A lightweight 2D / 3D convolutional network structure is used to extract appearance and action information from the I-frame and P-frame, respectively, thereby reducing the number of model parameters and computational complexity. Furthermore, the method combines a cross-layer connection module with a spatiotemporal feature pooling module to fuse appearance and action information, achieving spatiotemporal representation of distracted driving behavior. Finally, the method adopts a knowledge distillation learning strategy, introducing a teacher model to guide the lightweight model, further improving recognition accuracy. This method has high accuracy, low complexity, and good real-time performance, making it suitable for in-vehicle embedded devices.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition in computer vision, and in particular to a method for driver behavior recognition using compressed video for vehicle-mounted equipment. Background Art

[0002] Recent traffic accident investigations have revealed that over 40% of accidents are related to driver distraction, such as phone calls, smoking, and chatting with passengers. Therefore, driver behavior recognition for in-vehicle devices is of vital importance for road safety and intelligent transportation.

[0003] The existing technology patent CN109866686A discloses an intelligent active safety assisted driving system and method based on real-time video analysis, including a real-time video acquisition module, a video processing module, an identification system, a risk assessment system, a high-performance image processing center, and a voice display module: the real-time video acquisition module is responsible for real-time acquisition of video data; the video processing module compresses and grayscales the video; the identification system uses a deep learning neural network to identify and analyze external risk factors of the vehicle and driver behavior; the risk assessment system evaluates according to the behavioral rules of possible dangers, speeding, and running red lights, and gives digital results of the evaluation; the high-performance image processing center uses a low-power, high-computing-power image processing microcomputer; the voice display module uses the results of the risk assessment system and visualizes the results, displays them in real time, and prompts the driver through voice.

[0004] Driver behavior recognition is a fundamental functional module of the connected vehicle system. Onboard cameras monitor the driver in real time inside the vehicle cabin. When distracted driving occurs, the system can promptly alert the driver or switch to autonomous driving mode, providing early warning of traffic accidents. However, automatic driver behavior recognition based on computer vision currently faces a significant bottleneck: balancing recognition accuracy and computational efficiency. Specifically:

[0005] (1) Driver behavior recognition methods based on static images usually take specific frames as input and directly identify different driver behavior categories through end-to-end training. The advantage of this type of model is that it has low computational complexity and is suitable for on-board equipment with high real-time requirements. However, its disadvantage is that it does not utilize inter-frame dynamic information, making it difficult to break through the bottleneck of recognition accuracy.

[0006] (2) Driver behavior recognition methods based on video sequences use a video sequence as input. This type of method effectively utilizes inter-frame motion information to extract the spatiotemporal representation of driver behavior, resulting in higher recognition accuracy. However, it consumes more computing resources and has poor real-time performance, making it unsuitable for vehicle-mounted devices. Summary of the Invention

[0007] To solve the above problems, the present invention provides a method for driver behavior recognition using compressed video for an on-board device, comprising the following steps:

[0008] Step 1: Use the ffmpeg open source library to process the MPEG-4 compressed video data stream captured by the camera in real time;

[0009] Step 2: Design a multi-branch spatiotemporal convolutional network model, which contains three network branches, namely I-branch, M-branch, and R-branch, to process I-frame, motion vector, and residual error respectively. I-branch uses the ShuffleNetV2 structure to extract appearance cues. M-branch and R-branch expand the 2D convolution kernel in ShuffleNetV2 to 3D convolution kernel to process motion cues and compress the number of convolution channels.

[0010] Step 3: Combine the cross-layer connection unit with the spatiotemporal feature pooling module to fuse appearance information and action information. The cross-layer connection module can fuse the intermediate layer feature maps of different network branches, map the 3D feature maps learned by the M-branch and R-branch into 2D feature maps through temporal convolution, and perform feature splicing with the I-branch. The spatiotemporal feature pooling module is intended to aggregate the last layer feature vectors of different network branches, learn the high-dimensional potential information of the feature vectors through Conv1D convolution, and use trilinear pooling operations to aggregate the feature vectors extracted by different branches, thereby realizing the spatiotemporal representation of driving behavior.

[0011] Step 4: Adopting a knowledge distillation learning strategy, we introduce a two-stream I3D model as the teacher model and a lightweight spatiotemporal network model as the student model. The teacher model's output is used as soft labels to guide the training of the lightweight model, compensating for the performance degradation caused by the reduced number of parameters and thereby improving the model's recognition accuracy.

[0012] Step 5: Model training and testing.

[0013] A further improvement is that the step 1 is specifically as follows: MPEG-4 encodes the video sequence through the group of pictures GOP, that is, only the initial frame I-frame of each GOP is saved, and the predicted frame P-frame is estimated through motion vector and residual error.

[0014] A further improvement is that the motion vector records the motion offset of the image block from the I-frame to the P-frame, and the residual error represents the pixel change between the P-frame and its reference I-frame after motion compensation; The reconstructed frame at time is expressed as:

[0015] (1)

[0016] in express Video frame at the moment, and Respectively Motion vector and residual error within a time period.

[0017] A further improvement is that the step 2 includes:

[0018] Step 201: I-branch network branch design. I-branch adopts ShuffleNetV2 network structure. Its input is the initial frame I-frame of the video, with a size of ; The first layer of I-branch is The basic convolution layer and the intermediate layer use a lightweight convolution unit, which consists of Convolutional layer and It consists of a depthwise separable convolutional layer, which uses channel separation and channel rearrangement operations; the last layer of I-branch is a global mean pooling layer;

[0019] I-branch feature learning process Defined as:

[0020] (2)

[0021] in Represents the convolution operation; For I-branch The convolution parameters of the layer; and Respectively represent I-branch Layer and The output feature map of the layer;

[0022] Step 202: M-branch network branch design. M-branch adopts 3DShuffleNetV2 0.25x network structure. Its input is motion vector sequence with size ; The first layer of M-branch is The basic convolution layer and the intermediate layer use a lightweight convolution unit, which consists of Convolutional layer and It is composed of depth-wise separable convolutional layers. In addition, the unit also uses channel separation and channel rearrangement operations;

[0023] The last layer of M-branch is the global mean pooling layer, and 3DShuffleNetV2 0.25x expands the 2D convolution kernel to a 3D convolution kernel;

[0024] M-branch feature learning process Defined as:

[0025] (3)

[0026] in Represents the convolution operation; For M-branch The convolution parameters of the layer; and Respectively represent the M-branch Layer and The output feature map of the layer;

[0027] Step 203: R-branch network branch design. R-branch adopts 3DShuffleNetV2 0.25x network structure. Its input is residual error sequence with size ; Feature learning process of R-branch and Same, defined as:

[0028] (4)

[0029] in Represents the convolution operation; For R-branch The convolution parameters of the layer; and Respectively represent R-branch Layer and The output feature map of the layer.

[0030] A further improvement is that step 3 includes:

[0031] Step 301: Design a cross-layer connection unit, which fuses the intermediate layer feature maps of different network branches; the layer connection unit first adopts Temporal convolution maps the 3D feature maps learned by M-branch and R-branch into 2D feature maps; then concatenates them with the 2D feature maps output by I-branch and performs Convolution realizes dimensional transformation; specifically, the first The layer feature maps are: , and , then the cross-layer connection unit operation Defined as:

[0032] (5)

[0033] in represents the convolution operation, [ ] represents the feature cascade operation; and for Temporal convolution parameters, for Convolution parameters; represents the output feature map;

[0034] Step 302: Design a spatiotemporal feature pooling module, which is used to fuse the top-level feature vectors of I-branch, M-branch, and R-branch: , , ; The fused feature vector is defined as:

[0035] (6)

[0036] in represents the spatiotemporal feature pooling module, Indicates the relevant parameters of the module. is the fusion feature vector;

[0037] The spatiotemporal feature pooling module can be decomposed into three stages: first, the feature vector is mapped to a high-dimensional latent space through Conv1D convolution; second, the feature vectors extracted by different branches are aggregated using trilinear pooling operation; finally, the fused feature vector is normalized using regularization operation. The three stages of the spatiotemporal feature pooling module can be expressed as follows:

[0038] (7)

[0039] (8)

[0040] (9)

[0041] in , , Respectively Convolutional layer parameters; , is the number and size of convolution kernels; , , Represent high-dimensional latent feature vectors respectively; A full 1 vector is used for summing and pooling, It is the product operation of the corresponding elements of the matrix; and denote the symbolic function and the matrix two-norm respectively; Represents the normalized output feature vector.

[0042] A further improvement is that step 4 includes:

[0043] Step 401: Soft label acquisition, the output of the teacher model and the student model can be expressed as and ,in Indicates the number of categories; the Softmax function is used to calculate the probability distribution of different categories, which can be defined as:

[0044] (10)

[0045] (11)

[0046] in It is a hyperparameter used to adjust the knowledge distillation temperature threshold; express Belong to The probability of each category; Represents the soft labels obtained by knowledge distillation;

[0047] Step 402: Loss function design. The loss function of the model consists of two parts: cross entropy loss and KL divergence loss, which can be specifically expressed as:

[0048] (12)

[0049] (13)

[0050] in represents the indicator function; and Represent the true value and the predicted result respectively; represents the probability distribution of the student model; and Represents the soft labels generated by the student model and the teacher model;

[0051] The final loss of the model is a weighted combination of the losses of different learning tasks, expressed as:

[0052] (14)

[0053] in represents the weight hyperparameter; are the relevant parameters of the student model, represents the two-norm distance.

[0054] A further improvement is that step 5 includes:

[0055] Step 501: Model training. First, train the I-branch, M-branch, and R-branch separately. Then, use the feature fusion module and knowledge distillation strategy to learn the representation.

[0056] Step 502: Model testing, first split the test video into The I-frame, motion vector and residual error are extracted from the compressed video stream using FFmpeg; the multimodal data is then fed into the trained model to achieve driver behavior recognition.

[0057] The present invention has the following beneficial effects: First, it uses ffmpeg to directly process the MPEG-4 compressed video data stream captured by the camera, effectively utilizing the motion vectors and residual errors in the P-frame. Second, it designs a multi-branch spatiotemporal convolutional network model. Using a lightweight 2D / 3D convolutional network structure, it extracts appearance and motion information from the I-frame and P-frame, respectively, thereby reducing the model's parameter count and computational complexity. Furthermore, the present invention combines a cross-layer connection module with a spatiotemporal feature pooling module to fuse appearance and motion information, achieving spatiotemporal representation of distracted driving behavior. Finally, the present invention employs a knowledge distillation learning strategy, introducing a teacher model to guide the lightweight model, further improving recognition accuracy. The proposed method for identifying prospective driver behavior from compressed video features high accuracy, low complexity, and good real-time performance, making it suitable for in-vehicle embedded devices. The present invention has significant application value in the field of traffic safety. The present invention involves four methods: data processing, network design, feature fusion, and learning strategy. The proposed method for identifying prospective driver behavior from compressed video features high accuracy, low complexity, and good real-time performance, making it suitable for in-vehicle embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of the present invention.

[0059] Figure 2 It is the motion vector and residual error of adjacent frames of compressed video.

[0060] Figure 3 Schematic diagram of the multi-branch spatiotemporal convolutional network model.

[0061] Figure 4 Schematic diagram of the ShuffleNetV2 lightweight convolution unit. The stride is 1.

[0062] Figure 5Schematic diagram of the ShuffleNetV2 lightweight convolution unit with a step size of 2.

[0063] Figure 6 Schematic diagram of the cross-layer connection unit of the feature fusion module.

[0064] Figure 7 Schematic diagram of the spatiotemporal feature pool module of the feature fusion module.

[0065] Figure 8 Schematic diagram of knowledge distillation learning strategy.

[0066] Figure 9 This is the overall framework diagram of the compressed video driver behavior recognition model for vehicle-mounted equipment. DETAILED DESCRIPTION

[0067] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the examples. The examples are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0068] like Figure 1 As shown, this embodiment provides a method for driver behavior recognition using compressed video for an in-vehicle device, comprising the following steps:

[0069] S1: Use the ffmpeg open-source library to process the MPEG-4 compressed video data stream captured by the camera in real time. MPEG-4 encodes video sequences using groups of pictures (GOPs), meaning only the initial I-frame of each GOP is stored, while the predicted P-frame is estimated using motion vectors and residual errors. Compressed and encoded video occupies less storage space, and motion vectors and residual errors can be read directly from the encoded information without pre-calculation. Specifically, the motion vector records the motion offset of the image block from the I-frame to the P-frame, while the residual error represents the pixel change between the P-frame and its reference I-frame after motion compensation. The reconstructed frame at time is expressed as:

[0070] (1)

[0071] in express Video frame at the moment, and Respectively Motion vector and residual error within a time period; In short, I-frame, motion vector and residual error are the three basic elements in compressed video. Figure 2 As shown in Figure 1, the I-frame records the appearance information of the initial frame of the video; the motion vector represents the global offset, and the residual error reflects the detailed changes in the moving area.

[0072] S2: Design a multi-branch spatiotemporal convolutional network model, which contains three network branches that can process I-frame, motion vector and residual error respectively; these three network branches are named I-branch, M-branch and R-branch respectively; I-branch uses ShuffleNetV2 structure to extract appearance clues; M-branch and R-branch expand the 2D convolution kernel in ShuffleNetV2 to 3D convolution kernel to process action clues and compress the number of convolution channels, aiming to further reduce the number of model parameters and computational complexity; the overall structure of the multi-branch spatiotemporal convolutional network model is as follows Figure 3 shown.

[0073] S201: I-branch network branch design. I-branch adopts the ShuffleNetV2 network structure. Its input is the initial frame I-frame of the video, with a size of ; The first layer of I-branch is The basic convolution layer and the intermediate layer use lightweight convolution units, such as Figure 4 and 5 As shown; the unit consists of Convolutional layer and It consists of a depthwise separable convolutional layer, which uses channel separation and channel rearrangement operations; the last layer of I-branch is a global mean pooling layer;

[0074] I-branch feature learning process Defined as:

[0075] (2)

[0076] in Represents the convolution operation; For I-branch The convolution parameters of the layer; and Respectively represent I-branch Layer and The output feature map of the layer;

[0077] S202: M-branch network branch design. M-branch adopts 3DShuffleNetV2 0.25x network structure. Its input is motion vector sequence with size ; The first layer of M-branch is The basic convolution layer and the intermediate layer use a lightweight convolution unit, which consists of Convolutional layer and It is composed of depth-wise separable convolutional layers. In addition, the unit also uses channel separation and channel rearrangement operations;

[0078] The last layer of N-branch is the global mean pooling layer, and 3DShuffleNetV2 0.25x expands the 2D convolution kernel to a 3D convolution kernel;

[0079] M-branch feature learning process Defined as:

[0080] (3)

[0081] in Represents the convolution operation; For M-branch The convolution parameters of the layer; and Respectively represent the M-branch Layer and The output feature map of the layer;

[0082] S203: R-branch network branch design, R-branch adopts 3DShuffleNetV2 0.25x network structure, its input is residual error sequence, size is ; Feature learning process of R-branch and Same, defined as:

[0083] (4)

[0084] in Represents the convolution operation; For R-branch The convolution parameters of the layer; and Respectively represent R-branch Layer and The output feature map of the layer.

[0085] S3: Combine the cross-layer connection unit and the spatiotemporal feature pooling module to fuse the appearance information and action information. The cross-layer connection module can fuse the intermediate layer feature maps of different network branches, map the 3D feature maps learned by M-branch and R-branch into 2D feature maps through temporal convolution, and perform feature splicing with I-branch; the spatiotemporal feature pooling module is intended to aggregate the last layer feature vectors of different network branches, learn the high-dimensional potential information of the feature vectors through Conv1D convolution, and use trilinear pooling operation to aggregate the feature vectors extracted by different branches, thereby realizing the spatiotemporal representation of driving behavior. The structure of the feature fusion module is as follows: Figure 6 and 7 shown.

[0086] S301: If Figure 6 As shown in the figure, a cross-layer connection unit is designed, which fuses the intermediate layer feature maps of different network branches; the layer connection unit first adopts Temporal convolution maps the 3D feature maps learned by M-branch and R-branch into 2D feature maps; then concatenates them with the 2D feature maps output by I-branch and performs Convolution realizes dimensional transformation; specifically, the first The layer feature maps are: , and , then the cross-layer connection unit operation Defined as:

[0087] (5)

[0088] in represents the convolution operation, [ ] represents the feature cascade operation; and for Temporal convolution parameters, for Convolution parameters; represents the output feature map;

[0089] S302: If Figure 7 As shown in the figure, a spatiotemporal feature pooling module is designed, which is used to fuse the top-level feature vectors of I-branch, M-branch and R-branch: , , ; The fused feature vector is defined as:

[0090] (6)

[0091] in represents the spatiotemporal feature pooling module, Indicates the relevant parameters of the module. is the fusion feature vector;

[0092] The spatiotemporal feature pooling module can be decomposed into three stages: first, the feature vector is mapped to a high-dimensional latent space through Conv1D convolution; second, the feature vectors extracted by different branches are aggregated using trilinear pooling operation; finally, the fused feature vector is normalized using regularization operation. The three stages of the spatiotemporal feature pooling module can be expressed as follows:

[0093] (7)

[0094] (8)

[0095] (9)

[0096] in , , Respectively Convolutional layer parameters; , is the number and size of convolution kernels; , , Represent high-dimensional latent feature vectors respectively; A full 1 vector is used for summing and pooling, It is the product operation of the corresponding elements of the matrix; and denote the symbolic function and the matrix two-norm respectively; Represents the normalized output feature vector.

[0097] S4: Using the knowledge distillation learning strategy, we introduce the Two-stream I3D as the teacher model and the lightweight spatiotemporal network model as the student model. The output of the teacher model is used as a soft label to guide the training of the lightweight model, compensating for the performance degradation caused by the reduction of parameters, thereby improving the recognition accuracy of the model. The knowledge distillation learning strategy framework is shown in the figure below. Figure 8 shown.

[0098] S401: Soft label acquisition, the output of the teacher model and the student model can be expressed as and ,in Indicates the number of categories; the Softmax function is used to calculate the probability distribution of different categories, which can be defined as:

[0099] (10)

[0100] (11)

[0101] in It is a hyperparameter used to adjust the knowledge distillation temperature threshold; express Belong to The probability of each category; Represents the soft labels obtained by knowledge distillation;

[0102] S402: Loss function design. The loss function of the model consists of two parts: cross entropy loss and KL divergence loss, which can be specifically expressed as:

[0103] (12)

[0104] (13)

[0105] in represents the indicator function; and Represent the true value and the predicted result respectively; represents the probability distribution of the student model; and Represents the soft labels generated by the student model and the teacher model;

[0106] The final loss of the model is a weighted combination of the losses of different learning tasks, expressed as:

[0107] (14)

[0108] in represents the weight hyperparameter; are the relevant parameters of the student model, represents the two-norm distance.

[0109] S5: Model training and testing; The overall framework of the driver behavior recognition model designed by the present invention is as follows Figure 9 shown.

[0110] S501: Model training. First, train the I-branch, M-branch, and R-branch separately. Then, use the feature fusion module and knowledge distillation strategy to learn the representation.

[0111] S502: Model testing. First, the test video is divided into We then use FFmpeg to extract I-frames, motion vectors, and residual errors from the compressed video stream. We then feed this multimodal data into a trained model to achieve driver behavior recognition.

[0112] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A method for driver behavior recognition based on compressed video for vehicle-mounted equipment, characterized in that: The following steps are involved: Step 1: Use the ffmpeg open source library to process the MPEG-4 compressed video data stream captured by the camera in real time; Step 2: Design a multi-branch spatiotemporal convolutional network model, which contains three network branches, namely I-branch, M-branch and R-branch, to process I-frame, motion vector and residual error respectively. I-branch adopts the ShuffleNetV2 structure to extract appearance cues, while M-branch and R-branch expand the 2D convolution kernel in ShuffleNetV2 to 3D convolution kernel to process motion cues and compress the number of convolution channels. Step 3: Combine the cross-layer connection unit with the spatiotemporal feature pooling module to fuse appearance information and action information. The cross-layer connection module can fuse the intermediate layer feature maps of different network branches, map the 3D feature maps learned by the M-branch and R-branch into 2D feature maps through temporal convolution, and perform feature splicing with the I-branch. The spatiotemporal feature pooling module is intended to aggregate the last layer feature vectors of different network branches, learn the high-dimensional potential information of the feature vectors through Conv1D convolution, and use trilinear pooling operations to aggregate the feature vectors extracted by different branches, thereby realizing the spatiotemporal representation of driving behavior. Step 4: Adopting a knowledge distillation learning strategy, we introduce a two-stream I3D model as the teacher model and a multi-branch spatiotemporal convolutional network model as the student model. The teacher model output is used as soft labels to guide the training of the lightweight model, compensating for the performance degradation caused by the reduction in parameters and thereby improving the model's recognition accuracy. Step 5: Model training and testing.

2. The method for driver behavior recognition using compressed video for vehicle-mounted equipment according to claim 1, characterized in that: The step 1 is specifically as follows: MPEG-4 encodes the video sequence through the group of pictures GOP, that is, only the initial frame I-frame of each GOP is saved, and the predicted frame P-frame is estimated through motion vector and residual error.

3. The method for driver behavior recognition using compressed video for vehicle-mounted equipment according to claim 2, wherein: The motion vector records the motion offset of the image block from the I-frame to the P-frame, while the residual error represents the pixel change between the P-frame and its reference I-frame after motion compensation. The reconstructed frame at time (t+τ) is expressed as: in represents the video frame at time t, and They represent the motion vector and residual error within the τ time period respectively.

4. The method for driver behavior recognition using compressed video for vehicle-mounted equipment according to claim 1, wherein: The step 2 includes, Step 201: I-branch network design. I-branch uses the ShuffleNetV2 network structure. Its input is the initial video frame I-frame, with a size of 3×112×112. The first layer of I-branch is a 3×3 basic convolution layer. The middle layer uses a lightweight convolution unit, which consists of a 1×1 convolution layer and a 3×3 depthwise separable convolution layer. This unit uses channel separation and channel rearrangement operations. The last layer of I-branch is a global mean pooling layer. The I-branch feature learning process INet(·|) is defined as: Where * represents the convolution operation; is the convolution parameter of the lth layer of I-branch; and Represent the output feature maps of the (l-1)th layer and the lth layer of I-branch respectively; Step 202: Design the M-branch network. The M-branch uses a 3DShuffleNetV2 0.25x network structure. Its input is a motion vector sequence with a size of 2×112×112×16. The first layer of the M-branch is a 3×3×3 base convolution layer. The middle layer uses a lightweight convolution unit, which consists of a 1×1×1 convolution layer and a 3×3×3 depthwise separable convolution layer. In addition, this unit also uses channel separation and channel rearrangement operations. The last layer of M-branch is the global mean pooling layer, and 3DShuffleNetV2 0.25x expands the 2D convolution kernel to a 3D convolution kernel; The M-branch feature learning process MNet(·|) is defined as: Where * represents the convolution operation; is the convolution parameter of the lth layer of M-branch; and Represent the output feature maps of the (l-1)th layer and the lth layer of M-branch respectively; Step 203: R-branch network design. R-branch uses a 3DShuffleNetV2 0.25x network structure. Its input is a residual error sequence with a size of 3×112×112×16. The feature learning process of R-branch RNet(·|) is the same as that of MNet(·|), and is defined as: Where * represents the convolution operation; is the convolution parameter of the lth layer of R-branch; and They represent the output feature maps of the (l-1)th layer and the lth layer of R-branch respectively.

5. The method for driver behavior recognition using compressed video for vehicle-mounted equipment according to claim 4, characterized in that: The step 3 includes: Step 301: Design a cross-layer connection unit to fuse the intermediate layer feature maps of different network branches. The cross-layer connection unit first uses 1×1×T temporal convolution to map the 3D feature maps learned by M-branch and R-branch into 2D feature maps. Then, it concatenates the 2D feature maps output by I-branch and implements dimensionality transformation through 1×1 convolution. Specifically, the l-th layer feature maps of I-branch, M-branch, and R-branch are: and Then the cross-layer connection unit operation CLCM(·) is defined as: Where * represents the convolution operation, [] represents the feature cascade operation; and is the 1×1×T time convolution parameter, is the 1×1 convolution parameter; represents the output feature map; Step 302: Design a spatiotemporal feature pooling module, which is used to fuse the last layer feature vectors of I-branch, M-branch, and R-branch: The fused feature vector is defined as: Where STTPM(·|) represents the spatiotemporal feature pooling module, Indicates the relevant parameters of the module. is the fusion feature vector; The spatiotemporal feature pooling module can be decomposed into three stages: first, the feature vector is mapped to a high-dimensional latent space through Conv1D convolution; second, the feature vectors extracted by different branches are aggregated using trilinear pooling operation; finally, the fused feature vector is normalized using regularization operation; The three stages of the spatiotemporal feature pooling module can be expressed as: in They represent the parameters of the Conv1d(·|) convolution layer respectively; n, k are the number and size of convolution kernels; Represent high-dimensional latent feature vectors respectively; A full 1 vector is used for summing and pooling, is the product operation of the corresponding matrix elements; sign(·|) and ||·||2 represent the sign function and the matrix two-norm respectively; Represents the normalized output feature vector.

6. The method for driver behavior recognition using compressed video for vehicle-mounted equipment according to claim 1, wherein: The step 4 includes, Step 401: Soft label acquisition, the output of the teacher model and the student model can be expressed as z t ={(z t ) 1 ,(z t ) 2 ,...,(z t ) J } and z s ={(z s ) 1 ,(z s ) 2 ,...,(z s ) J }, where J represents the number of categories; the Softmax function is used to calculate the probability distribution of different categories, which can be defined as: Where T is a hyperparameter used to adjust the knowledge distillation temperature threshold; p(j|z) represents the probability that z belongs to the jth category; p T (j|z) represents the soft label obtained by knowledge distillation; Step 402: Loss function design. The loss function of the model consists of two parts: cross entropy loss and KL divergence loss, which can be specifically expressed as: Where 1(·) represents the indicator function; y j and Represent the true value and the predicted result respectively; p(j|z s ) represents the probability distribution of the student model; p T (j|z s ) and p T (j|z t ) represents the soft labels generated by the student model and the teacher model; The final loss of the model is a weighted combination of the losses of different learning tasks, expressed as: Where λ represents the weight hyperparameter; θ stu are the relevant parameters of the student model, represents the two-norm distance.

7. The method for driver behavior recognition using compressed video for vehicle-mounted equipment according to claim 1, wherein: The step 5 includes: Step 501: Model training. First, train the I-branch, M-branch, and R-branch separately. Then, use the feature fusion module and knowledge distillation strategy to learn the representation. Step 502: Model testing. First, the test video is divided into K segments. I-frames, motion vectors, and residual errors are extracted from the compressed video stream using ffmpeg. Then, the multimodal data is fed into the trained model to achieve driver behavior recognition.

Images