Distraction driving behavior recognition method and system based on appearance posture fusion network

Abstract

The application discloses a distraction driving behavior recognition method and system based on an appearance posture fusion network, wherein the method comprises the following steps: an appearance posture fusion network after training is used to process driving behavior video data based on a human body posture estimation algorithm, joint coordinates of an upper body of a driver are obtained, and skeleton data is obtained according to the joint coordinates; the joint coordinates are input into a first graph convolution network to obtain joint features; the skeleton data is input into a second graph convolution network to obtain skeleton features; the driving behavior video data is input into a hybrid convolution network to obtain appearance features; the joint features and the skeleton features are fused to obtain primary fusion features; the primary fusion features and the appearance features are fused to obtain secondary fusion features; and the secondary fusion features are classified to obtain a driving behavior recognition result.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for recognizing distracted driving behavior based on appearance and pose fusion networks. Background Technology

[0002] Among the micro-level impacts of road traffic safety risks comprised of the four elements of "people, vehicles, roads, and environment," compared to certain extreme environments, severe road conditions, and rare vehicle malfunctions, the primary cause of traffic accidents is often driver distraction. Driver distraction refers to a phenomenon where a driver's attention is directed towards activities unrelated to normal driving, leading to a decline in driving ability. Domestic and international researchers have achieved substantial results in studies related to driver distraction detection. Most studies uncover characteristics of distracted driving from drivers' body language, with the identification of common distracted driving behaviors such as drinking water and answering the phone becoming the mainstream of current research.

[0003] In recent years, with the rise of Advanced Driver Assistance Systems (ADAS), in-vehicle equipment has undergone extensive upgrades. This has not only made it more convenient to acquire multi-dimensional data during driving but also provided hardware support for improving in-vehicle computing power. Meanwhile, thanks to the efficient feature extraction capabilities of deep learning algorithms, the identification and assessment of driver distraction has ushered in broad development prospects. By collecting visual sensor data and processing it intelligently through deep learning algorithms, drivers can detect potential dangers in the shortest possible time, thereby avoiding traffic accidents. However, although existing image-based processing methods can achieve good accuracy, these methods are inevitably affected by environmental noise in the driver's cabin. Furthermore, the lack of visual features representing the combined movements of the human body and objects makes it difficult to distinguish between similar driving behaviors when utilizing human skeletal joint data. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for recognizing distracted driving behavior based on an appearance and pose fusion network. By constructing a multi-stream network framework that integrates driver skeleton sequences and appearance features, complementary advantages are achieved, thereby combining multimodal input features to improve model performance.

[0005] On the one hand, a distracted driving behavior recognition method based on appearance pose fusion network is provided, including:

[0006] Acquire video data of the driving behavior to be identified;

[0007] The video data of the driving behavior to be identified is input into the trained appearance and pose fusion network to obtain the driving behavior recognition result.

[0008] The trained appearance and pose fusion network is used to: process driving behavior video data based on a human pose estimation algorithm to obtain the joint coordinates of the driver's upper body, and obtain skeletal data based on the joint coordinates; input the joint coordinates into a first-graph convolutional network to obtain joint features; input the skeletal data into a second-graph convolutional network to obtain skeletal features; input the driving behavior video data into a hybrid convolutional network to obtain appearance features; fuse the joint features and skeletal features to obtain a first-level fusion feature; fuse the first-level fusion feature with the appearance feature to obtain a second-level fusion feature; and classify the second-level fusion feature to obtain the driving behavior recognition result.

[0009] On the other hand, a distracted driving behavior recognition system based on an appearance and posture fusion network is provided, including:

[0010] The acquisition module is configured to acquire video data of the driving behavior to be identified.

[0011] The recognition module is configured to input the video data of the driving behavior to be recognized into the trained appearance pose fusion network to obtain the driving behavior recognition result.

[0012] The trained appearance and pose fusion network is used to: process driving behavior video data based on a human pose estimation algorithm to obtain the joint coordinates of the driver's upper body, and obtain skeletal data based on the joint coordinates; input the joint coordinates into a first-graph convolutional network to obtain joint features; input the skeletal data into a second-graph convolutional network to obtain skeletal features; input the driving behavior video data into a hybrid convolutional network to obtain appearance features; fuse the joint features and skeletal features to obtain a first-level fusion feature; fuse the first-level fusion feature with the appearance feature to obtain a second-level fusion feature; and classify the second-level fusion feature to obtain the driving behavior recognition result.

[0013] Furthermore, an electronic device is also provided, including:

[0014] Memory, used for non-transitory storage of computer-readable instructions; and

[0015] Processor, for executing the computer-readable instructions,

[0016] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0017] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0018] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0019] The above technical solution has the following advantages or beneficial effects:

[0020] This invention designs a multi-stream network framework based on spatiotemporal graph convolutional networks and hybrid convolutional networks for detecting distracted driving behavior. This invention benefits from the coordinated use of skeleton sequences and appearance features: on the one hand, SRTD-GCN is designed for extracting spatiotemporal information from the skeleton sequence; on the other hand, RT-HCN is designed for modeling appearance features from RGB video. Ultimately, this invention achieves complementary advantages by constructing a multi-stream network framework integrating driver skeleton sequences and appearance features, thereby combining multimodal input features to improve model performance. Experiments on two benchmark datasets demonstrate that our framework outperforms state-of-the-art models, and a detailed analysis of the contribution of each component is provided. These results offer an effective solution for improving the safety of intelligent vehicles. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 It is the overall framework of the system;

[0023] Figure 2 It is the Spatial Remote associative and Temporal Densely connected GraphConvolutional Network (SRTD-GCN) structure diagram;

[0024] Figure 3 This is a schematic diagram of the convolution kernels of the Spatial Remote Configuration Graph Convolutional Network (SRCP-GCN);

[0025] Figure 4 This is a schematic diagram of the internal structure of the first SE module;

[0026] Figure 5 This is a schematic diagram of the internal structure of the transition layer;

[0027] Figure 6 This is a diagram of the Residual Transformer Module-based Hybrid Convolutional Network (RT-HCN) architecture;

[0028] Figure 7 This is a schematic diagram of the internal structure of the first TSM module. Detailed Implementation

[0029] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] Example 1

[0031] This embodiment provides a method for recognizing distracted driving behavior based on an appearance and pose fusion network;

[0032] Distracted driving behavior recognition methods based on appearance pose fusion networks include:

[0033] S101: Acquire video data of the driving behavior to be identified;

[0034] S102: Input the video data of the driving behavior to be identified into the trained appearance pose fusion network to obtain the driving behavior recognition result;

[0035] The trained appearance pose fusion network is used for:

[0036] The driving behavior video data is processed based on the human pose estimation algorithm to obtain the joint coordinates of the driver's upper body, and the skeletal data is obtained based on the joint coordinates.

[0037] The joint coordinates are input into the first graph convolutional network to obtain joint features;

[0038] The skeletal data is input into the second convolutional network to obtain skeletal features;

[0039] The driving behavior video data is input into a hybrid convolutional network to obtain appearance features;

[0040] The joint features and skeletal features are fused to obtain the first-level fused features; the first-level fused features are fused with the appearance features to obtain the second-level fused features.

[0041] The secondary fusion features are classified to obtain the driving behavior recognition results.

[0042] Furthermore, the training process of the trained appearance pose fusion network includes:

[0043] Construct a training set, which consists of driving behavior data with known driving behavior recognition labels;

[0044] The training set is input into the appearance and pose fusion network to train the network. Training is stopped when the total loss function value of the network no longer decreases, or when the number of iterations reaches a set number, and the trained appearance and pose fusion network is obtained.

[0045] Furthermore, the trained appearance pose fusion network includes:

[0046] First graph convolutional network, second graph convolutional network, and hybrid convolutional network;

[0047] The output terminals of the first and second convolutional networks are both connected to the input terminal of the first cascaded splicing unit.

[0048] The output of the hybrid convolutional network and the output of the first cascaded splicing unit are both connected to the input of the second cascaded splicing unit;

[0049] The output of the second cascaded splicing unit is connected to the input of the classifier, and the output of the classifier outputs the final classification label.

[0050] Furthermore, such as Figure 2 As shown, the internal structure of the first graph convolutional network and the second graph convolutional network is the same. The first graph convolutional network includes:

[0051] The system consists of a batch normalization layer (BN), a first spatiotemporal graph convolutional module, a second spatiotemporal graph convolutional module, a third spatiotemporal graph convolutional module, a fourth spatiotemporal graph convolutional module, a fifth spatiotemporal graph convolutional module, a sixth spatiotemporal graph convolutional module, a seventh spatiotemporal graph convolutional module, an eighth spatiotemporal graph convolutional module, a ninth spatiotemporal graph convolutional module, a tenth spatiotemporal graph convolutional module, a max pooling layer, a fully connected layer, and an activation function layer.

[0052] Furthermore, the first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, and tenth spatiotemporal graph convolution modules have the same internal structure. The first spatiotemporal graph convolution module includes:

[0053] The input end of the first spatiotemporal graph convolution module, the SRCP-GCN spatial domain module, the first time domain processing module, the second time domain processing module, and the third time domain processing module are connected in sequence.

[0054] The first time-domain processing module includes: a first batch of normalization layers, a first activation function layer ReLU, a first convolutional layer, a second batch of normalization layers, a second activation function layer ReLU, a second convolutional layer, and a first SE module connected in sequence;

[0055] The second time-domain processing module includes: a third batch normalization layer, a third activation function layer ReLU, a third convolutional layer, a fourth batch normalization layer, a fourth activation function layer ReLU, a fourth convolutional layer, and a second SE module connected in sequence.

[0056] The third time-domain processing module includes: a fifth batch normalization layer, a fifth activation function layer ReLU, a fifth convolutional layer, a sixth batch normalization layer, a sixth activation function layer ReLU, a sixth convolutional layer, and a third SE module connected in sequence;

[0057] The output of the third SE module is connected to the input of the transition layer, the output of the transition layer is connected to the input of the seventh batch normalization layer, and the output of the seventh batch normalization layer is connected to the input of the fourth SE module.

[0058] The output of the SRCP-GCN spatial domain module is connected to the input of the third batch of normalization layers, the input of the fifth batch of normalization layers, and the input of the transition layer, respectively.

[0059] The output of the first SE module is connected to the input of the fifth batch of normalization layer and the input of the transition layer, respectively.

[0060] The output of the second SE module is connected to the input of the transition layer.

[0061] The input data of the first spatiotemporal graph convolution module and the output value of the fourth SE module are concatenated along the channel dimension, and the concatenated value is used as the output value of the first spatiotemporal graph convolution module.

[0062] Furthermore, the input of the first spatiotemporal graph convolution module is a trainable mask, which updates parameters through backpropagation and adaptively learns node weights by directly multiplying with the adjacency matrix.

[0063] Furthermore, such as Figure 5 As shown, the transition layer includes: a batch normalization layer, a ReLU activation function layer, a two-dimensional convolutional layer, and an average pooling layer connected in sequence.

[0064] Furthermore, such as Figure 5 As shown, the transition layer is used to reduce the number of feature channels and the scale of feature maps, thereby reducing the complexity of the model.

[0065] Furthermore, the SRCP-GCN spatial domain module refers to: treating each joint as a root node, and capturing the structure of motion connections between distant body parts by modeling the node information of the second-order neighborhood where the root node is located.

[0066] Furthermore, such as Figure 3As shown, the SRCP-GCN spatial domain module captures motion connections between distant body parts by modeling the node information of the second-order neighborhood where the root node is located. It achieves weighted aggregation of the node information of the second-order neighborhood and enhances the network model's perception of overall movement by modeling the relationship between distant nodes.

[0067] Furthermore, the SRCP-GCN spatial domain module defines the internal connections of a single frame skeleton using an adjacency matrix A and an identity matrix I representing self-connections. The SRCP-GCN spatial domain module is represented as follows:

[0068]

[0069] Where W represents the weighting function, l ti (v tj () represents the mapping function in the weighting function, used to project nodes within the root node's neighborhood to their corresponding weight subsets. This represents the degree matrix of the adjacency matrix (A+I). This represents the partitioning of the adjacency matrix according to a partitioning strategy, where A0 = I and A1 + A2 + A3 + A4 + A5 = A, where A represents the adjacency matrix, I represents the self-connection of key nodes in the adjacency matrix, and f in f represents the input features. out Indicates output features, Represents symmetric normalization, A j This represents the adjacency matrix after partitioning using the partitioning strategy. W represents symmetric normalization. j Let A1 represent the weight function, where j takes values ​​that are traversed through each node. Let A1 represent the adjacency matrix of the centripetal set, A2 represent the adjacency matrix of the centrifugal set, A3 represent the adjacency matrix of the far centripetal set, A4 represent the adjacency matrix of the far centripetal set, and A5 represent the adjacency matrix of the symmetric set.

[0070] Furthermore, the SRCP-GCN spatial domain module is used to expand the first-order node neighborhood of the root node in the graph convolution kernel in the spatial domain, and improve the partitioning of the expanded second-order neighborhood nodes through a spatial remote configuration partitioning strategy, assigning different weights to each type of node set.

[0071] Furthermore, such as Figure 3 As shown, the expansion of the first-order node neighborhood of the root node within the graph convolution kernel includes not only the first-order node neighborhood of the root node, but also the second-order node neighborhood of the root node.

[0072] Furthermore, the spatial remote configuration partitioning strategy is used to improve the partitioning of the extended second-order neighborhood nodes. Specifically, this includes proposing a spatial remote configuration partitioning strategy that includes a total of six types of node representations.

[0073] After expanding the first-order neighborhood nodes of the root node, the partitioning of its expanded second-order neighborhood nodes is improved, including a total of six types of node representations: (1) root node; (2) centripetal set; (3) centrifugal set; (4) far centripetal set; (5) far centripetal set; (6) symmetric set. Independent weights are assigned to nodes within the neighborhood by mapping them to the corresponding subsets.

[0074] Furthermore, the partitioning strategy includes: six types of node representations:

[0075] 1) Root node;

[0076] 2) Centripetal set: The set of first-order neighboring nodes that are closer to the centroid of the skeleton than the root node;

[0077] 3) Centrifugal set: The set of first-order neighboring nodes that are farther from the centroid of the skeleton than the root node;

[0078] 4) Far centroid set: The set of second-order neighbor nodes that are closer to the centroid of the skeleton than the root node;

[0079] 5) Farthest from the centroid: The set of second-order neighbor nodes that are farther from the centroid of the skeleton than the root node;

[0080] 6) Symmetric set: The symmetric relationship predefined by the root node.

[0081] Furthermore, the mapping function l ti (v tj The mathematical expression for ) is as follows:

[0082]

[0083] in, Used to describe node v in the upper body structure of the driver i With node v j A predefined symmetry relationship between them. j r represents the average distance from the centroid to joint j across all frames in the training set. i d(v) represents the average distance from the centroid to joint i across all frames in the training set. j ,v i () represents the distance between nodes i and j. This represents the predefined mapping relationship of joint i (referring to the symmetry relationship between the left and right parts of the body).

[0084] The root node is Figure 3 Center 0, the centripetal set is Figure 3 No. 1 in the middle, centrifugal set is Figure 3 Zhong No. 2, Yuanxiangxinji is Figure 3 No. 3 in the middle, far from the heart is Figure 3 The symmetric set of number 4 is Figure 3 In point 5, if a partitioning strategy is not used, all joints will be assigned the same weight, making it impossible to distinguish similar actions.

[0085] Figure 3 This is a schematic diagram of the convolutional kernels of the Spatial Remote Configuration Partition Graph Convolutional Network (SRCP-GCN). The numbers represent the labels of the six types of node sets divided according to the partitioning strategy, corresponding to the numerical representation in the mapping function formula.

[0086] Furthermore, assigning different weights to each type of node set specifically includes: the network model learning the weight parameters corresponding to each subset based on each action.

[0087] Furthermore, the functions of the first time domain processing module, the second time domain processing module, and the third time domain processing module are identical. The first time domain processing module is used for:

[0088] In the time domain, cascaded temporal convolution kernels are used to extract multi-scale temporal information.

[0089] By employing dense connections between temporal convolution kernels, it becomes easier for temporal information of different scales to flow between convolution modules end-to-end.

[0090] In the temporal domain, a concatenated temporal convolutional kernel approach is used to improve the extraction of multi-scale temporal information. However, while simply increasing the number of network layers can obtain more temporal features, it fails to fully utilize the multi-scale information expressed by different layers within the network. Therefore, a multi-scale densely connected temporal convolutional network is designed. By employing dense connections between temporal convolutional kernels, the flow of temporal information at different scales between convolutional modules becomes more convenient, improving the reusability of shallow features.

[0091] In dense blocks, any hidden layer is superimposed on all preceding hidden layers along the channel dimension, thus serving as the input to the next layer. According to... Figure 2 The display shows that x0 represents the original time characteristics of the input, and x1 is the time characteristics after processing by the nonlinear combination function H. (1) The resulting feature map is then concatenated with x0 and x1 along the channel dimension and used together as the next hidden layer H. (2) Input.

[0092] Similarly, a multi-scale densely connected temporal convolutional network yields feature maps x3:

[0093] x3=H (3) ([x0,x1,x2])

[0094] Where x2 represents the feature map, H3 represents the nonlinear combination function, and x3 represents the feature map.

[0095] [x0,x1,x2] represents the concatenation of feature maps generated by each layer, and the nonlinear combination function H (x) It consists of a 1×1 convolution kernel and a standard one-dimensional convolution. The former can reduce the number of parameters in the network, so that feature extraction can be performed more effectively after dimensionality reduction, while the latter is used to process temporal features at different scales.

[0096] Nonlinear combination function H (1) The specific working process is implemented through the first time domain processing module;

[0097] Nonlinear combination function H (2) The specific working process is implemented through the second time domain processing module;

[0098] Nonlinear combination function H (3) The specific working process is implemented through the third time domain processing module.

[0099] Furthermore, such as Figure 4 As shown, the internal structures of the first SE module, the second SE module, the third SE module, and the fourth SE module are identical. The first SE module includes:

[0100] The system comprises a compression operation unit, an excitation operation unit, and a scaling operation unit connected in sequence. The excitation operation unit includes a first fully connected layer, a first activation function layer, a second fully connected layer, and a second activation function layer connected in sequence. The input terminal of the compression operation unit is used to input a feature map. The feature map is also input into the scaling operation unit.

[0101] Furthermore, the first SE module includes: a compression operation unit (Squeeze), an excitation operation unit (Excitation), and a scaling operation unit (Scale).

[0102] The compression operation unit refers to the process of compressing the feature map using global average pooling (GAP) to give it a global receptive field.

[0103] The compression operation unit's calculation process is described below:

[0104]

[0105] In the formula, V and T represent the number of skeletal joints of the driver and the number of keyframes in the skeletal sequence in the current video segment, respectively, u c U represents the c-th channel in the feature map to be compressed. c(i,j) represents the feature value at position (i,j) of this channel. After compression, the dimension of the feature map is compressed from T×V×C to 1×1×C.

[0106] Next, the activation operation unit extracts channel-wise weight information. The first fully connected layer reduces the dimensionality of the average pooling vector, while the second fully connected layer restores it to the original dimensionality. The number of channels after dimensionality reduction is determined by the hyperparameter r. The activation operation unit enhances the nonlinear fitting capability of the network model and significantly reduces model complexity.

[0107] The excitation operation unit and its calculation process are described below:

[0108] s = F Excitation (z,W)=σ(W2δ(W1z));

[0109] Here, W1 and W2 are two fully connected operations, δ is the ReLU activation function, and σ is the Sigmoid activation function.

[0110] Finally, the scaling operation unit multiplies the learned activation values ​​of each channel by the original features on the feature map to obtain the weighted attention-enhanced feature map. This completes the feature map u c The recalibration process is described below. The calculation process for the scaling operation unit is as follows:

[0111]

[0112] Furthermore, the functions of the first SE module, the second SE module, the third SE module, and the fourth SE module are consistent. The first SE module is used to recalibrate the extracted features, thereby capturing the most significant temporal information in the skeleton action sequence.

[0113] It should be understood that while achieving low-level feature integration and reuse, important information in complex channels may not be extracted. Therefore, by incorporating a channel attention mechanism, information-rich features are selectively enhanced while useless features are suppressed, strengthening the network's filtering ability for channel information. Given the superior performance of the previous channel attention mechanism SENet in modeling inter-channel dependencies and its plug-and-play advantages, this invention introduces an SENet module after each temporal convolutional layer and transition layer to achieve feature recalibration, thereby capturing the most significant temporal information in the skeleton action sequence.

[0114] Furthermore, such as Figure 5As shown, the transition layer includes a batch normalization layer, an activation layer, a convolutional layer, and a pooling layer connected in sequence, which are used to reduce the number of feature channels and the scale of feature maps, thereby reducing the complexity of the model.

[0115] Furthermore, the first spatiotemporal graph convolution module is used to model the spatiotemporal information of the limb skeleton sequence.

[0116] Furthermore, the hybrid convolutional network includes: a first three-dimensional convolutional module, a second three-dimensional convolutional module, a first time-shifting module, a first two-dimensional convolutional module, a second time-shifting module, a second two-dimensional convolutional module, a third two-dimensional convolutional module, a max pooling layer, a fully connected layer, and a Softmax activation function layer.

[0117] Furthermore, the internal structure of the first 3D convolutional module is identical to that of the second 3D convolutional module. The first 3D convolutional module includes:

[0118] The first TSM module, the first two-dimensional convolutional layer, the sixth batch normalization layer, the seventh activation function layer ReLU, the second TSM module, the second two-dimensional convolutional layer, the seventh batch normalization layer, and the eighth activation function layer ReLU are connected in sequence.

[0119] The input of the first TSM module is connected to the input of the first three-dimensional convolutional layer, the output of the first three-dimensional convolutional layer is connected to the input of the eighth batch normalization layer, and the output of the eighth batch normalization layer is connected to the input of the eighth activation function layer ReLU.

[0120] Furthermore, such as Figure 7 As shown, the internal structure of the first TSM module and the second TSM module is the same. The first TSM module moves several channels of the input features along the time dimension.

[0121] This allows the spatial semantic information of adjacent frames to be fused into the current frame without consuming additional computing resources, thereby making up for the difficulty in modeling deep abstract features in hybrid convolutional networks.

[0122] The first TSM module moves partial channel information of the input features at each time step along the time dimension, fusing spatial semantic information from adjacent frames into the current frame without consuming additional computing resources. A schematic diagram of TSM is shown below. Figure 7 As provided in [the document / reference], TSM stands for Temporal Shift Module.

[0123] It should be understood that the difference between the two-dimensional convolutional layer and the three-dimensional convolutional layer lies in the lack of modeling of temporal information.

[0124] Furthermore, the working process of the first time displacement module and the second time displacement module is the same. The first time displacement module is used to perform displacement operation of 1 / 8 channel of the feature map along the time dimension between the previous frame and the next frame; that is, to move 1 / 8 channel of the feature map of the previous frame along the time dimension to the next frame, and to move 1 / 8 channel of the feature map of the next frame along the time dimension to the previous frame.

[0125] Therefore, the feature map at the current moment contains spatial semantic information from adjacent frames, which can be regarded as an expression of the temporal characteristics in the video. Furthermore, since the time-shifting module only involves the movement of some channel information, this method does not incur additional computational costs.

[0126] For example, taking a one-dimensional convolution with a kernel size of 3 as an example, assuming the convolution weights W = (w1, w2, w3) and the input vector of the one-dimensional feature is X, then the convolution operator Y can be written as:

[0127] Y = Conv(W,X) = w1X -1 +w2X 0 +w3X +1

[0128] Among them, X -1 This indicates that a portion of the channel information of X has been shifted in the time dimension by the previous frame; X 0 This indicates that no shift operation of channel information in the time dimension is performed on X; X +1 This indicates that some channel information of X has been shifted in the time dimension by the next frame. To address this, this invention adds a time shifting module before the two-dimensional convolution of the residual mapping branch, based on the Mixed Convolutional Network (MC), while retaining the complete information of the original spatial features in the identity mapping branch.

[0129] Furthermore, the internal structure of the first two-dimensional convolutional module is identical to that of the second two-dimensional convolutional module. The first two-dimensional convolutional module includes:

[0130] The third two-dimensional convolutional layer, the ninth batch of normalized layers, the ninth activation function layer ReLU, the fourth two-dimensional convolutional layer, the tenth batch of normalized layers, and the tenth activation function layer ReLU are connected in sequence.

[0131] The input of the third two-dimensional convolutional layer is also connected to the input of the tenth activation function layer, ReLU.

[0132] Furthermore, the third two-dimensional convolutional module includes:

[0133] The layers are connected in sequence: the first multi-head self-attention mechanism layer, the eleventh batch normalization layer, the eleventh activation function layer ReLU, the second multi-head self-attention mechanism layer, the twelfth batch normalization layer, and the twelfth activation function layer ReLU.

[0134] The input of the first multi-head self-attention mechanism layer is also connected to the input of the twelfth activation function layer ReLU.

[0135] Understandably, after extracting abstract information from the low-level features using the hybrid convolutional network MC3, the size of the input feature X can be represented as C×T×H×W, where T, H, and W represent the temporal and spatial dimensions of the input feature matrix, respectively, and C represents the dimension of a single token. Then, a linear transformation is performed on the input feature X to obtain the query vector Q, the key vector K, and the value vector V.

[0136] In multi-head self-attention mechanisms, to enable the model to focus on different locations and capture relevant information in different subspace representations, the Q, K, and V matrix vectors need to undergo different linear transformations to generate multiple prediction heads. Then, the self-attention mechanism is calculated for each prediction head in a different subspace. The calculation formula for multi-head self-attention is as follows:

[0137] MultiHead(Q,K,V)=Concat(head1,head2,...,head k W0

[0138] The prediction result of the multi-head self-attention mechanism is obtained by concatenating the self-attention matrices of each prediction head. The calculation process of the i-th prediction head is represented as follows:

[0139]

[0140] The inputs to the i-th prediction head are Q, K, and V, which are processed by the weight matrix. The generated linear change representation, The scaling factor is used to prevent the dot product from becoming too large. It can be seen that the self-attention mechanism calculates the similarity between the query vector and all key vectors to obtain global information, performs scaling, then uses the Softmax function to obtain the weights of the value vectors, and finally multiplies the value vectors and weights to obtain the attention score.

[0141] Furthermore, since multi-head self-attention mechanisms can perform parallel computations, positional encoding is needed to compensate for the lack of positional relationships. The positional encoding in the spatiotemporal domain is represented as follows, where R... h R w R tThese represent the positional encodings at different locations along the height, width, and time dimensions, respectively. Here, the three learnable parameter vectors are randomly initialized, then summed using a broadcast mechanism and integrated into the self-attention mechanism's computation.

[0142]

[0143] Furthermore, such as Figure 3 As shown, the hybrid convolutional network is used to extract the appearance features of driving behavior.

[0144] It should be understood that the hybrid convolutional network MC3 was selected as the baseline. The first two layers of this network are three-dimensional convolutions, and the last three layers are standard two-dimensional convolutional structures. Finally, after pooling operations, the scores for each category are calculated through fully connected layers and softmax layers.

[0145] Before the two-dimensional convolutions in the last two layers of MC3, a time shifting module is added. By shifting some channels of the feature map along the time dimension between the previous and next frames, the feature map at the current moment will contain the spatial semantic information of adjacent frames, which can be regarded as the expression of the temporal characteristics in the video.

[0146] Considering that Transformer inherently has the ability to model global information, the two-dimensional convolution in the last residual structure of MC3 is replaced by a 3D multi-head self-attention mechanism. By fusing the attention mechanism through shallow networks, global dependencies can be modeled while saving memory overhead.

[0147] It should be understood that 3D convolutional neural networks suffer from large model sizes in practical applications. Therefore, some researchers have proposed a hybrid convolutional network, MC3, combining 3D and 2D convolutions for learning spatiotemporal features. Regarding the network structure, the authors believe that motion modeling is necessary in the early layers, but may be negligible at higher levels of semantic abstraction. Therefore, MC3 only designs the first two layers as 3D convolutions, while the last three layers retain the standard 2D convolutional structure. Finally, after pooling, scores for each category are calculated via fully connected layers and a softmax layer. The hybrid convolutional network structure based on residual Transformer constructed in this invention is as follows: Figure 3 As shown.

[0148] While MC3 emphasizes the importance of early temporal modeling, this invention argues that the approach of abandoning the extraction of high-level temporal information to reduce the computational cost of the network model requires further optimization. This is because although the temporal representation in later layers is more abstract, it still contains extremely rich motion information, and how to integrate these resources with lower computational cost is a pressing issue. Therefore, this invention introduces a time displacement module to model the high-level representation of temporal information.

[0149] After capturing video data of the driver through a camera, the data is sent to... Figure 1 The three branches shown are used to learn features of different input types. Then, the Softmax scores predicted by these two branches are weighted and fused 1:1 using channel-dimensional feature fusion to obtain the final prediction result. The first and second branches are designed for the driver's skeletal sequence data. First, the lightweight human pose estimation algorithm OpenPose is used to extract the joint coordinates of the driver's upper body, and from this, skeletal data is calculated. Then, the joint coordinates and skeletal data are respectively input into a spatiotemporal graph convolutional network for extracting spatial structure information and learning temporal dependencies. The third branch directly feeds the video data acquired by the camera into a hybrid convolutional network to learn the appearance features of joint human and object movements.

[0150] The key technologies are reflected in two aspects: the spatially distant association and temporally dense connection graph convolutional network (SRTD-GCN) for skeleton sequence recognition and the residual Transformer-based hybrid convolutional network (RT-HCN) for RGB video recognition.

[0151] Furthermore, joint features and skeletal features are fused to obtain primary fusion features; primary fusion features are fused with appearance features to obtain secondary fusion features, both of which are fused using a weighted fusion method.

[0152] Furthermore, for constructing the training set, this invention selected the ASU dataset. The ASU dataset, proposed by Mohamed et al. from Ain Shams University, is a dataset of distracted driving behavior under low-light driving conditions. This dataset includes both daytime and nighttime driving scenarios and was collected using advanced NoIR technology. In terms of scale, this dataset invited 70 drivers to collect data in real-world in-vehicle environments, resulting in a total of 52,350 video frames. Compared to publicly available datasets such as StateFarm, it not only surpasses them in scale but also offers the advantage of temporal continuity. Since this dataset provides continuous frame data for 10 categories of distracted driving, data preprocessing is required.

[0153] First, consecutive frames from the ASU dataset need to be merged into video segments for the OpenPose human pose estimation algorithm to extract human pose information. This extracted data can then be used in the improved model of this invention for recognition and classification tasks. It's important to note that, to increase the difficulty of this task, this invention segments each complete action video segment, ensuring that each segment contains only a portion of the distracted driving behavior, and that the segmentation does not affect the classification of the skeleton data. Then, this invention performs data augmentation by mirroring each video segment horizontally. Finally, the newly constructed distracted driving dataset is used for driver pose extraction. The driver's upper body pose information extracted by OpenPose consists of the horizontal and vertical coordinates and confidence scores of 14 joints.

[0154] Further, the training phase. Regarding the experimental setup, the experiment was conducted using Python as the programming language on an Ubuntu 20.04 operating system with PyCharm installed, employing the PyTorch deep learning framework. Hardware-wise, the CPU was an Intel Xeon Gold 6226R with 16 cores and a clock speed of 2.90GHz. The GPU was an NVIDIA GeForce RTX 3090, and it had 24GB of RAM.

[0155] In terms of parameters, the training process of SRTD-GCN consists of 50 iterations and is optimized using stochastic gradient descent. The initial learning rate is set to 0.1 and decays by a factor of 0.1 in the 30th and 40th iterations. The batch size is set to 32 and the dropout size is set to 0. Finally, the probability results of 10 categories are output through fully connected layers.

[0156] like Figure 6 The parameters of RT-HCN are set as follows: Due to the relatively small size of the ASU dataset, to alleviate the difficulty in optimizing 3D convolutions, this invention first pre-trained the hybrid convolutional network on the NTU-RGB+D dataset and saved the training parameters for initialization of model training on the ASU dataset. The entire training process on the ASU dataset consisted of 100 iterations, and the Adam optimizer was used to automatically adjust the learning rate. The initial learning rate was set to 0.001, the batch size was set to 32, the dropout size was set to 0, and finally, the probability results for 10 categories were output.

[0157] To ensure data source independence, two main branches are trained separately for the skeleton sequence and RGB video, ensuring model independence. Then, the two branches are combined using a late-stage fusion method, and the predicted Softmax scores are weighted and fused to obtain the final prediction result. According to... Figure 1 The first main branch is designed to process driver skeleton data. It first uses the lightweight human pose estimation algorithm OpenPose to extract the joint coordinates of the driver's upper body, obtaining second-order skeletal information. Then, the joint coordinates are input into the first sub-branch SRTD-GCN for learning spatial structure features and temporal dependencies. The second-order skeletal information is then input into the second sub-branch SRTD-GCN for learning spatial structure features and temporal dependencies. The second main branch directly inputs video data into RT-HCN to learn the appearance features of combined human and object movements. These two branches complement each other, comprehensively utilizing multimodal input features to effectively improve the accuracy of identifying distracted driving behavior.

[0158] Example 2

[0159] This embodiment provides a distracted driving behavior recognition system based on an appearance and posture fusion network;

[0160] A distracted driving behavior recognition system based on appearance and posture fusion networks includes:

[0161] The acquisition module is configured to acquire video data of the driving behavior to be identified.

[0162] The recognition module is configured to input the video data of the driving behavior to be recognized into the trained appearance pose fusion network to obtain the driving behavior recognition result.

[0163] The trained appearance and pose fusion network is used to: process driving behavior video data based on a human pose estimation algorithm to obtain the joint coordinates of the driver's upper body, and obtain skeletal data based on the joint coordinates; input the joint coordinates into a first-graph convolutional network to obtain joint features; input the skeletal data into a second-graph convolutional network to obtain skeletal features; input the driving behavior video data into a hybrid convolutional network to obtain appearance features; fuse the joint features and skeletal features to obtain a first-level fusion feature; fuse the first-level fusion feature with the appearance feature to obtain a second-level fusion feature; and classify the second-level fusion feature to obtain the driving behavior recognition result.

[0164] It should be noted that the acquisition module and the identification module mentioned above correspond to steps S101 to S102 in Embodiment 1. The examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0165] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0166] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0167] Example 3

[0168] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0169] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0170] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0171] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0172] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0173] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0174] Example 4

[0175] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distracted driving behavior recognition method based on appearance pose fusion network, characterized by: include: Acquire video data of the driving behavior to be identified; The video data of the driving behavior to be identified is input into the trained appearance and pose fusion network to obtain the driving behavior recognition result. The trained appearance and pose fusion network is used for: processing driving behavior video data based on a human pose estimation algorithm to obtain the joint coordinates of the driver's upper body, and obtaining skeletal data based on the joint coordinates; inputting the joint coordinates into a first-image convolutional network to obtain joint features; inputting the skeletal data into a second-image convolutional network to obtain skeletal features; inputting the driving behavior video data into a hybrid convolutional network to obtain appearance features; fusing the joint features and skeletal features to obtain a first-level fusion feature; fusing the first-level fusion feature with the appearance feature to obtain a second-level fusion feature; and classifying the second-level fusion feature to obtain the driving behavior recognition result. The trained appearance pose fusion network includes: First graph convolutional network, second graph convolutional network, and hybrid convolutional network; The first graph convolutional network includes: The system consists of a batch normalization layer (BN), a first spatiotemporal graph convolutional module, a second spatiotemporal graph convolutional module, a third spatiotemporal graph convolutional module, a fourth spatiotemporal graph convolutional module, a fifth spatiotemporal graph convolutional module, a sixth spatiotemporal graph convolutional module, a seventh spatiotemporal graph convolutional module, an eighth spatiotemporal graph convolutional module, a ninth spatiotemporal graph convolutional module, a tenth spatiotemporal graph convolutional module, a max pooling layer, a fully connected layer, and an activation function layer. The first spatiotemporal graph convolution module includes: an input terminal of the first spatiotemporal graph convolution module, an SRCP-GCN spatial domain module, a first temporal domain processing module, a second temporal domain processing module, and a third temporal domain processing module connected in sequence. The SRCP-GCN spatial domain module connects the internal links of a single frame skeleton using an adjacency matrix. And represents the self-connected identity matrix The SRCP-GCN spatial domain module is defined as follows: ； in, Represents the weighting function. Representing the adjacency matrix The degree matrix, This indicates the partitioning of the adjacency matrix based on a partitioning strategy. and , Represents the adjacency matrix. This indicates a self-connection of key nodes in the adjacency matrix. Indicates input features, Indicates output features, This indicates symmetric normalization. This represents the adjacency matrix after partitioning using the partitioning strategy. Represents the weighting function. The range of values ​​for is determined by traversing each node. The adjacency matrix of the centripetal set is represented by the following matrix. Denotes the adjacency matrix of the centrifugal set. Denotes the adjacency matrix of the far centroid set. Denotes the adjacency matrix of the sets far from the heart set. Represents the adjacency matrix of a symmetric set.

2. The distracted driving behavior recognition method based on appearance pose fusion network as described in claim 1, characterized in that, The output terminals of the first graph convolutional network and the second graph convolutional network are both connected to the input terminal of the first cascaded splicing unit; The output of the hybrid convolutional network and the output of the first cascaded splicing unit are both connected to the input of the second cascaded splicing unit. The output of the second cascaded splicing unit is connected to the input of the classifier, and the output of the classifier outputs the final classification label.

3. The distracted driving behavior recognition method based on appearance pose fusion network as described in claim 2, characterized in that, The first time-domain processing module includes: a first batch of normalization layers, a first activation function layer ReLU, a first convolutional layer, a second batch of normalization layers, a second activation function layer ReLU, a second convolutional layer, and a first SE module connected in sequence; The second time-domain processing module includes: a third batch normalization layer, a third activation function layer ReLU, a third convolutional layer, a fourth batch normalization layer, a fourth activation function layer ReLU, a fourth convolutional layer, and a second SE module connected in sequence. The third time-domain processing module includes: a fifth batch normalization layer, a fifth activation function layer ReLU, a fifth convolutional layer, a sixth batch normalization layer, a sixth activation function layer ReLU, a sixth convolutional layer, and a third SE module connected in sequence.

4. The distracted driving behavior recognition method based on appearance and pose fusion network as described in claim 3, characterized in that, The output of the third SE module is connected to the input of the transition layer, the output of the transition layer is connected to the input of the seventh batch normalization layer, and the output of the seventh batch normalization layer is connected to the input of the fourth SE module. The output of the SRCP-GCN spatial domain module is connected to the input of the third batch of normalization layers, the input of the fifth batch of normalization layers, and the input of the transition layer, respectively. The output of the first SE module is connected to the input of the fifth batch of normalization layer and the input of the transition layer, respectively. The output of the second SE module is connected to the input of the transition layer. The input data of the first spatiotemporal graph convolution module and the output value of the fourth SE module are concatenated along the channel dimension, and the concatenated value is used as the output value of the first spatiotemporal graph convolution module.

5. The distracted driving behavior recognition method based on appearance pose fusion network as described in claim 3, characterized in that, The first SE module includes: a compression operation unit, an activation operation unit, and a scaling operation unit connected in sequence; the activation operation unit includes: a first fully connected layer, a first activation function layer, a second fully connected layer, and a second activation function layer connected in sequence; the input terminal of the compression operation unit is used to input a feature map; the feature map is also input into the scaling operation unit; the first SE module includes: a compression operation unit, an activation operation unit, and a scaling operation unit.

6. The distracted driving behavior recognition method based on appearance pose fusion network as described in claim 2, characterized in that, The hybrid convolutional network includes: The system consists of a first 3D convolutional module, a second 3D convolutional module, a first time-shifting module, a first 2D convolutional module, a second time-shifting module, a second 2D convolutional module, a third 2D convolutional module, a max pooling layer, a fully connected layer, and a Softmax activation function layer. The first three-dimensional convolutional module includes: a first TSM module, a first two-dimensional convolutional layer, a sixth batch normalization layer, a seventh activation function layer ReLU, a second TSM module, a second two-dimensional convolutional layer, a seventh batch normalization layer, and an eighth activation function layer ReLU, connected in sequence. The input of the first TSM module is connected to the input of the first three-dimensional convolutional layer, the output of the first three-dimensional convolutional layer is connected to the input of the eighth batch normalization layer, and the output of the eighth batch normalization layer is connected to the input of the eighth activation function layer ReLU. The first TSM module moves several channels of the input features along the time dimension; The first time displacement module is used to shift 1 / 8 channel of the feature map along the time dimension between the previous frame and the next frame.

7. The distracted driving behavior recognition method based on appearance and pose fusion network as described in claim 6, characterized in that, The first two-dimensional convolutional module includes: a third two-dimensional convolutional layer, a ninth batch normalization layer, a ninth activation function layer ReLU, a fourth two-dimensional convolutional layer, a tenth batch normalization layer, and a tenth activation function layer ReLU connected in sequence; wherein, the input end of the third two-dimensional convolutional layer is also connected to the input end of the tenth activation function layer ReLU; The third two-dimensional convolutional module includes: a first multi-head self-attention mechanism layer, an eleventh batch normalization layer, an eleventh activation function layer ReLU, a second multi-head self-attention mechanism layer, a twelfth batch normalization layer, and a twelfth activation function layer ReLU connected in sequence; wherein, the input end of the first multi-head self-attention mechanism layer is also connected to the input end of the twelfth activation function layer ReLU.

8. A distracted driving behavior recognition system based on appearance and posture fusion network, characterized in that, include: The acquisition module is configured to acquire video data of the driving behavior to be identified. The recognition module is configured to input the video data of the driving behavior to be recognized into the trained appearance pose fusion network to obtain the driving behavior recognition result. The trained appearance and pose fusion network is used for: processing driving behavior video data based on a human pose estimation algorithm to obtain the joint coordinates of the driver's upper body, and obtaining skeletal data based on the joint coordinates; inputting the joint coordinates into a first-image convolutional network to obtain joint features; inputting the skeletal data into a second-image convolutional network to obtain skeletal features; inputting the driving behavior video data into a hybrid convolutional network to obtain appearance features; fusing the joint features and skeletal features to obtain a first-level fusion feature; fusing the first-level fusion feature with the appearance feature to obtain a second-level fusion feature; and classifying the second-level fusion feature to obtain the driving behavior recognition result. The trained appearance pose fusion network includes: First graph convolutional network, second graph convolutional network, and hybrid convolutional network; The first graph convolutional network includes: The system consists of a batch normalization layer (BN), a first spatiotemporal graph convolutional module, a second spatiotemporal graph convolutional module, a third spatiotemporal graph convolutional module, a fourth spatiotemporal graph convolutional module, a fifth spatiotemporal graph convolutional module, a sixth spatiotemporal graph convolutional module, a seventh spatiotemporal graph convolutional module, an eighth spatiotemporal graph convolutional module, a ninth spatiotemporal graph convolutional module, a tenth spatiotemporal graph convolutional module, a max pooling layer, a fully connected layer, and an activation function layer. The first spatiotemporal graph convolution module includes: an input terminal of the first spatiotemporal graph convolution module, an SRCP-GCN spatial domain module, a first temporal domain processing module, a second temporal domain processing module, and a third temporal domain processing module connected in sequence. The SRCP-GCN spatial domain module connects the internal links of a single frame skeleton using an adjacency matrix. And represents the self-connected identity matrix The SRCP-GCN spatial domain module is defined as follows: ； in, Represents the weighting function. Representing the adjacency matrix The degree matrix, This indicates the partitioning of the adjacency matrix based on a partitioning strategy. and , Represents the adjacency matrix. This indicates a self-connection of key nodes in the adjacency matrix. Indicates input features, Indicates output features, This indicates symmetric normalization. This represents the adjacency matrix after partitioning using the partitioning strategy. Represents the weighting function. The range of values ​​for is determined by traversing each node. The adjacency matrix of the centripetal set is represented by the following matrix. Denotes the adjacency matrix of the centrifugal set. Denotes the adjacency matrix of the far centroid set. Denotes the adjacency matrix of the sets far from the heart set. Represents the adjacency matrix of a symmetric set.

9. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-7.

10. A storage medium, characterized in that, Non-transitory storage of computer-readable instructions, wherein, when executed by a computer, the instructions of the method according to any one of claims 1-7 are executed.

Images