Driver behavior recognition method, device and equipment and storage medium

Abstract

The invention provides a driver behavior recognition method and device, equipment and a storage medium, and belongs to the technical field of computer image recognition. The driver behavior recognition method comprises the following steps: acquiring a video frame sequence and a heat map frame sequence containing driver behaviors; generating a first feature map and a second feature map based on the video frame sequence and the heat map frame sequence by using a multi-modal interactive neural network and time weight adaptive convolution; the time weight adaptive convolution is used for endowing each frame with a corresponding time weight based on the position of each frame in the respective time sequence; the first feature map is a skeleton-guided appearance feature map based on time weighting, and the second feature map is a skeleton-guided appearance feature map based on time weighting; and performing driver behavior recognition based on the first feature map and the second feature map to obtain a recognition result. According to the invention, the multi-modal interaction and time weight adaptive mechanism is introduced, and the method focuses on the fine-grained behavior of the driver, so that the accuracy of the recognition result is effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of computer image recognition, and in particular to a driver behavior recognition method, device, equipment and storage medium. Background Art

[0002] As a core element in the development of advanced driver assistance systems (ADAS), driver behavior recognition technology plays a vital role in promoting the development of autonomous driving technology and improving driving safety.

[0003] However, existing recognition methods based on RGB frames, skeleton postures, or a simple combination of RGB frames and skeleton postures cannot effectively focus on the fine-grained behaviors of drivers, and their recognition accuracy still needs to be improved. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a driver behavior recognition method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the accuracy of identifying the fine-grained behavior of the driver.

[0005] In order to achieve the above-mentioned objectives, in a first aspect, an embodiment of the present disclosure provides a driver behavior recognition method, comprising: obtaining a video frame sequence and a heat map frame sequence containing driver behavior; using a multimodal interactive neural network and a time-weighted adaptive convolution to generate a first feature map and a second feature map based on the video frame sequence and the heat map frame sequence; wherein the time-weighted adaptive convolution is used to assign a corresponding time weight to each frame based on the position of each frame in the respective time series; the first feature map is an appearance feature map guided by a skeleton based on time weighting, and the second feature map is an appearance-guided skeleton feature map based on time weighting; driver behavior recognition is performed based on the first feature map and the second feature map to obtain a recognition result.

[0006] In some embodiments, the multimodal interactive neural network includes an asymmetric feature extraction module, an appearance-guided heat map enhancement module and a skeleton-guided attention module; wherein the asymmetric feature extraction module includes an appearance feature extraction neural network and a skeleton feature extraction neural network; using the multimodal interactive neural network and time-weighted adaptive convolution, based on the video frame sequence and the heat map frame sequence, generating a first feature map and a second feature map, including: using the appearance feature extraction neural network, the skeleton-guided attention module and the time-weighted adaptive convolution, based on the video frame sequence, generating a first feature map; using the skeleton feature extraction neural network, the appearance-guided heat map enhancement module and the time-weighted adaptive convolution, based on the heat map frame sequence, generating a second feature map; wherein the appearance-guided heat map enhancement module and the skeleton-guided attention module are used for feature interaction between the appearance feature extraction neural network and the skeleton feature extraction neural network.

[0007] In some embodiments, the appearance feature extraction neural network includes a convolution layer, a batch normalization layer, a linear rectification layer, a maximum pooling layer, a residual connection layer, a global pooling layer and a fully connected layer; the time weight adaptive convolution is set in each residual block of the residual connection layer of the appearance feature extraction neural network; the skeleton-guided attention module is set between any two residual blocks of the residual connection layer of the appearance feature extraction neural network; using the appearance feature extraction neural network, the skeleton-guided attention module and the time weight adaptive convolution, based on the video frame sequence, generating a first feature map, including: using the convolution layer, the batch normalization layer, the linear rectification layer and the maximum pooling layer, The invention relates to a method for extracting preliminary appearance features from a video frame sequence to generate a first appearance feature map; using multiple residual blocks of a residual connection layer, performing multiple deep appearance feature extractions on the first appearance feature map; using a skeleton-guided attention module between any two residual blocks, enhancing the appearance features of the first appearance feature map based on skeleton features, and generating a second appearance feature map; wherein, in the process of extracting deep appearance features from each residual block, using temporal weight adaptive convolution to assign corresponding temporal weights to each video frame for temporal weighting; using a global pooling layer and a fully connected layer, performing dimensionality reduction and feature fusion on the second appearance feature map to generate a first feature map.

[0008] In some embodiments, the skeleton feature extraction neural network includes a convolution layer, a batch normalization layer, a linear rectification layer, a residual connection layer, a global pooling layer and a fully connected layer; the time weight adaptive convolution is arranged in each residual block of the residual connection layer of the skeleton feature extraction neural network; the appearance-guided heat map enhancement module is arranged between any two residual blocks of the residual connection layer of the skeleton feature extraction neural network; using the skeleton feature extraction neural network, the appearance-guided heat map enhancement module and the time weight adaptive convolution, based on the heat map frame sequence, generating a second feature map, including: using the convolution layer, the batch normalization layer and the linear rectification layer to perform the heat map frame sequence A preliminary skeleton feature extraction is performed to generate a first skeleton feature map; multiple residual blocks of the residual connection layer are used to perform multiple deep skeleton feature extractions on the first skeleton feature map, and an appearance-guided heat map enhancement module is used between any two residual blocks to enhance the skeleton features of the first skeleton feature map based on the appearance features to generate a second skeleton feature map; wherein, in the process of deep skeleton feature extraction in each residual block, a temporal weight adaptive convolution is used to assign a corresponding temporal weight to each heat map frame for temporal weighting; a global pooling layer and a fully connected layer are used to perform dimensionality reduction and feature fusion on the second skeleton feature map to generate a second feature map.

[0009] In some embodiments, the temporal weight adaptive convolution decomposes the temporal weight of each frame in the spatial convolution into the product of the base weight shared by all frames and the calibration weight of each frame. The temporal weight is: t =W b *α t , where Wt is the weight of the tth frame, W b is the base weight shared by all frames, α t is the calibration weight of the tth frame.

[0010] In some embodiments, obtaining a video frame sequence and a heat map frame sequence containing driver behavior includes: obtaining video data containing driver behavior; generating a video frame sequence based on the video data; and using a heat map generation module to determine skeleton joint points in the video data based on the video data, and generating a heat map frame sequence based on the skeleton joint points.

[0011] In some embodiments, the multimodal interactive neural network includes a feature fusion module; driver behavior recognition is performed based on the first feature map and the second feature map to obtain a recognition result, including: using the feature fusion module to generate a comprehensive feature map based on the first feature map and the second feature map; driver behavior recognition is performed based on the comprehensive feature map to obtain a recognition result.

[0012] In the second aspect, an embodiment of the present disclosure provides a driver behavior recognition device, including: an acquisition unit, used to acquire a video frame sequence and a heat map frame sequence containing driver behavior; a feature map generation unit, used to generate a first feature map and a second feature map based on the video frame sequence and the heat map frame sequence by using a multimodal interactive neural network and a time weight adaptive convolution; wherein the time weight adaptive convolution is used to assign a corresponding time weight to each frame based on the position of each frame in the respective time series; the first feature map is an appearance feature map guided by a skeleton based on the time dimension, and the second feature map is an appearance guided by a skeleton based on the time dimension; and a recognition unit, used to perform driver behavior recognition based on the first feature map and the second feature map to obtain a recognition result.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory configured to store instructions; and a processor configured to call instructions from the memory and to implement the driver behavior identification method provided in the first aspect or any embodiment of the first aspect when executing the instructions.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the method for driver behavior identification provided in the first aspect or any embodiment of the first aspect.

[0015] Through the above technical solution, the driver behavior recognition method provided by the embodiment of the present disclosure can achieve high-precision recognition of driver behavior through the effective combination of multimodal interactive neural network and time-weighted adaptive convolution. This method can not only make full use of the information contained in the video frame sequence and the heat map frame sequence, but also highlight the key behavior moments by assigning appropriate time weights to the data at different time points, thereby improving the accuracy and robustness of recognition. In addition, by generating a skeleton-guided appearance feature map based on time weighting and an appearance-guided skeleton feature map, the ability to capture subtle changes in the driver's movements is further enhanced, ensuring the integrity and diversity of information in the recognition process, and ultimately achieving efficient and reliable driver behavior recognition, providing strong support for advanced driver assistance systems.

[0016] Other features and advantages of the embodiments of the present disclosure will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present disclosure but do not constitute a limitation on the embodiments of the present disclosure. In the accompanying drawings:

[0018] Figure 1 is a flowchart of a driver behavior identification method provided according to the first embodiment of the present disclosure;

[0019] Figure 2 is a flowchart of a driver behavior identification method provided according to the second embodiment of the present disclosure;

[0020] Figure 3 is a flowchart of a driver behavior identification method provided according to Embodiment 3 of the present disclosure;

[0021] Figure 3a is a flowchart of a driver behavior identification method provided according to Embodiment 3 of the present disclosure;

[0022] Figure 3b is a flowchart of a driver behavior identification method provided according to Embodiment 3 of the present disclosure;

[0023] Figure 4 is a flowchart of a driver behavior identification method provided according to a fourth embodiment of the present disclosure;

[0024] Figure 5 is a schematic diagram of the structure of a multimodal interactive neural network provided according to an embodiment of the present disclosure;

[0025] Figure 6 is a structural schematic diagram of an appearance-guided heat map enhancement module provided according to an embodiment of the present disclosure;

[0026] Figure 7 is a schematic diagram of the structure of a skeleton-guided attention module provided according to an embodiment of the present disclosure;

[0027] Figure 8 is a schematic diagram of a structure of a time weight adaptive convolution provided according to an embodiment of the present disclosure;

[0028] Fig. 9 It is a structural schematic diagram of a driver behavior recognition device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present disclosure, and are not used to limit the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0030] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution disclosed herein are in compliance with the relevant provisions of national laws and regulations. In the embodiments disclosed herein, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution disclosed herein, but it does not mean that the applicant has or will necessarily use the solution.

[0031] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present disclosure, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present disclosure.

[0032] Identifying driver behavior is a key element in the development of advanced driver assistance systems (ADAS), and has important applications in autonomous driving and driving safety monitoring. Driver behavior and decision-making are the main factors affecting driving safety, and using an accurate driver behavior monitoring system can reduce the accident rate.

[0033] Traditional driver behavior recognition methods are usually divided into two steps. The first step is to extract hand-crafted features from the raw data, and the second step is to feed the features into a classifier for classification. With the gradual development of deep learning models, there have been many studies on driver behavior recognition based on deep learning. It can be mainly divided into two types: RGB frame-based and skeleton posture-based. The RGB frame-based method usually uses some general action recognition models to extract RGB features. Although this method can use the information of the target and scene for recognition, it cannot pay attention to the subtle differences between different behaviors of the driver. The skeleton posture-based method focuses on modeling the key points of the driver's skeleton to identify the behavior, usually using a graph convolutional network. Since this method lacks RGB appearance information, it cannot pay attention to the difference between the interacting objects, resulting in poor results in distinguishing behaviors with similar postures but targeting different objects.

[0034] Furthermore, there is a method in the prior art that simply combines RGB frames and skeleton postures, but this method uses shared convolution weights and cannot pay close attention to the discriminative key frames of fine-grained behaviors, and its recognition accuracy still needs to be improved.

[0035] Based on this, a first embodiment of the present disclosure provides a driver behavior recognition method. Figure 1 It is a flowchart of a driver behavior identification method provided according to the first embodiment of the present disclosure.

[0036] like Figure 1 As shown, the driver behavior recognition method includes steps S11 to S13.

[0037] Step S11, obtaining a video frame sequence and a heat map frame sequence containing driver behavior.

[0038] In the driver behavior recognition system, obtaining video frame sequences and heat map frame sequences can be achieved in a variety of ways, including but not limited to: using a high-resolution camera installed inside the vehicle to capture the movements of the driver's face, hands and other key parts in real time; using an infrared camera to obtain clear images under low-light or no-light conditions; using a depth sensor to generate depth information of the driver's body parts to generate a heat map frame sequence; and using a multi-camera system to collect data from different angles simultaneously to provide a more comprehensive perspective and higher data quality.

[0039] Step S12: Generate a first feature map and a second feature map based on a video frame sequence and a heat map frame sequence by using a multimodal interactive neural network and a time-weighted adaptive convolution.

[0040] After obtaining the video frame sequence and heat map frame sequence containing the driver's behavior, the multimodal interactive neural network and time-weighted adaptive convolution are used to process these data to generate the first feature map and the second feature map. Specifically, the multimodal interactive neural network is used to fuse the multimodal information in the video frames and heat map frames, while the time-weighted adaptive convolution assigns corresponding time weights to each frame based on the position of each frame in the respective time series. The first feature map is a skeleton-guided appearance feature map based on time weighting, and the second feature map is a skeleton-guided appearance feature map based on time weighting. In this way, the model can more effectively capture and utilize the spatiotemporal information in the driver's behavior, thereby improving the accuracy and robustness of feature extraction.

[0041] Step S13: performing driver behavior recognition based on the first characteristic graph and the second characteristic graph to obtain a recognition result.

[0042] After generating the first feature map (appearance feature map guided by skeleton based on time weighting) and the second feature map (skeleton feature map guided by appearance based on time weighting), the next step is to use these two feature maps to identify the driver's behavior. This process usually involves inputting the first feature map and the second feature map into a pre-trained classifier, which can be a deep learning model such as convolutional neural network (CNN), recurrent neural network (RNN) or a combination of multiple models, aiming to extract more advanced behavior pattern information from the feature map. The classifier will analyze and judge the driver's behavior based on the time-weighted skeleton and appearance information contained in the feature map, and finally output the recognition result, such as whether the driver is in a normal driving state, whether he is using a mobile phone, whether he has signs of fatigue driving, etc.

[0043] The driver behavior recognition method provided in the first embodiment of the present disclosure focuses on the fine-grained behavior of the driver by introducing multimodal interaction and time weight adaptive mechanism, and effectively improves the accuracy of the recognition result. Specifically, the skeleton data and appearance information in the time series are fused through steps S11 and S12 to enhance the ability to capture the fine-grained behavior changes of the driver, and step S13 uses these enhanced feature maps for efficient recognition.

[0044] In some embodiments, the temporal weight adaptive convolution can decompose the temporal weight of each frame in the spatial convolution into the product of the base weight shared by all frames and the calibration weight of each frame. The temporal weight is: t =W b *α t , where W t is the weight of the tth frame, W b is the base weight shared by all frames, α tis the calibration weight of the tth frame. The temporal weight adaptive convolution calibrates the convolution weight of each frame according to the local and global temporal context to give the spatial convolution the temporal modeling capability.

[0045] First, the input features are pooled in the global space and a layer of 1D convolution. The output of the 1D convolution is modeled globally using the self-attention mechanism to obtain the frame description vector v representing the current frame information. The frame description vector v is added to the output of the 1D convolution, and then a layer of 1D convolution is used to obtain different calibration weights α for each frame containing temporal context information. t In order to enable the time-adaptive convolution of the present disclosure to effectively use the pre-trained weights, the base weights W shared by all frames are b Initialized as the initial weights of the 2D convolution, using W b and α t Perform weight calibration on input features.

[0046] For example, suppose there is a video frame sequence containing driver behavior, the video frame sequence is F1, F2, ... F 10 , where each F t Represents the image of the tth frame. t The time weight W t When all frames share the same base weight W b =0.8, where F5 is detected as the driver's hand approaching the phone, so the calibration weight of F5 is 1.5, and the calibration weights of other frames are 0.7, indicating that the importance of the image of the 5th frame is greater than the importance of the images of other frames. Taking the 5th frame and the 2nd frame as examples, after calculating the time weight, the time weight of the 5th frame is W5=0.8×1.5=1.2, and the time weight of the 2nd frame is W2=0.8×0.7=0.56.

[0047] By introducing calibration weights, the importance of each frame can be flexibly adjusted according to its specific content, so as to better capture the dynamic changes in time series data, which is particularly important for identifying the driver's behavior, because certain instantaneous actions may have a decisive influence on the overall behavior judgment.

[0048] In the disclosed embodiment, a video frame sequence and a heat map frame sequence can be obtained respectively according to the video data, and a heat map generation module can be used to generate a heat map frame sequence according to the video data. Among them, the heat map generation module can be a 3D heat map generation module, which can be set before the data is input into the multimodal interactive neural network, and can also be used as a component of the multimodal interactive neural network, so that the user does not need to process the video frame sequence and the heat map frame sequence in advance, and directly inputs the video data into the multimodal interactive neural network to output the recognition result of the driver's behavior. One of the situations is described below.

[0049] Figure 2 It is a flowchart of a driver behavior identification method provided according to the second embodiment of the present disclosure.

[0050] like Figure 2 As shown, in the driver behavior recognition method, obtaining a video frame sequence and a heat map frame sequence containing the driver behavior may include steps S21 to S23.

[0051] Step S21: Acquire video data containing driver behavior.

[0052] Step S22: Generate a video frame sequence based on the video data.

[0053] Step S23: using a heat map generation module to determine skeleton joint points in the video data based on the video data, and generating a heat map frame sequence based on the skeleton joint points.

[0054] In step S23, the 2D posture of the video is estimated using a posture estimator. Since the four joints of the driver's legs have limited recognition effects, the present disclosure extracts 13 skeleton joints of the upper limbs and hips to obtain a 2D skeleton. Next, the 2D skeleton is represented as a heat map of size K*H*W, where K is the number of joints, and H and W are the height and width of the heat map. The joint heat map is obtained by combining K Gaussian maps centered on each joint point, accumulating K Gaussian maps to obtain a complete heat map frame, and finally stacking all heat map frames along the time dimension to obtain a 3D heat map of size K*T*H*W.

[0055] For example, take the 3D heat map generation module as an example. The input of the 3D heat map generation module is the 2D coordinates of the 13 joint points of a frame image, and the 3D heat map is composed of the joint heat map and the limb heat map. When constructing the joint heat map, an 8×8 area centered on the joint point is constructed for each joint point, and the joint point weight is used as the Gaussian weight to calculate the Gaussian map in the 8×8 area. The Gaussian maps of the 8×8 areas of the 13 joint points are combined to obtain the Gaussian heat map of the joint points of a frame image. All the Gaussian heat maps of the joint points are combined along the time dimension T to obtain a 3D heat map of size 13×32×56×56. The expression of the joint point Gaussian map is:

[0056]

[0057] where c k is the joint point confidence, x k and kare the joint coordinates, σ is the variance of the Gaussian mapping, which is 0.7. When constructing a limb heat map, a limb heat map is constructed for two joints, a rectangular area is constructed with the two joints as the edge, the minimum value of the two joint weights is used as the Gaussian weight, and the distance between the points in the rectangle and the two joints is calculated for Gaussian mapping. 12 limb heat maps are constructed from 13 joints, and the 12 limb heat maps are combined to obtain a limb heat map of one frame of image. All limb and joint Gaussian heat maps are combined along the time dimension T, and finally a 3D heat map of size 13×32×56×56 is obtained. The limb Gaussian mapping expression is:

[0058]

[0059] The function D calculates the distance between the point in the rectangle and the joint point.

[0060] The driver behavior recognition method provided in the second embodiment of the present disclosure can effectively obtain a video frame sequence by dividing the acquired video data into frames; and can effectively obtain a heat map frame sequence based on the video data through a heat map generation module. Obtaining the video frame sequence and the heat map frame sequence provides data support for subsequent recognition.

[0061] In the disclosed embodiment, the multimodal interactive neural network includes an asymmetric feature extraction module, an appearance-guided heat map enhancement module and a skeleton-guided attention module. Among them, the asymmetric feature extraction module includes an appearance feature extraction neural network and a skeleton feature extraction neural network. The multimodal interactive neural network as a whole can be divided into a video branch and a skeleton branch. The appearance feature extraction neural network (including time-weighted adaptive convolution) and the skeleton-guided attention module are components of the video branch, and the skeleton feature extraction neural network (including time-weighted adaptive convolution) and the appearance-guided heat map enhancement module are components of the skeleton branch. The generation of the first feature map and the second feature map is further explained below.

[0062] Figure 3 It is a flowchart of a driver behavior identification method provided according to the fourth embodiment of the present disclosure.

[0063] like Figure 3 As shown, in the driver behavior recognition method, a multimodal interactive neural network and a time-weighted adaptive convolution are used to generate a first feature map and a second feature map based on a video frame sequence and a heat map frame sequence, which may include steps S31 to S32.

[0064] Step S31, using the appearance feature extraction neural network, the skeleton-guided attention module and the time-weighted adaptive convolution to generate a first feature map based on the video frame sequence.

[0065] Step S32, using the skeleton feature extraction neural network, the appearance-guided heat map enhancement module and the time-weighted adaptive convolution, to generate a second feature map based on the heat map frame sequence.

[0066] Among them, the appearance-guided heatmap enhancement module and the skeleton-guided attention module are used for feature interaction between the appearance feature extraction neural network and the skeleton feature extraction neural network. Specifically, the attention module uses the features of the skeleton branch to guide the appearance features to focus on the key local areas. The appearance-guided heatmap enhancement module uses the features of the video frame to guide the heatmap frame of the skeleton branch for feature enhancement.

[0067] In step S31, the appearance feature extraction neural network may include a convolution layer, a batch normalization layer, a linear rectification layer, a maximum pooling layer, a residual connection layer, a global pooling layer and a fully connected layer. The temporal weight adaptive convolution may be provided in each residual block of the residual connection layer of the appearance feature extraction neural network. The skeleton-guided attention module may be provided between any two residual blocks of the residual connection layer of the appearance feature extraction neural network.

[0068] Figure 3a is a flow chart of a method for generating a first feature map according to Embodiment 3 of the present disclosure. Figure 3a As shown, step S31 may further include steps S311 to S31.

[0069] Step S311: Use the convolution layer, batch normalization layer, linear rectification layer and maximum pooling layer to perform preliminary appearance feature extraction on the video frame sequence to generate a first appearance feature map.

[0070] For example, after obtaining the video frame sequence, preliminary appearance feature extraction is performed using the "1×7×7 Conv+BN+Relu+maxpool" structure, where Conv represents the convolution layer, BN represents the batch normalization layer, Relu represents the linear rectification layer, and maxpool represents the maximum pooling layer.

[0071] Step S312: perform multiple deep-level appearance feature extractions on the first appearance feature map using multiple residual blocks of the residual connection layer, and use a skeleton-guided attention module between any two residual blocks to enhance the appearance features of the first appearance feature map based on the skeleton features to generate a second appearance feature map.

[0072] In the process of deep appearance feature extraction in each residual block, temporal weight adaptive convolution is used to assign corresponding temporal weights to each video frame for temporal weighting.

[0073] For example, the residual connection layer of the video branch is divided into four stages of deep feature extraction, each stage includes several residual blocks, the numbers are 3, 4, 6, and 3 respectively. A skeleton-guided attention module is added after the output of the residual block in the third stage, and a time-adaptive convolution is added after the second 3D convolution (3DConv) in the residual block of each stage.

[0074] Step S313: Use the global pooling layer and the fully connected layer to perform dimension reduction and feature fusion on the second appearance feature map to generate a first feature map.

[0075] In step S32, the skeleton feature extraction neural network may include a convolution layer, a batch normalization layer, a linear rectification layer, a residual connection layer, a global pooling layer and a fully connected layer. The temporal weight adaptive convolution may be provided in each residual block of the residual connection layer of the skeleton feature extraction neural network. The appearance-guided heat map enhancement module may be provided between any two residual blocks of the residual connection layer of the skeleton feature extraction neural network.

[0076] Figure 3b is a flow chart of a method for generating a second feature map according to Embodiment 3 of the present disclosure. Figure 3b As shown, step S32 may further include steps S321 to S323.

[0077] Step S321: Use the convolution layer, batch normalization layer and linear rectification layer to perform preliminary skeleton feature extraction on the heat map frame sequence to generate a first skeleton feature map.

[0078] For example, after obtaining the heat map frame sequence, preliminary skeleton feature extraction is performed using a "1×7×7 Conv+BN+Relu" structure, where Conv represents a convolutional layer, BN represents a batch normalization layer, and Relu represents a linear rectification layer.

[0079] Step S322, using multiple residual blocks of the residual connection layer, perform multiple deep-level skeleton feature extractions on the first skeleton feature map, and use an appearance-guided heat map enhancement module between any two residual blocks to enhance the skeleton features of the first skeleton feature map based on the appearance features to generate a second skeleton feature map.

[0080] Among them, in the process of deep skeleton feature extraction in each residual block, the temporal weight adaptive convolution is used to assign corresponding temporal weights to each heat map frame for temporal weighting.

[0081] For example, the residual connection layer of the skeleton branch is divided into three stages of deep feature extraction, each stage includes several residual blocks, the numbers are 4, 6, and 3 respectively. An appearance-guided heat map enhancement module is added after the output of the residual block of the first stage, and a time-adaptive convolution is added after the second 3D convolution (3DConv) in the residual block of each stage.

[0082] Step S323: Use the global pooling layer and the fully connected layer to perform dimensionality reduction and feature fusion on the second skeleton feature map to generate a second feature map.

[0083] The asymmetric feature extraction module, the appearance-guided heat map enhancement module, and the skeleton-guided attention module are exemplified below in conjunction with specific embodiments.

[0084] The specific structure of the asymmetric feature extraction module is shown in Table 1.

[0085] Table 1 Feature extraction network of video branch and skeleton branch

[0086]

[0087]

[0088] Referring to Table 1, for the video branch, the image input to the network is sampled from the video. For the input 640*480 video, it is divided into six stages. In the first stage, 8 frames are sampled from it. After image preprocessing, the input image is 3×8×224×224. The second stage contains a 1×3×3 convolution layer with a step size of 2 and a maximum pooling layer. After the second stage, the output feature map is 64×8×56×56. The third to sixth stages are several residual blocks, the number is 3, 4, 6, and 3 respectively. The residual blocks use two 3D convolutions (3DConv) of size 1×1×1, one 3D convolution (3DConv) of size 1×3×3, and a time-adaptive convolution of size 3×1×1. The output feature map of the third stage after three residual blocks is 256×8×56×56. The output feature map of the fourth stage after 4 residual blocks is 512×8×28×28, and the output feature map of the fifth stage after 6 residual blocks is 1024×8×14×14. There is a skeleton-guided attention module between the fifth and sixth stages. The feature map after the output of this module is 1280×8×14×14. The output feature map of the sixth stage after 3 residual blocks is 2048×8×7×7. The output feature map of the sixth stage is used as the input of the fully connected layer for classification.

[0089] Referring to Table 1, for the skeleton branch, the image input to the network is sampled from the video and then subjected to skeleton extraction and 3D heat map generation. For the input 640*480 video, there are five stages. In the first stage, 32 frames are sampled from it, and skeletons are extracted frame by frame through HRnet. A total of 13 joint points are extracted, and the 2D coordinates of the 13 joint points are obtained. Then, a 13×32×56×56 heat map is generated as input through the 3D heat map generation module. The second stage includes a 1×7×7 convolution layer with a stride of 2 and a maximum pooling layer, and the output feature map is 32×32×56×56. The third to fifth stages are several residual blocks, with the number of 4, 6, and 3 respectively. The residual blocks use two 1×1×1 3D convolutions (3DConv), one 1×3×3 3D convolution (3DConv), and one 3×1×1 time adaptive convolution. The output feature map of the third stage after 4 residual blocks is 128×32×28×28. Between the third and fourth stages is the appearance-guided heat map enhancement module, the feature map after the output of this module is 160×32×28×28, the output feature map of the fourth stage after 6 residual blocks is 256×32×14×14, and the output feature map of the fifth stage after 3 residual blocks is 512×32×7×7. The output feature map of the fifth stage is used as the input of the fully connected layer for classification.

[0090] Referring to Table 1, for the appearance-guided heat map enhancement module, the input of the heat map enhancement module is the output of the fifth stage of feature extraction, which is 512×8×28×28 in size. First, after dimension transformation and 1×1×1 3D convolution (3DConv), a feature map of size 8×1×28×28 is obtained. The joint point coordinates of the current frame of the skeleton branch are used to align the ROI with the feature map of size 8×1×28×28 to obtain an alignment result of size 1×8×13. The alignment result is used as the Gaussian weight, and a Gaussian heat map of size 1×8×28×28 is generated through the 3D heat map generation module. Then, after 1×1×1 3D convolution (3DConv), a feature map of size 32×8×28×28 is obtained, which is concatenated with the output of the third stage of the skeleton branch 128×32×28×28 to obtain 160×32×28×28 as the input of the fourth stage of the skeleton branch.

[0091] Referring to Table 1, for the skeleton-guided attention module, the input of the attention module is the output feature map A of the fourth stage of the skeleton branch and the output feature map B of the fifth stage of the video branch, with sizes of 1024×8×14×14 and 256×8×14×14, respectively. First, the feature map B of the video branch with a size of 1024×8×14×14 is reduced in dimension by 1×1×1 3D convolution (3DConv), and the size of the output feature map B1 is 256×8×14×14. The feature map A of the skeleton branch and the feature map B1 of the video branch after dimensionality reduction are transformed in dimension and matrix multiplied, and sent to the sparse softmax to obtain the feature map C of size 256×8×14×14 after sparse softmax normalization. The feature map C is matrix multiplied with the feature map A of the skeleton branch. The result of the matrix multiplication is added to the feature map B1 of the video branch after dimensionality reduction to obtain the feature map C1 of size 256×8×14×14. After two layers of fully connected layers and ReLu activation function, the feature map C2 is obtained. The dimension of the feature map C2 is 1024×8×14×14, and then it is concatenated with the feature map C1 to obtain the final enhanced feature with a size of 1280×8×14×14.

[0092] The driver behavior recognition method provided in the third embodiment of the present disclosure can effectively extract multimodal information from video frames and heat map frames, and highlight the importance of key frames through a time weight adaptive convolution mechanism, thereby generating a first feature map and a second feature map. These feature maps not only capture the driver's behavior pattern, but also enhance the ability to recognize important behavior moments, providing richer and more accurate information support for the subsequent behavior recognition process.

[0093] In the disclosed embodiment, the multimodal interactive neural network includes a feature fusion module, which uses the feature fusion module to perform weighted summation on the first feature map and the second feature map to obtain a comprehensive feature map. The comprehensive feature map can be used to more comprehensively and accurately capture various aspects of driver behavior, thereby improving the performance of behavior recognition.

[0094] Figure 4 It is a flowchart of a driver behavior identification method provided according to the fourth embodiment of the present disclosure.

[0095] like Figure 4 As shown, in the driver behavior recognition method, driver behavior recognition is performed based on the first feature map and the second feature map to obtain a recognition result, which may include steps S401 to S402.

[0096] Step S401: Using a feature fusion module, a comprehensive feature map is generated based on the first feature map and the second feature map.

[0097] For example, the feature fusion module of the present disclosure adopts a weighted product fusion method to normalize and weight the first feature map and the second feature map, amplify the prediction contribution of the most certain class, weightedly fuse the two classifiers, and improve the overall prediction accuracy.

[0098] Step S402: Perform driver behavior recognition based on the comprehensive feature graph to obtain a recognition result.

[0099] Through the cooperation of various modules, the trained neural network can effectively identify the driver's behavior, avoiding the problems of single video modality being susceptible to background interference and single skeleton modality lacking scene information.

[0100] The driver behavior recognition method provided in the fourth embodiment of the present disclosure effectively combines the first feature map and the second feature map through a feature fusion module, and adopts a weighted product fusion strategy to perform normalized weighted processing on the two, which not only amplifies the prediction contribution of the most certain category, but also improves the synergy between different classifiers, thereby significantly improving the overall prediction accuracy.

[0101] For ease of understanding, the structures of the multimodal interactive neural network and time-weighted adaptive convolution provided in the embodiments of the present disclosure are explained below with reference to diagrams.

[0102] Figure 5 It is a structural schematic diagram of a multimodal interactive neural network provided according to an embodiment of the present disclosure.

[0103] like Figure 5 As shown, a multimodal interactive neural network 3D heat map generation module (heat map generation module), an asymmetric downsampling feature extraction module (asymmetric feature extraction module), an appearance-guided heat map enhancement module (appearance-guided heat map enhancement module), a skeleton-guided attention module (skeleton-guided attention module) and a post-fusion module (feature fusion module).

[0104] refer to Figure 5The neural network is divided into two branches: video branch and skeleton branch. The feature extraction network of the video branch can be divided into six stages. The first stage is sampling. Since the amount of video frame input data is large, 8 frames (uniform 8) are sampled as input. The second stage is composed of the "1×7×7 Conv+BN+Relu+maxpool" operation block. The third to sixth stages also use the ResNet residual structure, with 3, 4, 6, and 3 layers respectively. Similarly, a time-adaptive convolution is added after the second 3D convolution (3DConv) in the residual block of each stage, and finally the global pooling GAP and the fully connected layer fc. The feature extraction network of the skeleton branch can be divided into five stages. The first stage is sampling. Since the amount of heat map input data is small, 32 frames (uniform 32) are sampled from the video as input. The second stage is composed of the "1×7×7 Conv+BN+Relu" operation block. Stages 3 to 5 use the RestNet residual structure RestNetBlock, which consists of several feature extraction blocks, 4, 6, and 3 respectively. A time-adaptive convolution is added after the second 3D convolution (3DConv) in the residual block of each stage, and finally the global pooling GAP and the fully connected layer fc. In terms of the number of channels, the number of channels of the residual block feature output of the video branch is four times that of the skeleton branch, because the skeleton is also a mid-level feature, so there is no need for too deep channels. After the output of the third-stage residual structure of the skeleton branch, an appearance-guided heat map enhancement module is added, and the heat map features of the skeleton branch are guided by the output of the third-stage residual structure of the video branch. A skeleton-guided attention module is added after the fifth stage of the video branch, and the output of the fifth stage of the skeleton branch is used to enhance the features of the video branch. After the fully connected layers of the two branches, a post-fusion module is added to perform weighted product fusion to calculate the final result.

[0105] Appearance-guided heatmap enhancement module, the structure is as follows Figure 6 As shown. The present disclosure considers the local area of ​​the video centered at each joint point to enhance the features of the skeleton branch. Figure 6 First, the appearance features are subjected to 1×1×1 3D convolution (3DConv) for feature dimensionality reduction, and a 5×5 bounding box is constructed for the skeleton key points. The bounding box is used to align the appearance features with the ROI to obtain the appearance information in the appearance features. The appearance information is transmitted in the form of a Gaussian heat map, and the result of the ROI alignment is used as the Gaussian weight in the Gaussian heat map calculation to generate the Gaussian heat map of the video frame, which can better guide the skeleton heat map. The Gaussian heat map of the video frame is spliced ​​with the skeleton feature after 3D convolution (3DConv) to obtain the enhanced skeleton feature. Through this heat map enhancement module, the skeleton feature obtains the appearance information of the local area in the appearance feature, solving the problem that the skeleton feature lacks scene information.

[0106] The skeleton-guided attention module has a structure like Figure 7 As shown in Figure 2. This module uses skeleton features to guide appearance features to focus on local key areas. Figure 7 , the module first reduces the dimension of the appearance features through 1×1×1 3D convolution (3DConv), fuses the skeleton features and the reduced-dimensional appearance features through the sparse Sparse Softmax function along the spatial dimension, multiplies the skeleton features with the fused features through residual connection, and adds them to the reduced-dimensional appearance features. Then, after two fully connected layers FC, ReLu activation function and one residual connection Concat, the enhanced appearance features are obtained. Through this attention module, the video branch focuses on the key area and reduces the interference caused by the background in the video.

[0107] In the disclosed embodiment, the skeleton branch generates a heat map as the input of the branch through a 3D heat map generation module, and the two branches respectively use an asymmetric downsampling feature extraction module, and use an appearance-guided heat map enhancement module and a skeleton-guided attention module for pre-fusion. The heat map enhancement module uses the heat map of the video frame to guide the heat map of the skeleton branch for feature enhancement, and the attention module uses the features of the skeleton branch to guide the appearance features to focus on key local areas, and the post-fusion module uses a weighted product fusion method for post-fusion.

[0108] Figure 8 It is a structural diagram of a time-weighted adaptive convolution provided according to an embodiment of the present disclosure.

[0109] like Figure 8 As shown, the structure of the time adaptive convolution disclosed in the present invention is as follows Figure 1 Reference Figure 2 , the time adaptive convolution TADAConv is added after the second convolution of ResNetBlock. Assume that the input feature map X t The size is C×T×K×K. First, the global spatial pooling layer Sprt Pool is used. After global spatial pooling, the feature map size is C×T×1×1. Then, after the 1D convolution Conv with a convolution kernel size of 3×1×1, the LN layer and the GELU activation function, a feature map with a size of C / 2×T×1×1 is obtained. The feature map is sent to the multi-head self-attention of MHSA. After softmax calculation, the vector V is added to the feature map of C / 2×T×1×1. After the 1D convolution with a convolution kernel size of 3×1×1, a feature map with a size of C×T×1×1 is obtained. Then comes the last convolution layer. The last convolution layer uses the pre-trained weights of 2D convolution, which is C×T×K×K. After the last convolution layer, the calibration weight α is obtained. t , and the basic weight W b After multiplication, the output feature Xt ', the feature map output is C×T×K×K.

[0110] In the embodiment of the present disclosure, the training function is a cross entropy loss function. The embodiment of the present disclosure first uses separate cross entropy loss training for the video branch and the skeleton branch, and then uses the joint cross entropy loss for joint training.

[0111] The cross entropy function of the video branch trained separately: in is an indicator function, which is 1 when category i is the true value label, otherwise it is 0. i,v is the accuracy of the i-th category of video branch prediction.

[0112] Cross entropy function for separate training of the skeleton branch: in is an indicator function, which is 1 when category i is the true value label, otherwise it is 0. i,s is the accuracy of the i-th category predicted by the skeleton branch.

[0113] Cross entropy function for joint training of video branch and skeleton branch: in is an indicator function, which is 1 when category i is the true value label, otherwise it is 0. i,v is the accuracy of the i-th category predicted by the appearance feature extraction neural network, p i,s is the accuracy of the i-th category predicted by the skeleton feature extraction neural network.

[0114] For example, during the training process, the two branches are first trained separately, and then the joint training is performed. The number of iterations of the video branch training is set to 100, the batch size of each training is 12, and the random gradient descent optimization algorithm with a momentum of 0.9 and a weight decay of 0.0001 is used. The number of training set samples is 5962, and the number of validation set samples is 1328. The initial learning rate is 0.15, and the center crop is used to crop the input 640*480 image to a size of 224*224 as input. The number of iterations of the skeleton branch training is set to 200, the batch size of each training is 16, and the random gradient descent optimization algorithm with a momentum of 0.9 and a weight decay of 0.0003 is used. The number of training set samples is 5962, and the number of validation set samples is 1328. The initial learning rate is 0.2, and the center crop is also used. The number of training set and validation set samples is consistent with the video branch. The number of training iterations for joint training was set to 50, the batch size for each training was 12, and the same stochastic gradient descent optimization algorithm as for separate training was used, with an initial learning rate of 0.05.

[0115] Based on this, the present disclosure also provides a driver behavior recognition device, referring to Fig. 9 As shown, Fig. 9 It is a structural schematic diagram of a driver behavior recognition device provided according to an embodiment of the present disclosure.

[0116] like Fig. 9 As shown, the driver behavior recognition device 100 includes an acquisition unit 110 , a feature map generating unit 120 , and a recognition unit 130 .

[0117] The acquisition unit 110 is used to acquire a video frame sequence and a heat map frame sequence containing driver behavior.

[0118] The feature map generating unit 120 is used to generate a first feature map and a second feature map based on a video frame sequence and a heat map frame sequence by using a multimodal interactive neural network and a time weight adaptive convolution.

[0119] Among them, the temporal weight adaptive convolution is used to assign corresponding temporal weights to each frame based on the position of each frame in its respective time series; the first feature map is a skeleton-guided appearance feature map based on the time dimension, and the second feature map is a skeleton feature map guided by the appearance based on the time dimension.

[0120] The recognition unit 130 is used to perform driver behavior recognition based on the first feature map and the second feature map to obtain a recognition result.

[0121] The driver behavior recognition device provided by the embodiment of the present disclosure adopts the driver behavior recognition method in the above embodiment, which can solve the technical problems in the background technology.

[0122] The beneficial effects of the driver behavior recognition device provided by the present disclosure are the same as the beneficial effects of the driver behavior recognition method provided by the above-mentioned embodiment, and other technical features in the driver behavior recognition device are the same as the features disclosed in the driver behavior recognition method, which will not be repeated here.

[0123] An embodiment of the present disclosure also provides an electronic device, which includes: a memory configured to store instructions; and a processor, which is configured to call instructions from the memory and implement the driver behavior recognition method provided by the above embodiment when executing the instructions.

[0124] The beneficial effects of the electronic device provided by the embodiment of the present disclosure are the same as the beneficial effects of the driver behavior recognition method provided by the above-mentioned embodiment, and will not be elaborated here.

[0125] An embodiment of the present disclosure also provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the method for driver behavior identification provided in the first aspect or any embodiment of the first aspect.

[0126] The beneficial effects of the machine-readable storage medium provided in the present disclosure are the same as the beneficial effects of the driver behavior identification method provided in the above-mentioned embodiment, and will not be elaborated here.

[0127] The embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the steps of the driver behavior identification method as described above when executed by a processor.

[0128] The beneficial effects of the computer program product provided by the embodiments of the present disclosure are the same as the beneficial effects of the driver behavior recognition method provided by the above embodiments, and are not elaborated here.

[0129] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0135] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] The above are only embodiments of the present disclosure and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A driver behavior recognition method, characterized in that: include: Obtaining a video frame sequence and a heat map frame sequence containing driver behavior; A first feature map and a second feature map are generated based on the video frame sequence and the heat map frame sequence by using a multimodal interactive neural network and a time-weighted adaptive convolution; wherein the time-weighted adaptive convolution is used to assign a corresponding time weight to each frame based on the position of each frame in the respective time sequence; the first feature map is a skeleton-guided appearance feature map based on time weighting, and the second feature map is a skeleton-guided appearance feature map based on time weighting; Driver behavior recognition is performed based on the first feature map and the second feature map to obtain a recognition result.

2. The driver behavior recognition method according to claim 1, characterized in that: The multimodal interactive neural network includes an asymmetric feature extraction module, an appearance-guided heatmap enhancement module, and a skeleton-guided attention module; wherein the asymmetric feature extraction module includes an appearance feature extraction neural network and a skeleton feature extraction neural network; The method of using a multimodal interactive neural network and a time weight adaptive convolution to generate a first feature map and a second feature map based on the video frame sequence and the heat map frame sequence includes: Generate the first feature map based on the video frame sequence using the appearance feature extraction neural network, the skeleton-guided attention module and the temporal weight adaptive convolution; generating the second feature map based on the heat map frame sequence using the skeleton feature extraction neural network, the appearance-guided heat map enhancement module and the temporal weight adaptive convolution; Among them, the appearance-guided heat map enhancement module and the skeleton-guided attention module are used for feature interaction between the appearance feature extraction neural network and the skeleton feature extraction neural network.

3. The driver behavior recognition method according to claim 2, characterized in that: The appearance feature extraction neural network includes a convolution layer, a batch normalization layer, a linear rectification layer, a maximum pooling layer, a residual connection layer, a global pooling layer and a fully connected layer; the temporal weight adaptive convolution is arranged in each residual block of the residual connection layer of the appearance feature extraction neural network; the skeleton-guided attention module is arranged between any two residual blocks of the residual connection layer of the appearance feature extraction neural network; The method of using the appearance feature extraction neural network, the skeleton-guided attention module, and the temporal weight adaptive convolution to generate the first feature map based on the video frame sequence includes: Performing preliminary appearance feature extraction on the video frame sequence using a convolutional layer, a batch normalization layer, a linear rectification layer, and a maximum pooling layer to generate a first appearance feature map; Using multiple residual blocks of the residual connection layer, multiple deep-level appearance feature extractions are performed on the first appearance feature map, and the skeleton-guided attention module is used between any two residual blocks to enhance the appearance features of the first appearance feature map based on skeleton features to generate a second appearance feature map; wherein, in the process of performing deep-level appearance feature extraction on each residual block, the temporal weight adaptive convolution is used to assign a corresponding temporal weight to each video frame for temporal weighting; The second appearance feature map is subjected to dimensionality reduction and feature fusion by using a global pooling layer and a fully connected layer to generate the first feature map.

4. The driver behavior recognition method according to claim 2, characterized in that: The skeleton feature extraction neural network includes a convolution layer, a batch normalization layer, a linear rectification layer, a residual connection layer, a global pooling layer and a fully connected layer; the temporal weight adaptive convolution is arranged in each residual block of the residual connection layer of the skeleton feature extraction neural network; the appearance-guided heat map enhancement module is arranged between any two residual blocks of the residual connection layer of the skeleton feature extraction neural network; The step of generating the second feature map based on the heat map frame sequence by using the skeleton feature extraction neural network, the appearance-guided heat map enhancement module and the temporal weight adaptive convolution comprises: Performing preliminary skeleton feature extraction on the heat map frame sequence by using a convolution layer, a batch normalization layer, and a linear rectification layer to generate a first skeleton feature map; Using multiple residual blocks of the residual connection layer, multiple deep-level skeleton feature extractions are performed on the first skeleton feature map, and the appearance-guided heat map enhancement module is used between any two residual blocks to enhance the skeleton features of the first skeleton feature map based on the appearance features to generate a second skeleton feature map; wherein, in the process of deep-level skeleton feature extraction of each residual block, the temporal weight adaptive convolution is used to assign a corresponding temporal weight to each heat map frame for temporal weighting; The second skeleton feature map is subjected to dimensionality reduction and feature fusion by using a global pooling layer and a fully connected layer to generate the second feature map.

5. The driver behavior recognition method according to claim 1, characterized in that: The temporal weight adaptive convolution decomposes the temporal weight of each frame in the spatial convolution into the product of the base weight shared by all frames and the calibration weight of each frame. The temporal weight is: W t =W b *α t , where W t is the weight of the tth frame, W b is the base weight shared by all frames, α t is the calibration weight of the tth frame.

6. The driver behavior recognition method according to claim 1, characterized in that: The step of obtaining a video frame sequence and a heat map frame sequence containing driver behavior includes: Acquire video data containing driver behavior; Based on the video data, generating the video frame sequence; and The heat map generation module is used to determine skeleton joint points in the video data based on the video data, and to generate the heat map frame sequence based on the skeleton joint points.

7. The driver behavior recognition method according to claim 1, characterized in that: The multimodal interactive neural network includes a feature fusion module; the driver behavior recognition based on the first feature map and the second feature map is performed to obtain a recognition result, including: Using the feature fusion module, based on the first feature map and the second feature map, a comprehensive feature map is generated; The driver behavior is identified based on the comprehensive feature graph to obtain the identification result.

8. A driver behavior recognition device, characterized in that: include: An acquisition unit, used for acquiring a video frame sequence and a heat map frame sequence containing driver behavior; A feature map generating unit is used to generate a first feature map and a second feature map based on the video frame sequence and the heat map frame sequence by using a multimodal interactive neural network and a time weight adaptive convolution; wherein the time weight adaptive convolution is used to assign a corresponding time weight to each frame based on the position of each frame in the respective time sequence; the first feature map is a skeleton-guided appearance feature map based on the time dimension, and the second feature map is a skeleton-guided appearance feature map based on the time dimension; The recognition unit is used to perform driver behavior recognition based on the first characteristic map and the second characteristic map to obtain a recognition result.

9. An electronic device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the method for driver behavior recognition according to any one of claims 1 to 7 when executing the instructions.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for enabling a machine to execute the driver behavior identification method according to any one of claims 1 to 7.

Images