Driver behavior recognition method and system based on skeleton and semantic information mixing

Abstract

The application discloses a driver behavior recognition method and system based on skeleton and semantic information mixing, and relates to the technical field of driver behavior recognition. The driver behavior recognition method based on semantic information and skeleton information is fused through a spatial soft attention mechanism based on driver skeleton key points to recognize driver behavior. First, a spatial soft attention module based on driver skeleton key point information is designed. The module gives greater weight to the semantic information of the driver's hands and head, and suppresses global background information, so as to improve the performance of extracting semantic features related to driver behavior. Then, a hybrid network based on driver skeleton and semantic information is designed to extract semantic and temporal features related to driver behavior recognition. In addition, in the task of recognizing driver behavior at night, adaptive histogram equalization and transfer learning methods are introduced to solve the problem of low brightness and small amount of night driving data.

Description

Technical Field

[0001] This invention relates to the field of driver behavior recognition technology, and in particular to a driver behavior recognition method and system based on a combination of skeleton and semantic information. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Whether a driver's behavior is standardized during driving significantly impacts driving and road safety. Driver behavior recognition refers to identifying various driver actions within the vehicle cabin, such as making phone calls, drinking water, and adjusting the radio. It is widely used in assisted driving and autonomous driving systems and is crucial for driving safety. Existing driver behavior recognition methods are mostly based on semantic or skeleton information. Semantic information-based methods often use image or video data as input; the former lacks temporal information and motion patterns, while the latter contains a large amount of redundant information and is computationally intensive. Skeleton information-based driver behavior recognition methods use the sequence of key points on the driver's skeleton as input, resulting in low computational cost and fast inference speed, but lacking scene semantic information and target interaction information, leading to low recognition accuracy. Attention mechanisms are also widely used in driver behavior recognition methods, but they are often not optimized for driver behavior recognition tasks and cannot accurately locate the most relevant parts of the driver's behavior. Furthermore, in nighttime environments, low brightness and limited available data make model training difficult, leading to inaccurate recognition results and significantly increasing driving safety hazards. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a driver behavior recognition method and system based on a hybrid skeleton and semantic information approach. This invention extends the data modality for driver behavior recognition by using semantic and skeleton information as model inputs to extract spatiotemporal features of driving behavior from the semantic stream and skeleton stream, respectively. It leverages a spatial soft attention mechanism based on key points of the driver skeleton to efficiently extract detailed semantic information related to driving behavior from the semantic stream. Finally, it employs a linear fusion strategy to fuse the results of the semantic stream and skeleton stream, outputting the final driver behavior recognition result.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a driver behavior recognition method that utilizes semantic information and skeleton information, and is based on a spatial soft attention mechanism and a linear fusion strategy using driver skeleton key point information, including: a spatial soft attention module based on driver skeleton key point information, a hybrid network based on driver skeleton and semantic information, and an adaptive histogram equalization and transfer learning method to address the problems of low brightness and small quantity of nighttime driving data.

[0006] The first aspect of this invention provides a driver behavior recognition method based on a combination of skeleton and semantic information, comprising the following steps:

[0007] Input a video frame sequence, use a pose estimation algorithm to obtain the driver's skeleton key point sequence, and extract the coordinates of the driver's hand and head key points from it;

[0008] In the input video frame, the region of interest related to the driver's behavior is initially determined by combining circular regions with the driver's two key points on his hands and head as the center and the set value as the radius.

[0009] Generate an attention mask of the same size as the input video frame, and overlay the attention mask onto the corresponding input video frame to obtain a video frame sequence with the attention mask;

[0010] Skeleton flow prediction results are obtained using skeleton keypoint sequences and adaptive graph convolutional neural networks; semantic flow prediction results are obtained using video frame sequences with attention masks and slow semantic networks.

[0011] Using skeleton flow prediction results and semantic flow prediction results as input, a linear fusion strategy is used to fuse the behavior prediction results to obtain the final driver behavior recognition result.

[0012] Furthermore, after identifying the region of interest related to driver behavior, the preliminary results are adjusted according to the characteristics of driver behavior to obtain the final region of interest for driver behavior.

[0013] Furthermore, the region of interest in the attention mask is set with a high grayscale value, while the background region is set with a low grayscale value.

[0014] Furthermore, the specific steps for obtaining the skeleton flow prediction results are as follows: taking the skeleton key point sequence as input, constructing a skeleton graph sequence with the driver's skeleton key points as vertices and the natural connection relationship of the skeleton key points as edges, and using an adaptive graph convolutional neural network to extract the spatiotemporal features of the driver's behavior to obtain the skeleton flow prediction results.

[0015] Furthermore, the input skeleton keypoint sequence is first augmented and batch normalized. Then, an adaptive graph convolutional network is used to extract spatiotemporal features. The adaptive graph convolutional network consists of 9 adaptive graph convolutional modules. At the end of the skeleton flow are global average pooling layers and softmax layers, which output the skeleton flow prediction results.

[0016] Furthermore, the specific steps for obtaining semantic flow prediction results are as follows: taking a video frame sequence with an attention mask as input, extracting a sparse video frame sequence, and using a slow semantic network to efficiently extract the spatiotemporal features of driver behavior to obtain semantic flow prediction results.

[0017] Furthermore, after taking a video frame sequence with an attention mask as input, the input frames are sparsely sampled, and feature extraction is performed using an adjusted 3D ResNet network. The semantic flow is treated as a slow semantic feature extraction network, and global average pooling, fully connected layers, and softmax layers are introduced at the end of the slow semantic feature extraction network to output the semantic flow prediction results.

[0018] Furthermore, adaptive histogram equalization and transfer learning methods are used to process nighttime driving data, thereby enabling nighttime driver behavior recognition.

[0019] Furthermore, using a nighttime driver video frame sequence as input, histogram equalization is used to adjust the grayscale values ​​of the image, dividing the image into several small regions to adaptively adjust the local features and boundaries of each region, and limiting the image contrast to reduce noise, thereby obtaining a nighttime driver video frame sequence with stronger grayscale contrast.

[0020] Using a processed sequence of nighttime driver video frames as input, the driver behavior recognition model trained on a sequence of daytime driver video frames is transferred to a driver behavior recognition network based on a semantic skeleton space attention hybrid network to train a driver behavior recognition model for nighttime environments.

[0021] A second aspect of the present invention provides a driver behavior recognition system based on a combination of skeleton and semantic information, comprising:

[0022] The skeleton key point acquisition module is configured to take a sequence of video frames as input, use a pose estimation algorithm to acquire a sequence of key points of the driver's skeleton, and extract the coordinates of key points of the driver's hands and head from it;

[0023] The region of interest module is configured to initially determine the region of interest related to the driver's behavior in the input video frame by combining circular regions centered on the driver's two hand and head key points and with a set value as the radius.

[0024] The attention mask module is configured to generate an attention mask of the same size as the input video frame, and to overlay the attention mask onto the corresponding input video frame to obtain a video frame sequence with the attention mask.

[0025] The result prediction module is configured to obtain skeleton flow prediction results using skeleton keypoint sequences and adaptive graph convolutional neural networks; and to obtain semantic flow prediction results using video frame sequences with attention masks and slow semantic networks.

[0026] The behavior recognition result module is configured to take skeleton flow prediction results and semantic flow prediction results as input, and use a linear fusion strategy to fuse the behavior prediction results to obtain the final driver behavior recognition result.

[0027] The above one or more technical solutions have the following beneficial effects:

[0028] (1) The present invention designs a spatial soft attention module based on skeleton key point information to accurately locate the driver behavior related region of interest, which is conducive to efficiently extracting the features of the driver behavior related region.

[0029] (2) The present invention designs a hybrid network based on driver skeleton and semantic information, which effectively integrates semantic and skeleton information related to driver behavior. The two types of information complement each other and effectively improve the accuracy of driver behavior recognition.

[0030] (3) The present invention introduces a histogram equalization operation with limited contrast and a transfer learning method in the task of nighttime driver behavior recognition, which helps to solve the problems of low brightness and contrast and small quantity of nighttime driver behavior data and improves the performance of the nighttime driver behavior recognition method.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 The flowchart of the driver behavior recognition method based on the combination of skeleton and semantic information in Embodiment 1 of the present invention;

[0034] Figure 2 : A schematic diagram of the definition of skeleton key points in driver behavior recognition according to Embodiment 1 of the present invention;

[0035] Figure 3 The structure diagram of the driver behavior recognition network based on the skeleton, semantic information and spatial attention hybrid model of the present invention is shown in Embodiment 1 of the present invention.

[0036] Figure 4 The following is a framework diagram of the adaptive histogram equalization and transfer learning method for nighttime driver behavior data according to Embodiment 1 of the present invention.

[0037] Figure 5 Example image of the 3MDAD driver behavior recognition dataset. Detailed Implementation

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0040] Example 1:

[0041] Embodiment 1 of the present invention provides a driver behavior recognition method based on a combination of skeleton and semantic information, the flowchart of which is shown below. Figure 1 As shown, the specific steps include:

[0042] Step 1: Input a video frame sequence, use a pose estimation algorithm to obtain the driver's skeleton keypoint sequence, and extract the coordinates of the driver's hands and head keypoints. The keypoints are defined according to one of the two commonly used definitions in behavior recognition: 18 points or 25 points. The main difference between 25 and 18 keypoints lies in the feet, while driver behavior recognition tasks are primarily related to the driver's upper body movements; 18 keypoints are sufficient to represent driver behavior. Specific keypoints are as follows... Figure 2 As shown, 0 represents the nose, 1 represents the neck, 2 represents the right shoulder, 3 represents the right elbow, 4 represents the right wrist, 5 represents the left shoulder, 6 represents the left elbow, 7 represents the left wrist, 8 represents the right hip, 9 represents the right knee, 10 represents the right ankle, 11 represents the left hip, 12 represents the left knee, 13 represents the left ankle, 14 represents the right eye, 15 represents the left eye, 16 represents the right ear, and 17 represents the left ear.

[0043] Step 1.1: Input the driver behavior RGB video frame sequence I t The AlphaPose pose estimation algorithm was used to obtain the driver's skeleton key point sequence V. t ={V ti |t=1,2,...,n; i=1,2,...,18}.

[0044] Step 2: In the input video frame, the region of interest related to the driver's behavior is initially determined by the combination of circular regions centered on the driver's two hands and head key points and set as the radius. In this embodiment, the radii are set to a, a, and e according to the size of the area occupied by the driver in the image. All three radii are approximately twice the radius of the area occupied by the driver's head. Since the driver's behavior is mostly related to the object held in hand, the radius of the hands can be appropriately increased.

[0045] Step 2.1: Extract the driver's hand and head keypoint sequences H from the driver's skeleton keypoint sequence obtained in Step 1. t ={H ti |t=1,2,...,n;i=4,7,17}, where i=4, i=7 and i=17 represent the key points of the right hand, left hand and head, respectively.

[0046] Step 2.2: Initially determine the circular areas centered on key points of the hands and head as regions of interest, since driver behavior is mostly related to the hands and head (e.g., making a phone call). In order to cover as much of the target area as possible while covering as little of the background area as possible, set the radius of the area centered on the left and right hands as a and a, and the radius of the area centered on the head as e.

[0047] Step 2.3: After determining the region of interest (ROI) related to driver behavior, adjust the preliminary results according to the characteristics of driver behavior to obtain the final ROI. Specifically, considering the characteristics of driving behavior, the ROI is mostly located in the extension from the elbow to the hand and in front of the face (e.g., when tuning the radio, the extension from the elbow to the hand is highly relevant to the radio; when drinking water, the cup is highly relevant to the driver's face). Therefore, further adjust the above preliminary ROI, as shown in the formula below:

[0048]

[0049]

[0050] sgn(H' ix -E jx ) = sgn(H ix -E jx (3)

[0051] Among them (H) ix H iy ) and (H' ix H' iy ) represents the coordinates of the center of the region of interest in the hand before and after adjustment, i = {4, 7}, (E jx E jy ) represents the key point at the elbow, j = {3, 6}.

[0052] Step 3: As Figure 3 As shown, a spatial attention hybrid network model based on skeleton keypoints and a hybrid network of driver skeleton and semantic information is established. This model is used to generate skeleton flow prediction results and semantic flow prediction results. The driver skeleton and semantic information hybrid network includes a semantic flow and a skeleton flow. The semantic flow is an adjusted 3D ResNet structure used to extract spatiotemporal features of video frame sequences, focusing on semantic features. The skeleton flow is an adaptive graph convolutional neural network structure, mainly composed of 9 adaptive graph convolutional blocks, used to extract spatiotemporal features of skeleton keypoint sequences, focusing on dynamic changes in driver behavior.

[0053] Step 3.1: Generate an attention mask of the same size as the input video frame using a spatial soft attention module based on the skeleton key points. Overlay the attention mask with the corresponding input video frame to obtain a video frame sequence with the attention mask.

[0054] Specifically, an attention mask M is generated that is the same size as the input video frame. t The region of interest is set with a high grayscale value, and the background region is set with a low grayscale value. This mask is superimposed on the corresponding input video frame to obtain a video frame sequence A with skeleton spatial soft attention. t ,

[0055]

[0056] λ is the weight that controls the brightness of the original video frame, λ>0.

[0057] Step 3.2: Obtain skeleton flow prediction results using skeleton keypoint sequences and adaptive graph convolutional neural networks; obtain semantic flow prediction results using video frame sequences with attention masks and slow semantic networks;

[0058] Step 3.2.1: The specific steps for obtaining the skeleton flow prediction result are as follows: Using the skeleton keypoint sequence as input, construct a skeleton graph sequence with the driver's skeleton keypoints as vertices and the natural connection relationships of the skeleton keypoints as edges. Use an adaptive graph convolutional neural network to extract the spatiotemporal features of driver behavior to obtain the skeleton flow prediction result. Special attention is paid to the dynamic change features related to driver behavior. The result is mapped to the (0, 1) interval through a softmax network layer.

[0059] Specifically, the skeleton flow uses the skeleton key point sequence V tAs input, each frame of skeleton data can be viewed as a graph structure, with keypoints as vertices and natural connections between keypoints as edges. First, the input skeleton keypoint sequence undergoes data augmentation and batch normalization. Then, an adaptive graph convolutional network (CBR) is used for spatiotemporal feature extraction. The CBR consists of 9 adaptive graph convolutional blocks, and the skeleton stream utilizes K... t ×1 convolution is used for temporal feature extraction, and spatial features are extracted using an adaptive graph convolutional layer. The adaptive graph convolutional layer is an improvement on the basic graph convolutional layer. The basic graph convolutional layer and the adaptive graph convolutional layer can be represented by the following equations (5) and (6), respectively:

[0060]

[0061]

[0062] Where f and v represent the feature map and graph node, l i Mapping functions designed for ST-GCN; B i Z is a sampling region that can be divided into three sub-regions. ij K is used to balance the contributions of the three sampling regions. v The kernel size is set to 3, W k Let A be a weight vector. k B represents the physical connection relationship of key points in the driver's skeleton. k C represents the strength of the connection between key points in the skeleton. k This is learnable graph structure data that relies on skeleton keypoint data. At the end of the skeleton flow are a global average pooling layer and a softmax layer, outputting the skeleton flow prediction result O. sk .

[0063] Step 3.3: The specific steps for obtaining the semantic flow prediction results are as follows: Taking the video frame sequence with attention mask as input, sparse video frame sequences are extracted, and the spatiotemporal features of driver behavior are efficiently extracted using a slow semantic network to obtain the semantic flow prediction results. Special attention is paid to the spatial semantic features related to driver behavior, and the results are mapped to the (0, 1) interval through a softmax network layer.

[0064] Specifically, the semantic stream uses the video frame sequence A with attention mask obtained in step 3. t As input, the semantic stream focuses on extracting spatial features because semantic features in driving behavior change slowly over time. Using a sequence of video frames with an attention mask as input, the input frames are sparsely sampled, and feature extraction is performed using an adjusted 3DResNet network. The semantic stream is considered a slow semantic feature extraction network, details of which are shown in Table 1. Global average pooling, fully connected layers, and a softmax layer are introduced at the end of the slow semantic feature extraction network, outputting the semantic stream prediction result O.sm .

[0065] More specifically, skeleton flow and semantic flow are two network models for extracting features from skeleton sequences and video frame sequences, respectively.

[0066] The skeleton flow is an adaptive graph convolutional network, mainly composed of 9 adaptive graph convolutional blocks. Each adaptive graph convolutional block consists of {adaptive graph convolutional layer, batch normalization layer, ReLU layer, dropout layer, temporal convolutional layer, batch normalization layer, ReLU layer}. The adaptive graph convolutional layer extracts spatial features, and the temporal convolutional layer (K... t ×1) Extract time-related dynamic features. The adaptive graph convolutional layer (Formula 6) is an improvement on the basic graph convolutional layer (Formula 5).

[0067] Since the input sequence has a large time span after sparse sampling, using temporal convolution in the early layers will reduce accuracy. Therefore, the semantic flow is adjusted to a 3D ResNet structure: temporal convolution is only used on res4 and res5, and temporal convolution is not used on conv1, res2 and res3 to obtain the semantic flow structure, as shown in Table 1.

[0068] Table 1: Details of each network layer in semantic flow

[0069]

[0070] In Table 1, {T×S 2 In {C}, T, S, and C represent the temporal kernel dimension, spatial kernel dimension, and channel size, respectively.

[0071] Step 4: Using the skeleton flow prediction results and semantic flow prediction results as input, the behavior prediction results are fused using a linear fusion strategy to obtain the final driver behavior recognition results.

[0072] Specifically, based on the prediction results of the skeleton stream and semantic stream obtained in step 3 (O sk and O sm Taking as input, the final prediction result O is output after fusing using a linear fusion strategy. s The linear fusion strategy can be expressed by the following formula:

[0073] O s =α×O sk +O sm (7)

[0074] Where α is the weight of the skeleton flow prediction result. If the skeleton flow prediction result is better, then α>1, otherwise 0<α<1.

[0075] Step 5: Process nighttime driving data using adaptive histogram equalization and transfer learning methods to achieve nighttime driver behavior recognition.

[0076] Specifically, nighttime driver behavior data is difficult to collect, has a limited quantity, and suffers from low brightness and contrast, making nighttime driver behavior recognition more challenging.

[0077] Using a sequence of video frames of drivers at night as input, histogram equalization is used to adjust the grayscale values ​​of the images. The histogram equalization operation can be expressed by the following formula:

[0078]

[0079] Where cdf represents the cumulative distribution function, with a normalized interval of [0, 255]. min It is the smallest non-zero value (value 1) of the cumulative distribution function, M×N represents the image size, and L is the gray level used (value 255). To make the histogram equalization result smoother, the image is divided into several segments of size P. x ×P y Small regions are used to adaptively adjust the local features and boundaries of each region, and image contrast is limited to reduce noise, to obtain a nighttime driver video frame sequence with stronger grayscale contrast;

[0080] Using a processed sequence of nighttime driver video frames as input, the driver behavior recognition model trained on a sequence of daytime driver video frames is transferred to a driver behavior recognition network based on a semantic skeleton space attention hybrid network to train a driver behavior recognition model for nighttime environments.

[0081] As a further technical solution, the training process of a spatial attention hybrid network based on skeleton and semantic information is as follows: The input is a sequence of video frames depicting driver behavior. The skeleton stream takes the sequence of skeleton keypoints obtained from the video frame sequence through a pose estimation algorithm as input. SGD with momentum is selected as the network optimizer, with a training period of 55 epochs and an initial learning rate of 0.1, varying to 0.01 and 0.001 at epochs 10 and 45 respectively. Momentum is set to 0.9. The semantic stream takes the video frame sequence processed by the skeleton keypoint-based spatial soft attention module as input. Sparse sampling of the video frame sequence with skeleton spatial soft attention is performed with a stride of 16. Similar to the skeleton stream, SGD with momentum is selected as the network optimizer, with a training period of 600 epochs, a learning rate of 0.1, and momentum set to 0.9. The weights used in the linear fusion strategy are set to 0.7.

[0082] like Figure 4As shown, the nighttime driver behavior recognition method takes a sequence of video frames after contrast-limited adaptive histogram equalization as input. The input frame image size is 640×480, which is divided into 8×8 blocks for adaptive histogram equalization. The contrast extension value is limited to 5. Its network parameters are initialized by the driver behavior recognition model in the daytime environment. The initial learning rate is set to 0.01 in the skeleton stream, and the training period is adjusted to 300 epochs in the semantic stream. Other hyperparameters are consistent with the driver behavior recognition network in the daytime environment.

[0083] The present invention has experimentally verified the technical effects of the above-mentioned solution:

[0084] 1. Experimental conditions

[0085] The hardware requirements for all verification experiments of this invention are as follows: CPU: Intel Xeon 4114, RAM: 256G, Graphics card: 1 TITAN RTX.

[0086] The software environment used in the experiment was: Ubuntu 16.04, Python 3.7, PyTorch 1.8.0, and TorchVision 0.9.0.

[0087] The experimental data comes from the 3MDAD public dataset, which contains 16 categories of driving actions, such as Figure 5 The following represents the categories of driving behavior:

[0088] The first line represents: A1 Safe driving, A2 Grooming, A3 Tuning the radio, A4 Adjusting the GPS, A5 Using the phone with the left hand;

[0089] The second row represents: A6 using a mobile phone with the right hand, A7 making a phone call with the left hand, A8 making a phone call with the right hand, A9 taking a selfie, and A10 talking to a passenger.

[0090] The third line represents: A11 dancing with joy, A12 being tired or sleepy, A13 drinking water with the left hand, A14 drinking water with the right hand, A15 taking something from behind, and A16 smoking.

[0091] Data was collected from 50 people in a daytime environment and from 19 people in a nighttime environment. This invention only uses the RGB video frame sequence collected by the side camera in the data set.

[0092] 2. Experimental Content and Results

[0093] This invention conducted driver behavior recognition experiments on the 3MDAD dataset under both daytime and nighttime environments, with nighttime driver behavior recognition being more challenging. Table 2 compares the method designed in this invention with previous algorithms, where the evaluation metric is recognition accuracy; a higher value indicates better performance.

[0094] Table 2 Performance comparison of different algorithms on the 3MDAD driver behavior recognition dataset

[0095]

[0096]

[0097] Table 2 shows that the present invention outperforms previous algorithms in terms of performance (the experimental results given in Table 2 are the mean of the median of 10 independent experiments, while the experimental results of previous algorithms are from relevant papers). In addition, the present invention also conducted ablation experiments, and Table 3 demonstrates the effectiveness of the spatial soft attention module based on driver skeleton key points and the hybrid network based on driver skeleton and semantic information (the experimental results given in Table 3 are from 10 independent experiments).

[0098] Table 3. Experimental results of driver behavior recognition based on a semantic skeleton spatial attention hybrid network.

[0099]

[0100]

[0101] Note: (-SSSA-M) and (+SSSA-M) indicate that the semantic flow does not use and uses a spatial soft attention mechanism based on skeleton keypoints, respectively.

[0102] Furthermore, Table 4 demonstrates the effectiveness of adaptive histogram equalization and the spatial soft attention module based on skeleton key points in the nighttime driver behavior recognition method.

[0103] Table 4 Nighttime Driver Behavior Recognition Results

[0104] method Accuracy (%) skeleton-stream 64.46 skeleton-stream(+equalization) 61.16 semnatic-stream 67.77 semnatic-stream(+equalization) 66.94 semnatic-stream(+equalization+SSSA-M) 74.38 SSAH-Net (nighttime) 78.51

[0105] Note: (+equalization) indicates that the adaptive histogram equalization with contrast limitation is used during training, and (+SSSA-M) indicates that a spatial soft attention mechanism based on skeleton keypoints is used.

[0106] Example 2:

[0107] Embodiment 2 of the present invention provides a driver behavior recognition system based on a hybrid skeleton and semantic information, comprising:

[0108] The skeleton key point acquisition module is configured to take a sequence of video frames as input, use a pose estimation algorithm to acquire a sequence of key points of the driver's skeleton, and extract the coordinates of key points of the driver's hands and head from it;

[0109] The region of interest module is configured to initially determine the region of interest related to the driver's behavior in the input video frame by combining circular regions centered on the driver's two hand and head key points and with a set value as the radius.

[0110] The attention mask module is configured to generate an attention mask of the same size as the input video frame, and to overlay the attention mask onto the corresponding input video frame to obtain a video frame sequence with the attention mask.

[0111] The result prediction module is configured to obtain skeleton flow prediction results using skeleton keypoint sequences and adaptive graph convolutional neural networks; and to obtain semantic flow prediction results using video frame sequences with attention masks and slow semantic networks.

[0112] The behavior recognition result module is configured to take skeleton flow prediction results and semantic flow prediction results as input, and use a linear fusion strategy to fuse the behavior prediction results to obtain the final driver behavior recognition result.

[0113] The nighttime data processing module is configured to process nighttime driving data using adaptive histogram equalization and transfer learning methods, thereby enabling nighttime driver behavior recognition.

[0114] Specifically, taking a nighttime driver video frame sequence as input, the grayscale value of the image is adjusted using histogram equalization, the image is divided into several small regions, the local features and boundaries of each region are adaptively adjusted, and the image contrast is limited to reduce noise, so as to obtain a nighttime driver video frame sequence with stronger grayscale contrast.

[0115] Using a processed sequence of nighttime driver video frames as input, the driver behavior recognition model trained on a sequence of daytime driver video frames is transferred to a driver behavior recognition network based on a semantic skeleton space attention hybrid network to train a driver behavior recognition model for nighttime environments.

[0116] The network training module is configured to train a spatial attention hybrid network based on skeleton and semantic information.

[0117] Specifically, the input is a sequence of video frames depicting driver behavior. The skeleton stream takes the sequence of skeleton keypoints obtained from the video frame sequence using a pose estimation algorithm as input, and selects SGD with momentum as the network optimizer. The training period is set to 55 epochs, with an initial learning rate of 0.1, varying to 0.01 and 0.001 at epochs 10 and 45 respectively, and a momentum of 0.9. The semantic stream takes the video frame sequence processed by a spatial soft attention module based on skeleton keypoints as input, and performs sparse sampling on the video frame sequence with skeleton spatial soft attention at a stride of 16. Similar to the skeleton stream, SGD with momentum is selected as the network optimizer, with a training period of 600 epochs, a learning rate of 0.1, and a momentum of 0.9. The linear fusion strategy uses weights of 0.7.

[0118] The nighttime driver behavior recognition method takes a sequence of video frames after contrast-limited adaptive histogram equalization as input. The input frame image size is 640×480, which is divided into 8×8 blocks for adaptive histogram equalization. The contrast extension value is limited to 5. Its network parameters are initialized by the driver behavior recognition model in the daytime environment. The initial learning rate is set to 0.01 in the skeleton stream and the training period is adjusted to 300 epochs in the semantic stream. Other hyperparameters are consistent with the driver behavior recognition network in the daytime environment.

[0119] The steps and methods involved in Embodiment 2 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0120] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0121] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A driver behavior recognition method based on a hybrid skeleton and semantic information, characterized in that, Includes the following steps: Input a video frame sequence, use a pose estimation algorithm to obtain the driver's skeleton key point sequence, and extract the coordinates of the driver's hand and head key points from it; In the input video frame, the region of interest related to the driver's behavior is initially determined by combining circular regions with the driver's two key points on his hands and head as the center and the set value as the radius. Generate an attention mask of the same size as the input video frame, and overlay the attention mask onto the corresponding input video frame to obtain a video frame sequence with the attention mask; Skeleton flow prediction results are obtained using skeleton keypoint sequences and adaptive graph convolutional neural networks; semantic flow prediction results are obtained using video frame sequences with attention masks and slow semantic networks. Using skeleton flow prediction results and semantic flow prediction results as input, a linear fusion strategy is used to fuse the behavior prediction results to obtain the final driver behavior recognition result.

2. The driver behavior recognition method based on skeleton and semantic information hybridization as described in claim 1, characterized in that, After identifying the region of interest related to driver behavior, the preliminary results are adjusted based on the characteristics of driver behavior to obtain the final region of interest for driver behavior.

3. The driver behavior recognition method based on skeleton and semantic information hybridization as described in claim 1, characterized in that, In the attention mask, the region of interest is set to a high grayscale value, while the background region is set to a low grayscale value.

4. The driver behavior recognition method based on skeleton and semantic information hybridization as described in claim 1, characterized in that, The specific steps to obtain the skeleton flow prediction results are as follows: taking the skeleton key point sequence as input, constructing a skeleton graph sequence with the driver's skeleton key points as vertices and the natural connection relationship of the skeleton key points as edges, and using an adaptive graph convolutional neural network to extract the spatiotemporal features of the driver's behavior to obtain the skeleton flow prediction results.

5. The driver behavior recognition method based on skeleton and semantic information hybridization as described in claim 4, characterized in that, First, the input skeleton keypoint sequence is augmented and batch normalized. Then, an adaptive graph convolutional network is used to extract spatiotemporal features. The adaptive graph convolutional network consists of 9 adaptive graph convolutional blocks. At the end of the skeleton flow are global average pooling layers and softmax layers, which output the skeleton flow prediction results.

6. The driver behavior recognition method based on skeleton and semantic information hybridization as described in claim 1, characterized in that, The specific steps to obtain semantic flow prediction results are as follows: taking a video frame sequence with an attention mask as input, extracting a sparse video frame sequence, using a slow semantic network to efficiently extract the spatiotemporal features of driver behavior, and obtaining semantic flow prediction results.

7. The driver behavior recognition method based on skeleton and semantic information hybridization as described in claim 6, characterized in that, After taking a video frame sequence with an attention mask as input, the input frames are sparsely sampled, and features are extracted using an adjusted 3D ResNet network. The semantic flow is regarded as a slow semantic feature extraction network. Global average pooling, fully connected layers and softmax layers are introduced at the end of the slow semantic feature extraction network to output the semantic flow prediction results.

8. The driver behavior recognition method based on skeleton and semantic information hybridization as described in claim 1, characterized in that, Adaptive histogram equalization and transfer learning methods are used to process nighttime driving data, thereby enabling nighttime driver behavior recognition.

9. The driver behavior recognition method based on skeleton and semantic information hybridization as described in claim 8, characterized in that, Using a sequence of nighttime driver video frames as input, histogram equalization is used to adjust the grayscale values ​​of the image, dividing the image into several small regions to adaptively adjust the local features and boundaries of each region, and limiting the image contrast to reduce noise, thereby obtaining a nighttime driver video frame sequence with stronger grayscale contrast. Using a processed sequence of nighttime driver video frames as input, the driver behavior recognition model trained on a sequence of daytime driver video frames is transferred to a driver behavior recognition network based on a semantic skeleton space attention hybrid network to train a driver behavior recognition model for nighttime environments.

10. A driver behavior recognition system based on a hybrid skeleton and semantic information, characterized in that, include: The skeleton key point acquisition module is configured to take a sequence of video frames as input, use a pose estimation algorithm to acquire a sequence of key points of the driver's skeleton, and extract the coordinates of key points of the driver's hands and head from it; The region of interest module is configured to initially determine the region of interest related to the driver's behavior in the input video frame by combining circular regions centered on the driver's two hand and head key points and with a set value as the radius. The attention mask module is configured to generate an attention mask of the same size as the input video frame, and to overlay the attention mask onto the corresponding input video frame to obtain a video frame sequence with the attention mask. The result prediction module is configured to obtain skeleton flow prediction results using skeleton keypoint sequences and adaptive graph convolutional neural networks; and to obtain semantic flow prediction results using video frame sequences with attention masks and slow semantic networks. The behavior recognition result module is configured to take skeleton flow prediction results and semantic flow prediction results as input, and use a linear fusion strategy to fuse the behavior prediction results to obtain the final driver behavior recognition result.

Images