A non-driving posture recognition and lightweight method based on 3D convolution

Abstract

A non-driving posture recognition method based on 3D convolution relates to the field of traffic safety technology. To address the problem that existing recognition methods cannot accurately identify the driver's non-driving state, this application converts video data into image data, and then identifies the driver's non-driving posture based on a neural network by fusing and weighting global features and local features, thereby improving the recognition accuracy of non-driving posture.

Description

Technical Field

[0001] The present invention relates to the field of traffic safety technology, and in particular to a non-driving posture recognition and lightweight method based on 3D convolution. Background Art

[0002] The rapid development of autonomous driving technology has effectively alleviated driving stress for drivers. Under Level 3 autonomous driving conditions, drivers can engage in non-driving tasks unrelated to driving and adopt various non-driving postures while the vehicle is in operation. Furthermore, non-driving posture is a key factor influencing the performance and safety of L3 autonomous driving takeovers, making it necessary to identify the driver's non-driving posture. In emerging L3 autonomous driving scenarios, the driver's non-driving state is uncertain. Existing methods for identifying the driver's non-driving state require further improvement in terms of accuracy and timeliness. Summary of the Invention

[0003] The purpose of the present invention is to propose a non-driving posture recognition and lightweight method based on 3D convolution to address the problem that existing recognition methods cannot accurately identify the non-driving state of the driver.

[0004] The technical solution adopted by the present invention to solve the above technical problems is:

[0005] A non-driving posture recognition and lightweight method based on 3D convolution, the method comprising the following steps:

[0006] Step 1: Obtain non-driving posture video data;

[0007] Step 2: Convert the non-driving posture video data into images with coherent postures, and label the non-driving posture categories in the images. Finally, use the labeled images to train the non-driving posture recognition network to obtain a trained non-driving posture recognition network.

[0008] Step 3: Collect video data under autonomous driving conditions and obtain images with coherent postures according to step 2. Then, input the images with coherent postures into the trained non-driving posture recognition network to identify non-driving postures.

[0009] The non-driving posture recognition network includes an input module, a feature fusion module, a feature weighting module and an output module;

[0010] The input module is used to extract global features and key point features from the input image with coherent posture. The input module includes two branches, one of which uses a pooling layer to extract global features. , the other branch extracts key point features through multi-layer convolution ;

[0011] The feature fusion module is used to combine the global features and key point features Fusion is performed to obtain fusion features ;

[0012] The feature weighting module is used to fuse features As input, and using dynamic convolution, the convolution feature FID is obtained;

[0013] The output module is based on the convolution feature F ID , and use the SoftMax classifier to classify non-driving postures and output the class with the highest probability.

[0014] Furthermore, the global feature Expressed as:

[0015]

[0016] Among them, G represents the global channel extraction information, Represents global features, BN represents Batch normalization, conv j represents the jth convolutional layer, j=1,2, AP represents the average pooling layer, Represents the ReLu activation function.

[0017] Furthermore, the key point features Expressed as:

[0018]

[0019] in, Represents the local features of the i-th key point.

[0020] Furthermore, the fusion feature Expressed as:

[0021] .

[0022] Furthermore, the convolution feature F ID Expressed as:

[0023]

[0024]

[0025] Among them, F represents the input feature of the module, f i Represents the feature corresponding to the i-th convolution kernel in the input feature, Fc represents the fully connected layer, α i Represents the weight vector value of the i-th convolution kernel generated by the fully connected layer, W iRepresents the corresponding convolution kernel original weight, i=1,2...n, n represents the number of convolution kernels, Represents the Sigmoid activation function.

[0026] Furthermore, the non-driving posture recognition network is a lightweight non-driving posture recognition network, and the specific steps of lightweighting are:

[0027] The depthwise separable convolution technology is used to split the convolution layer of the non-driving posture recognition network into a channel-by-channel convolution layer and a point-by-point convolution layer. After the channel-by-channel convolution layer convolves each channel, all channels are stacked, and the point-by-point convolution layer convolves the stacked channels.

[0028] Furthermore, the specific steps of step one are:

[0029] Step 1: Use the long videos from the Drive&Act data source as a benchmark dataset. The data source includes 12 autonomous driving subtasks, each of which includes a non-driving driver posture.

[0030] Step 1 and 2: Split each long video into several short video segments of 2 seconds duration;

[0031] Step 13: Count the non-driving postures of the driver in the short video segment and select the 8 most common non-driving postures of the driver;

[0032] Step 14: Filter short video segments that contain only one type of driver's non-driving posture, and count the short video segments corresponding to each driver's non-driving posture to obtain the number of short video segments corresponding to each driver's non-driving posture. The number of the driver's non-driving posture with the least number of short video segments is used as the basis, and the short video segments corresponding to the remaining driver's non-driving postures are deleted to make the number of short video segments corresponding to the eight driver's non-driving postures equal.

[0033] Furthermore, the specific steps of converting the video data into images with coherent postures in step 2 are:

[0034] For each non-driving posture category selected, each small video segment is used as an independent sample. Frames of 2-second continuous videos are extracted, resized to 128×171, saved, and converted into images with coherent postures. Each independent sample contains at least 16 images.

[0035] Furthermore, the key point features include key point features of the head, hands, elbows and mouth, and the key point features are extracted by the OpenPose key point positioning algorithm.

[0036] Furthermore, the non-driving posture includes waiting for driving, picking up items sideways, operating the central control screen, reading materials, looking at a mobile phone, operating a laptop, talking on the phone, and eating.

[0037] The beneficial effects of the present invention are:

[0038] This application converts video data into image data, and then identifies the driver's non-driving posture based on a neural network by fusing and weighting global features and local features, thereby improving the recognition accuracy of non-driving posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the overall structure of this application. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the various embodiments disclosed in this application can be combined with each other.

[0041] Specific implementation method 1: refer to Figure 1 Specifically describing this embodiment, a non-driving posture recognition and lightweight method based on 3D convolution described in this embodiment includes the following steps:

[0042] Step 1: Obtain non-driving posture video data;

[0043] Step 2: Convert the non-driving posture video data into images with coherent postures, and label the non-driving posture categories in the images. Finally, use the labeled images to train the non-driving posture recognition network to obtain a trained non-driving posture recognition network.

[0044] Step 3: Collect video data under autonomous driving conditions and obtain images with coherent postures according to step 2. Then, input the images with coherent postures into the trained non-driving posture recognition network to identify non-driving postures.

[0045] The non-driving posture recognition network includes an input module, a feature fusion module, a feature weighting module and an output module;

[0046] The input module is used to extract global features and key point features from the input image with coherent posture. The input module includes two branches, one of which uses a pooling layer to extract global features. , the other branch extracts key point features through multi-layer convolution ;

[0047] The feature fusion module is used to combine the global features and key point features Fusion is performed to obtain fusion features ;

[0048] The feature weighting module is used to fuse features As input, and using dynamic convolution, the convolution feature FID is obtained;

[0049] The output module is based on the convolution feature F ID , and use the SoftMax classifier to classify non-driving postures and output the class with the highest probability.

[0050] The specific steps of step one are:

[0051] ① Using long videos from the Drive&Act data source as a benchmark database, the data source includes 12 autonomous driving tasks;

[0052] ② To accurately extract and classify non-driving posture data from long videos, we conducted multiple trials using Format Factory software and determined that the minimum video sample length was 2 seconds.

[0053] ③ Import all long video files into the Format Factory software platform and split each long video into several short video segments of 2 seconds duration;

[0054] ④ For each original long video, a dedicated folder is created to properly store all the video clips obtained from the video cutting;

[0055] ⑤ Eight representative non-driving postures were selected for study, including waiting for the vehicle to be driven, picking up an item sideways, operating the central control screen, reading materials, looking at a mobile phone, operating a laptop, talking on the phone, and eating. These eight postures were coded with numbers from 1 to 8.

[0056] ⑥ Manually review and clean the 2-second video clips, strictly select valid video clips that meet the research requirements, and reasonably delete the relevant videos of certain postures with excessive numbers to ensure the balance of the sample size of each type of posture.

[0057] ⑦ Organize the filtered 2s video clips into a folder according to the posture category to form a video clip dataset, namely the NDGF (Non-driving Gesture Fragments) dataset.

[0058] In step 2, the non-driving posture video data is converted into images with coherent postures as follows:

[0059] In each non-driving posture category selected, each video segment is used as an independent sample. Frames of 2-second continuous videos are extracted, resized to 128×171, saved, and converted into images with coherent postures. Each independent sample contains at least 16 images, and a sample set is constructed.

[0060] The representation of a neural network is:

[0061] ① Input data

[0062] The video content of the 3D convolutional neural network input can be represented by a tensor with dimensions c×f×h×w, where c represents the number of channels of the video, f represents the total number of frames, and h and w correspond to the vertical resolution and horizontal resolution of the frame, respectively.

[0063] ②Convolution kernel

[0064] A three-dimensional convolution kernel can directly extract temporal features from input data. The kernel size is expressed as t×x×y, where t is the time parameter and x and y are the kernel's height and width, respectively. Generally, in convolution operations, the kernel shape tends to be square or cube-shaped, meaning that t = x = y. Using a 3×3×3 kernel in the feature extraction convolution layer is most effective for extracting features from the input data.

[0065] ③ Output data

[0066] In the convolution operation, each element of the input data is multiplied by its corresponding weight one by one, and then these products are summed to calculate the value of a specific position in the convolution layer output data, as shown in formula (1).

[0067] (1)

[0068] Where w mnu Represents the elements of the input data, v mnu Indicates the weight corresponding to the element.

[0069] The non-driving posture recognition network in step 3 is specifically:

[0070] ① Input data

[0071] 3×16×112×112 data is used as the input of the 3D convolutional neural network, indicating that the input image is a 3-channel RGB image containing 16 frames, each of size 112×112, where 112×112 means that a rectangular box is used to add jitter to the sample with a resolution of 128×171 to randomly crop it.

[0072] A 3D convolutional neural network is used to extract global features of continuous frame images in the video, and the OpenPose key point positioning algorithm is used to extract local features of the head, hands, elbows and mouth.

[0073] ② Feature fusion module

[0074] The feature fusion module consists of two branches: one uses a pooling layer to extract the attention of global features, and the other uses multi-layer convolution to extract the channel attention of keypoint features. Different convolutional layers in the network are used to extract different features. The calculation formulas for global features and keypoint local features are shown in (2) and (3).

[0075] (2)

[0076] (3)

[0077] Among them, G represents the global channel extraction information, Represents global features, BN represents Batch normalization, convj represents the jth convolutional layer, j=1,2, AP represents the average pooling layer, represents the ReLu activation function, Represents the local features of the i-th key point.

[0078] The fusion features of local features and global features, that is, the output calculation formula of the feature fusion module is shown in (4).

[0079] (4)

[0080] ③ Feature weighting module

[0081] Different non-driving posture images require different keypoint features. To enhance useful keypoint features and suppress irrelevant ones, a feature weighting module is proposed to recalibrate the activation strengths of different keypoint feature channels. This module automatically selects different convolutional filters for classification calculations based on differences in the input images (such as classification difficulty and driver action type).

[0082] The weighted module sets multiple convolution kernels in the convolution layer. The weight of each convolution kernel is determined by the input features of the convolution layer through the fully connected layer. The features after the fusion of global features and key point features are used as the input of this module. Dynamic convolution is used to learn the importance of different key point channel features and generate corresponding weights α. i, a set of convolution kernels corresponding to the input is obtained by weighted summation, and then a convolution is performed. The fully connected layer realizes the conversion from the fusion feature map to the n convolution kernel weights. The calculation of the feature weights is shown in formula (5). After the convolution kernel weights are dynamically generated, they are combined with the corresponding original parameters to generate a new convolution kernel. The convolution feature F generated by the feature weighting module ID The calculation is shown in formula (6).

[0083] (5)

[0084] (6)

[0085] In the formula, F represents the input feature of the module, Fc represents the fully connected layer, α i Represents the weight vector value of the i-th convolution kernel generated by the fully connected layer, W i Represents the corresponding convolution kernel original weight, i=1,2...n, n represents the number of convolution kernels, Represents the Sigmoid activation function.

[0086] ④ Output data (classifier)

[0087] The output layer is a SoftMax classifier composed of the same units as the non-driving posture category. Each unit is fully connected to the feature weighting module, and a linear classifier (SoftMax) is applied to classify the non-driving posture and output the category with the highest probability.

[0088] The technical solution of this application also includes the steps of lightweighting the non-driving posture recognition network, specifically:

[0089] Based on the lightweight method of deep neural network that combines the maximum pooling layer with the convolution layer, depth-wise separable convolution, and stacked convolution, the number of parameters and computational complexity of the driver's non-driving posture recognition model are reduced.

[0090] ① Depthwise separable convolution lightweight

[0091] Depthwise separable convolution technology splits the traditional convolution process into two independent stages: depth-wise convolution, which performs independent convolution on each input channel; point-wise convolution, which fuses channels based on channel-by-channel convolution. Channel-by-channel convolution implements filtering function, while point-wise convolution implements superposition function.

[0092] The input dimension is c×f×h×w, and the output dimension is , the convolution kernel is , where c is the number of channels (depth) of the input data feature map, f is the number of input data feature maps, h and w are the height and width of the input data feature map in space, respectively. is the number of channels (depth) of the output data feature map, is the number of output data feature maps, and are the height and width of the output data feature map in space, is the single side length of the convolution kernel.

[0093] The first step is to divide the input data into c input channels, and perform channel-by-channel convolution on each channel with a dimension of The convolution calculation is then performed, and the c channels after channel-by-channel convolution are stacked, and the point-by-point convolution is performed again with a dimension of Convolution calculation.

[0094] Assuming the step size of the depth convolution is 1, the number of parameters of a depth convolution is and computational complexity It can be obtained from equations (7) and (8).

[0095] (7)

[0096] (8)

[0097] The default step size of point-by-point convolution is 1, so the number of parameters of a point-by-point convolution is and computational complexity It can be obtained from formula (9) and formula (10).

[0098] (9)

[0099] (10)

[0100] The number of parameters for a complete depth-wise separable convolution and computational complexity It can be obtained from formula (11) and formula (12).

[0101] (11)

[0102] (12)

[0103] ② Stacked convolution lightweight

[0104] In convolutional neural networks, the receptive field is the computational range of a convolution kernel's movement. Its size is closely related to the network's overall feature extraction capabilities. By applying stacked convolution, the receptive field of each convolution calculation can be aligned with the target sample, accurately capturing the characteristic information of non-driving postures.

[0105] The side length of the convolution kernel corresponding to the small receptive field is D k , let the number of stacked layers be n, the side length of the convolution kernel corresponding to the equivalent large receptive field is It can be calculated by formula (13).

[0106] (13) Example

[0107] Step 1: Fifteen participants were recruited and simulated in an autonomous driving environment using a driving simulator. Each participant participated in two independent autonomous driving experiments. The experiments involved eight non-driving postures: waiting for the vehicle to move, leaning over to pick up an object, operating the central control screen, reading, looking at a mobile phone, operating a laptop, talking on a mobile phone, and eating. A camera was mounted on the passenger side, facing the driver, to record the driver's postures. A total of 30 long videos were captured during the experiment. These 30 long videos were segmented into 2-second segments. The non-driving postures in all 2-second segments were categorized, balancing the relative number of each type of posture. The selected video segments were then saved to the same folder by posture category.

[0108] Step 2: Under each pose category, set each video segment as a separate folder, extract the frames of the 2-second continuous video, resize them to 128×171, which is about 1 / 3 of the original video resolution, and save them as images with coherent poses.

[0109] Step 3: When feeding the recognition network, 16 non-overlapping images from each saved independent folder are selected as input. The temporal and spatial information of the data is also extracted. Specifically, 3×16×112×112 data is used as the input for the 3D model, where 112×112 represents random cropping using a rectangular box with jitter added to a sample with a resolution of 128×171. A 3D convolutional neural network is used to extract global features from consecutive video frames. Local features of the head, hands, elbows, and mouth are extracted using the OpenPose keypoint localization algorithm. The global and local features are then fused and weighted.

[0110] Taking the "hand-held phone call" non-driving posture as an example, after video segmentation and classification in step 1, the posture folder contains 266 2-second video clips. After processing each 2-second video in step 2, 266 folders are created within the posture category folder, each containing 19 images of size 128×171. 16 frames of images are selected from each of the 266 independent folders as input data for the recognition network, with a data size of 3×16×112×112. When performing feature weighting, the classification in this scenario focuses more on the mouth, elbows, and hands, thereby reducing visual activation of the head feature channel.

[0111] The constructed 3D convolutional network non-driving posture recognition model was compared with a classic 2D network, and the recognition accuracy of each model was calculated. The results showed that the 3D recognition model achieved an accuracy of 96.26%, a 24.76% improvement over VGG-16 and a 9.24% improvement over ResNet-101.

[0112] Step 4: Based on the merging of maximum pooling layer and convolution layer, depthwise separable convolution and stacked convolution, lightweight recognition of driver non-driving posture is achieved by 3D convolutional neural network.

[0113] A comparative analysis of the recognition parameter count, speed, and accuracy of the constructed lightweight 3D convolutional network non-driving posture recognition model and a non-lightweight 3D recognition model was conducted. The results showed that the accuracy of the lightweight model decreased by 0.93% compared to the original model, but this decrease was within an acceptable range. The parameter count decreased from 78.03M to 3.91M after lightweighting. Furthermore, the recognition speed of the lightweight 3D model increased from 48 frames per second to 164 frames per second.

[0114] Under Level 3 autonomous driving conditions, this application cleans and classifies video data from the Drive&Act data source to form an NDGF video clip dataset. The data is then preprocessed to convert the video data into image data. Based on a 3D convolutional neural network, the driver's non-driving posture is identified by fusing and weighting global and local features, and the recognition model is lightweighted using convolution. The technical solution of this application can process video data collected under autonomous driving conditions to achieve non-driving posture recognition and lightweighting, which is beneficial for improving the accuracy of non-driving posture recognition and reducing the parameter scale of recognition.

[0115] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solutions of the present invention and cannot be used to limit the scope of protection. Any minor changes made based on the claims and description of the present invention shall still fall within the scope of protection of the present invention.

Claims

1. A non-driving posture recognition and lightweight method based on 3D convolution, characterized by The method comprises the following steps: Step 1: Obtain non-driving posture video data; Step 2: Convert the non-driving posture video data into images with coherent postures, and label the non-driving posture categories in the images. Finally, use the labeled images to train the non-driving posture recognition network to obtain a trained non-driving posture recognition network. Step 3: Collect video data under autonomous driving conditions and obtain images with coherent postures according to step 2. Then, input the images with coherent postures into the trained non-driving posture recognition network to identify non-driving postures. The non-driving posture recognition network includes an input module, a feature fusion module, a feature weighting module and an output module; The input module is used to extract global features and key point features from the input image with coherent posture. The input module includes two branches, one of which uses a pooling layer to extract global features. , the other branch extracts key point features through multi-layer convolution ; The feature fusion module is used to combine the global features and key point features Fusion is performed to obtain fusion features ; The feature weighting module is used to fuse features As input, and using dynamic convolution, we get the convolution feature F ID ; The output module is based on the convolution feature F ID , and use the SoftMax classifier to classify non-driving postures and output the class with the highest probability; The non-driving posture recognition network is a lightweight non-driving posture recognition network, and the specific steps of lightweighting are: The convolutional layer of the non-driving posture recognition network is split into a channel-by-channel convolution layer and a point-by-point convolution layer using depthwise separable convolution technology. After the channel-by-channel convolution layer performs convolution on each channel, all channels are stacked, and the point-by-point convolution layer performs convolution on the stacked channels. The global features Expressed as: Among them, G represents the global channel extraction information, Represents global features, BN represents Batch normalization, conv j represents the jth convolutional layer, j=1,2, AP represents the average pooling layer, Represents the ReLu activation function; The convolution feature F ID Expressed as: Among them, F represents the input feature of the module, f i Represents the feature corresponding to the i-th convolution kernel in the input feature, Fc represents the fully connected layer, α i Represents the weight vector value of the i-th convolution kernel generated by the fully connected layer, W i Represents the original weight of the corresponding convolution kernel, i=1,2...n, n represents the number of convolution kernels, Represents the Sigmoid activation function.

2. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 1 is characterized in that The key point features Expressed as: in, Represents the local features of the i-th key point.

3. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 2 is characterized in that The fusion features Expressed as: 。 4. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 1 is characterized in that The specific steps of step one are: Step 1: Use the long videos from the Drive&Act data source as a benchmark dataset. The data source includes 12 autonomous driving subtasks, each of which includes a non-driving driver posture. Step 1 and 2: Split each long video into several short video segments of 2 seconds duration; Step 13: Count the non-driving postures of the driver in the short video segment and select the 8 most common non-driving postures of the driver; Step 14: Filter short video segments that contain only one type of driver's non-driving posture, and count the short video segments corresponding to each driver's non-driving posture to obtain the number of short video segments corresponding to each driver's non-driving posture. The number of the driver's non-driving posture with the least number of short video segments is used as the basis, and the short video segments corresponding to the remaining driver's non-driving postures are deleted to make the number of short video segments corresponding to the eight driver's non-driving postures equal.

5. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 4 is characterized in that The specific steps of converting the video data into images with coherent postures in step 2 are: For each non-driving posture category selected, each small video segment is used as an independent sample. Frames of 2-second continuous videos are extracted, resized to 128×171, saved, and converted into images with coherent postures. Each independent sample contains at least 16 images.

6. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 1 is characterized in that The key point features include key point features of the head, hands, elbows and mouth, and the key point features are extracted by the OpenPose key point positioning algorithm.

7. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 1 is characterized in that The non-driving postures include waiting for driving, picking up items sideways, operating the central control screen, reading materials, looking at the mobile phone, operating a laptop, talking on the phone, and eating.

Images