Non-driving posture recognition method based on 3D convolution

Abstract

The invention discloses a non-driving posture recognition method based on 3D convolution, relates to the technical field of traffic safety, and aims to solve the problem that an existing recognition method cannot accurately recognize the non-driving state of a driver. The non-driving posture of the driver is recognized through fusion and weighting of the global features and the local features, and the recognition accuracy of the non-driving posture is improved.

Description

Technical Field

[0001] The present invention relates to the field of traffic safety technology, and in particular to a non-driving posture recognition method based on 3D convolution. Background Art

[0002] The rapid development of autonomous driving technology has effectively relieved the driving pressure of drivers. Under L3 autonomous driving conditions, drivers can engage in non-driving tasks unrelated to driving during vehicle operation and be in different non-driving postures. At the same time, non-driving posture is an important potential factor that determines the performance and safety of L3 autonomous driving takeover. Therefore, it is necessary to identify the driver's non-driving posture. In the emerging scenario of L3 autonomous driving, the driver's non-driving state is uncertain. However, the existing driver's non-driving state recognition method needs to be further improved in terms of accuracy and timeliness. Summary of the invention

[0003] The purpose of the present invention is to propose a non-driving posture recognition method based on 3D convolution to address the problem that existing recognition methods cannot accurately recognize the non-driving state of the driver.

[0004] The technical solution adopted by the present invention to solve the above technical problems is:

[0005] A non-driving posture recognition method based on 3D convolution, the method comprising the following steps:

[0006] Step 1: Obtain non-driving posture video data;

[0007] Step 2: Convert the non-driving posture video data into images with coherent postures, and mark the non-driving posture categories in the images. Finally, use the marked images to train the non-driving posture recognition network to obtain a trained non-driving posture recognition network.

[0008] Step 3: Collect video data under the condition of autonomous driving, and obtain images with coherent postures according to step 2, and then input the images with coherent postures into the trained non-driving posture recognition network to recognize the non-driving postures;

[0009] The non-driving posture recognition network comprises an input module, a feature fusion module, a feature weighting module and an output module;

[0010] The input module is used to extract global features and key point features from the input image with a coherent posture. The input module includes two branches, one of which uses a pooling layer to extract the global feature G(f g ), the other branch extracts key point features through multi-layer convolution

[0011] The feature fusion module is used to transform the global feature G(f g ) and key point features Fusion is performed to obtain the fusion feature F i ;

[0012] The feature weighting module is based on the fusion feature F i As input, and using dynamic convolution, the convolution feature FID is obtained;

[0013] The output module is based on the convolution feature F ID , and use the SoftMax classifier to classify non-driving postures and output the class with the highest probability.

[0014] Furthermore, the global feature G(f g ) is expressed as:

[0015] G(f g )=BN(conv2(δ(BN(conv1(AP(f g ))))))

[0016] Among them, G represents the global channel extraction information, f g represents global features, BN represents Batch normalization, conv j represents the jth convolutional layer, j = 1, 2, AP represents the average pooling layer, and δ represents the ReLu activation function.

[0017] Furthermore, the key point features It is expressed as:

[0018]

[0019] in, Represents the local features of the i-th key point.

[0020] Furthermore, the fusion feature F i It is expressed as:

[0021]

[0022] Furthermore, the convolution feature F ID It is expressed as:

[0023] F ID =σ((α1·W1+···+α n ·W n )*F)

[0024] α i =σ(Fc(AP(f i )))

[0025] Among them, F represents the input feature of the module, f i represents the feature corresponding to the i-th convolution kernel in the input feature, Fc represents the fully connected layer, α i Represents the weight vector value of the i-th convolution kernel generated by the fully connected layer, W i Represents the corresponding original weight of the convolution kernel, i=1,2...n, n represents the number of convolution kernels, and σ represents the Sigmoid activation function.

[0026] Furthermore, the non-driving posture recognition network is a lightweight non-driving posture recognition network, and the specific steps of lightweighting are:

[0027] The convolution layer of the non-driving posture recognition network is split into a channel-by-channel convolution layer and a point-by-point convolution layer by using the depthwise separable convolution technology. After the channel-by-channel convolution layer performs convolution on each channel, all channels are stacked, and the point-by-point convolution layer performs convolution on the stacked channels.

[0028] Furthermore, the specific steps of step one are:

[0029] Step 1: Use the long videos in the Drive&Act data source as the benchmark database. The data source includes 12 autonomous driving tasks, each of which contains a non-driving posture of the driver.

[0030] Step 1 and 2: Split each long video into several short video segments with a duration of 2 seconds;

[0031] Step 13: Count the non-driving postures of the driver in the short video segment, and select the 8 most numerous non-driving postures of the driver;

[0032] Step 14: Filter the short video segments that only contain one type of non-driving posture of the driver, and count the short video segments corresponding to each non-driving posture of the driver to obtain the number of short video segments corresponding to each non-driving posture of the driver, and delete the short video segments corresponding to the remaining non-driving postures of the driver based on the number of non-driving postures of the driver with the least number of short video segments, so that the number corresponding to the eight non-driving postures of the driver is equal.

[0033] Furthermore, the specific steps of converting the video data into an image with a coherent posture in step 2 are:

[0034] In each non-driving posture category selected, each small video segment is taken as an independent sample, and the frame images of the 2s continuous video are extracted, resized to 128×171 and saved, and converted into images with coherent postures. Each independent sample contains at least 16 images.

[0035] Furthermore, the key point features include key point features of the head, hands, elbows and mouth, and the key point features are extracted by an OpenPose key point positioning algorithm.

[0036] Furthermore, the non-driving posture includes waiting for driving, picking up items sideways, operating the central control screen, reading materials, looking at a mobile phone, operating a laptop computer, talking on the phone, and eating.

[0037] The beneficial effects of the present invention are:

[0038] This application converts video data into image data, and then identifies the driver's non-driving posture based on a neural network by fusing and weighting global features and local features, thereby improving the recognition accuracy of the non-driving posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the overall structure of this application. DETAILED DESCRIPTION

[0040] It should be particularly noted that, in the absence of conflict, the various embodiments disclosed in this application can be combined with each other.

[0041] Specific implementation method 1: refer to Figure 1 Specifically describing this embodiment, a non-driving posture recognition method based on 3D convolution described in this embodiment includes the following steps:

[0042] Step 1: Obtain non-driving posture video data;

[0043] Step 2: Convert the non-driving posture video data into images with coherent postures, and mark the non-driving posture categories in the images. Finally, use the marked images to train the non-driving posture recognition network to obtain a trained non-driving posture recognition network.

[0044] Step 3: Collect video data under the condition of autonomous driving, and obtain images with coherent postures according to step 2, and then input the images with coherent postures into the trained non-driving posture recognition network to recognize the non-driving postures;

[0045] The non-driving posture recognition network comprises an input module, a feature fusion module, a feature weighting module and an output module;

[0046] The input module is used to extract global features and key point features from the input image with a coherent posture. The input module includes two branches, one of which uses a pooling layer to extract the global feature G(f g ), the other branch extracts key point features through multi-layer convolution

[0047] The feature fusion module is used to transform the global feature G(f g ) and key point features Fusion is performed to obtain the fusion feature F i ;

[0048] The feature weighting module is based on the fusion feature F i As input, and using dynamic convolution, the convolution feature FID is obtained;

[0049] The output module is based on the convolution feature F ID , and use the SoftMax classifier to classify non-driving postures and output the class with the highest probability.

[0050] The specific steps of step one are:

[0051] ① Use the long videos in the Drive&Act data source as the benchmark database. The data source includes 12 autonomous driving tasks.

[0052] ② In order to accurately extract and classify non-driving posture data in long videos, Format Factory software was used to conduct multiple trials and experiments, and the minimum video sample length was determined to be 2 seconds;

[0053] ③ Import all long video files into the Format Factory software platform and divide each long video into several short video segments with a duration of 2 seconds;

[0054] ④ For each original long video, create a dedicated folder to properly save all the video clips cut from the video;

[0055] ⑤ Select 8 representative non-driving postures of drivers for research, including waiting for driving, picking up items sideways, operating the central control screen, reading materials, looking at the mobile phone, operating a laptop, talking on the phone, and eating. These 8 postures are replaced by 1 to 8 digital codes;

[0056] ⑥ Manually review and clean the 2s video clips, strictly select valid video clips that meet the research requirements, and reasonably delete the related videos of certain postures with too many videos to ensure the balance of the sample size of each type of posture. Finally, the number of video clips in each category is equal;

[0057] ⑦ Organize the selected 2s video clips into a folder according to the posture category to form a video clip dataset, namely the NDGF (Non-driving Gesture Fragments) dataset.

[0058] In step 2, the non-driving posture video data is converted into an image with a coherent posture as follows:

[0059] In each non-driving posture category selected, each video segment is taken as an independent sample, and the frame images of the 2s continuous video are extracted, resized to 128×171 and saved, and converted into images with coherent postures. Each independent sample contains at least 16 images, and then a sample set is constructed.

[0060] The representation of a neural network is:

[0061] ① Input data

[0062] The video content of the 3D convolutional neural network input can be represented by a tensor with dimensions c×f×h×w, where c represents the number of channels of the video, f represents the total number of frames, and h and w correspond to the vertical resolution and horizontal resolution of the frame, respectively.

[0063] ②Convolution kernel

[0064] The three-dimensional convolution kernel can directly extract the features of the time dimension from the input data. The size of the convolution kernel is expressed as t×x×y, where t is the time parameter, and x and y are the height and width of the convolution kernel respectively. Generally, in the convolution operation, the shape of the convolution kernel tends to be square or cube, that is, t=x=y. Using a convolution kernel of size 3×3×3 in the convolution layer of feature extraction is the best effect for extracting features from the input data.

[0065] ③ Output data

[0066] In the convolution operation, each element of the input data is multiplied by its corresponding weight one by one, and then these products are summed to calculate the value of a specific position in the convolution layer output data, as shown in formula (1).

[0067]

[0068] In the formula, w mnu Represents the elements of the input data, v mnu Indicates the weight corresponding to the element.

[0069] The non-driving posture recognition network in step 3 is specifically:

[0070] ① Input data

[0071] Using 3×16×112×112 data as the input of the 3D convolutional neural network means that the input image is a 3-channel RGB image, containing 16 frames, each with a size of 112×112, where 112×112 means randomly cropping by adding jitter to samples with a resolution of 128×171 using a rectangular box.

[0072] A 3D convolutional neural network is used to extract global features of continuous frame images of the video, and the OpenPose key point positioning algorithm is used to extract local features of the head, hands, elbows and mouth.

[0073] ②Feature fusion module

[0074] The feature fusion module consists of two branches. One branch uses the pooling layer to extract the attention of the global features, and the other branch extracts the channel attention of the key point features through multi-layer convolution. Different convolutional layers in the network are selected to extract different features. The calculation formulas for global features and key point local features are shown in (2) and (3).

[0075] G(f g )=BN(conv2(δ(BN(conv1(AP(f g )))))) (2)

[0076]

[0077] Among them, G represents the global channel extraction information, f g represents global features, BN represents Batch normalization, convj represents the jth convolutional layer, j = 1, 2, AP represents the average pooling layer, δ represents the ReLu activation function, Represents the local features of the i-th key point.

[0078] The fusion feature of local features and global features, that is, the output calculation formula of the feature fusion module is shown in (4).

[0079]

[0080] ③Feature weighting module

[0081] For different non-driving posture images, the key point features that need to be paid attention to are also different. In order to enhance the useful key point features and suppress irrelevant features, a feature weighting module is proposed to recalibrate the activation strength of different key point feature channels, and automatically select different convolution filters for classification calculation according to the differences in input images (classification difficulty, driver action category, etc.).

[0082] The weighted module sets multiple convolution kernels in the convolution layer, and the weight of each convolution kernel is determined by the input features of the convolution layer through the fully connected layer. The features after the fusion of global features and key point features are used as the input of this module, and the importance of different key point channel features is learned by dynamic convolution and the corresponding weight α is generated. i , a set of convolution kernels corresponding to the input is obtained by weighted summation, and then a convolution is performed. The fully connected layer realizes the conversion from the fusion feature map to the weights of n convolution kernels. The calculation of feature weights is shown in formula (5). After the convolution kernel weights are dynamically generated, they are combined with the corresponding original parameters to generate a new convolution kernel. The convolution feature F generated by the feature weighting module ID The calculation is shown in formula (6).

[0083] α i = r(f i )=σ(Fc(AP(f i ))) (5)

[0084] F ID =σ((α1·W1+···+α n ·W n )*F) (6)

[0085] In the formula, F represents the input feature of the module, Fc represents the fully connected layer, α i Represents the weight vector value of the i-th convolution kernel generated by the fully connected layer, W i Represents the corresponding original weight of the convolution kernel, i=1,2...n, n represents the number of convolution kernels, and σ represents the Sigmoid activation function.

[0086] ④Output data (classifier)

[0087] The output layer is a SoftMax classifier composed of the same units as the non-driving posture category. Each unit is fully connected to the feature weighting module, and a linear classifier (SoftMax) is applied to classify non-driving postures and output the class with the highest probability.

[0088] The technical solution of this application also includes the steps of lightweighting the non-driving posture recognition network, specifically:

[0089] Based on the lightweight method of deep neural network combining maximum pooling layer and convolution layer, depth-wise separable convolution and stacked convolution, the number of parameters and calculation amount of the driver's non-driving posture recognition model are reduced.

[0090] ① Depth-wise separable convolution is lightweight

[0091] The depth-wise separable convolution technology splits the traditional convolution process into two independent stages: channel-wise convolution, which performs independent convolution on each input channel; point-wise convolution, which fuses channels based on channel-wise convolution. Channel-wise convolution realizes filtering function, while point-wise convolution realizes superposition function.

[0092] The input dimension is c×f×h×w, the output dimension is c'×f'×h'×w', and the convolution kernel is D k ×D k ×D k, where c is the number of channels (depth) of the input data feature map, f is the number of input data feature maps, h and w are the height and width of the input data feature map in space, c' is the number of channels (depth) of the output data feature map, f' is the number of output data feature maps, h' and w' are the height and width of the output data feature map in space, D k is the single side length of the convolution kernel.

[0093] The first step is to divide the input data into c input channels, and perform channel-by-channel convolution on each channel with a dimension of D k ×D k ×D k ×1 convolution calculation, and then stack the c channels after channel-by-channel convolution, and perform point-by-point convolution on them with a dimension of 1×1×c×c'.

[0094] Assuming that the step size of the depthwise convolution is 1, the parameter amount Params and the computation amount Mult-Adds of a depthwise convolution can be obtained by equations (7) and (8).

[0095] Mult-Adds_depth=D k ×D k ×D k ×c×h×w (7)

[0096] Params_depth=D k ×D k ×D k ×c (8)

[0097] The default step size of the point-by-point convolution is 1, so the parameter amount Params and the calculation amount Mult-Adds of a point-by-point convolution can be obtained by equations (9) and (10).

[0098] Mult-Adds_point=c×c'×h×w (9)

[0099] Params_point = c × c' (10)

[0100] The parameter amount Params and computation amount Mult-Adds of a complete depth-wise separable convolution can be derived from equations (11) and (12).

[0101]

[0102] Params_depth&point=(D k ×D k ×D k +c')×c (12)

[0103] ② Stacked convolution is lightweight

[0104] In a convolutional neural network, the receptive field is the calculation range of the convolution kernel's movement at one time, and its size is closely related to the feature extraction capability of the entire network. By applying the stacked convolution method, the receptive field of each convolution calculation can be matched with the sample target to accurately capture the feature information of non-driving postures.

[0105] The side length of the convolution kernel corresponding to the small receptive field is D k , let the number of stacked layers be n, and the convolution kernel side length corresponding to the equivalent large receptive field is D k ' can be calculated by formula (13).

[0106] D k '=D k ×(n+1)-n (13)

[0107] Example

[0108] Step 1: Recruit 15 subjects, use a driving simulator to simulate the autonomous driving environment, and each person participates in two independent autonomous driving experiments. During the experiment, eight non-driving postures were set up, including waiting for driving, picking up objects sideways, operating the central control screen, reading materials, looking at the phone, operating a laptop, holding a phone call, and eating. The camera was set up in front of the co-pilot seat facing the driver to record the driver's posture. A total of 30 long videos were collected during the experiment. The 30 long videos were divided into several short video segments with a duration of 2s. The non-driving postures in all 2s video segments were classified, the relative number of each type of posture was balanced, and the selected video segments were saved in the same folder according to the posture category.

[0109] Step 2: Under each posture category, set each video segment as an independent folder, extract the frame of the 2s continuous video, resize it to 128×171, which is about 1 / 3 of the original video resolution, and save it as an image with a coherent posture.

[0110] Step 3: When inputting the recognition network, select 16 non-overlapping pictures from each saved independent folder as one input, and extract the spatiotemporal information of the data at the same time, that is, use 3×16×112×112 data as the input of the 3D model, where 112×112 means using a rectangular frame to add jitter to the sample with a resolution of 128×171 to randomly crop. Use a 3D convolutional neural network to extract the global features of the continuous frame images of the video, and use the OpenPose key point positioning algorithm to extract the local features of the head, hands, elbows and mouth, and perform fusion and weighting of global features and local features.

[0111] Taking the "hand-held call" non-driving posture as an example, after the video segmentation and classification in step 1, the posture folder contains 266 2s video clips. After each 2s video is processed in step 2, 266 folders are created under the posture category folder, each folder contains 19 pictures of size 128×171. 16 frames of images are selected from each of the 266 independent folders as input data for the recognition network, and the data size is 3×16×112×112. When performing feature weighting, the classification attention in this scene is more focused on the mouth, elbows and hands, so the visual activation of the head feature channel will be reduced.

[0112] The constructed 3D convolutional network non-driving posture recognition model was compared with the classic 2D network, and the recognition accuracy of each model was calculated. The results show that the accuracy of the 3D recognition model is 96.26%, which is 24.76% higher than that of VGG-16 and 9.24% higher than that of ResNet-101.

[0113] Step 4: Based on merging the maximum pooling layer with the convolution layer, depthwise separable convolution and stacked convolution, lightweight recognition of the driver's non-driving posture is achieved using a 3D convolutional neural network.

[0114] The recognition parameter quantity, speed and accuracy of the constructed lightweight 3D convolutional network non-driving posture recognition model and the non-lightweight 3D recognition model are compared and analyzed. The results show that the accuracy of the lightweight model is reduced by 0.93% compared with the model before lightweighting, but this decrease is within an acceptable range; in terms of parameter quantity, it is reduced from 78.03M before lightweighting to 3.91M after lightweighting; at the same time, the recognition speed of the lightweight 3D model is increased from 48 frames / second before lightweighting to 164 frames / second.

[0115] Under L3 autonomous driving conditions, this application cleans and classifies the video data in the Drive&Act data source to form an NDGF video clip data set, and pre-processes the data to convert the video data into image data. Then, based on a 3D convolutional neural network, the driver's non-driving posture is identified by fusing and weighting global and local features, and the recognition model is lightweighted using convolution. The technical solution of this application can process video data collected under autonomous driving conditions, realize the recognition and lightweighting of non-driving postures, and is conducive to improving the accuracy of non-driving posture recognition and reducing the scale of recognition parameters.

[0116] It should be noted that the specific implementation is only an explanation and description of the technical solution of the present invention, and cannot be used to limit the scope of protection of the rights. Any partial changes made according to the claims and description of the present invention should still fall within the scope of protection of the present invention.

Claims

1. A non-driving posture recognition method based on 3D convolution, characterized in that The method comprises the following steps: Step 1: Obtain non-driving posture video data; Step 2: Convert the non-driving posture video data into images with coherent postures, and mark the non-driving posture categories in the images. Finally, use the marked images to train the non-driving posture recognition network to obtain a trained non-driving posture recognition network. Step 3: Collect video data under the condition of autonomous driving, and obtain images with coherent postures according to step 2, and then input the images with coherent postures into the trained non-driving posture recognition network to recognize the non-driving postures; The non-driving posture recognition network comprises an input module, a feature fusion module, a feature weighting module and an output module; The input module is used to extract global features and key point features from the input image with a coherent posture. The input module includes two branches, one of which uses a pooling layer to extract the global feature G(f g ), the other branch extracts key point features through multi-layer convolution The feature fusion module is used to transform the global feature G(f g ) and key point features Fusion is performed to obtain the fusion feature F i ; The feature weighting module is based on the fusion feature F i As input, and using dynamic convolution, we get the convolution feature F ID ; The output module is based on the convolution feature F ID , and use the SoftMax classifier to classify non-driving postures and output the class with the highest probability.

2. The non-driving posture recognition method based on 3D convolution according to claim 1 is characterized in that The global feature G(f g ) is expressed as: G(f g )=BN(conv2(δ(BN(conv1(AP(f g )))))) Among them, G represents the global channel extraction information, f g represents global features, BN represents Batch normalization, conv j represents the jth convolutional layer, j = 1, 2, AP represents the average pooling layer, and δ represents the ReLu activation function.

3. The non-driving posture recognition method based on 3D convolution according to claim 2 is characterized in that The key point features It is expressed as: in, Represents the local features of the i-th key point.

4. The non-driving posture recognition method based on 3D convolution according to claim 3 is characterized in that The fusion feature F i It is expressed as:

5. The non-driving posture recognition method based on 3D convolution according to claim 4 is characterized in that The convolution feature F ID It is expressed as: F ID =σ((α1·W1+···+α n ·W n )*F) a i =σ(Fc(AP(f i ))) Among them, F represents the input feature of the module, f i represents the feature corresponding to the i-th convolution kernel in the input feature, Fc represents the fully connected layer, α i Represents the weight vector value of the i-th convolution kernel generated by the fully connected layer, W i Represents the corresponding original weight of the convolution kernel, i=1,2...n, n represents the number of convolution kernels, and σ represents the Sigmoid activation function.

6. The non-driving posture recognition method based on 3D convolution according to claim 1 is characterized in that The non-driving posture recognition network is a lightweight non-driving posture recognition network, and the specific steps of lightweighting are: The convolution layer of the non-driving posture recognition network is split into a channel-by-channel convolution layer and a point-by-point convolution layer by using the depthwise separable convolution technology. After the channel-by-channel convolution layer performs convolution on each channel, all channels are stacked, and the point-by-point convolution layer performs convolution on the stacked channels.

7. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 1 is characterized in that The specific steps of step one are: Step 1: Use the long videos in the Drive&Act data source as the benchmark database. The data source includes 12 autonomous driving tasks, each of which contains a non-driving posture of the driver. Step 1 and 2: Split each long video into several short video segments with a duration of 2 seconds; Step 13: Count the non-driving postures of the driver in the short video segment, and select the 8 most numerous non-driving postures of the driver; Step 14: Filter the short video segments that only contain one type of non-driving posture of the driver, and count the short video segments corresponding to each non-driving posture of the driver to obtain the number of short video segments corresponding to each non-driving posture of the driver, and delete the short video segments corresponding to the remaining non-driving postures of the driver based on the number of non-driving postures of the driver with the least number of short video segments, so that the number corresponding to the eight non-driving postures of the driver is equal.

8. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 7 is characterized in that The specific steps of converting the video data into an image with a coherent posture in step 2 are: In each non-driving posture category selected, each small video segment is taken as an independent sample, and the frame images of the 2s continuous video are extracted, resized to 128×171 and saved, and converted into images with coherent postures. Each independent sample contains at least 16 images.

9. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 1 is characterized in that The key point features include key point features of the head, hands, elbows and mouth, and the key point features are extracted by the OpenPose key point positioning algorithm.

10. The non-driving posture recognition and lightweight method based on 3D convolution according to claim 1, characterized in that The non-driving postures include waiting for driving, picking up items sideways, operating the central control screen, reading materials, looking at the mobile phone, operating a laptop computer, talking on the phone, and eating.

Images