C3D improvement method and machine shop human behavior recognition method

Abstract

The application provides a C3D improvement method and a machine tool workshop human behavior recognition method.The C3D improvement method: referring to a channel excitation module CE in an ACTION-Net algorithm, the CE is improved to obtain an improved channel excitation module ICE, and the channel information of the C3D convolution network for extracting features is excited and perceived by using the ICE; the ICE is embedded into the C3D convolution network to obtain an improved C3D convolution network IC3D, and the IC3D is used for machine tool workshop human behavior recognition; the machine tool workshop human behavior recognition method: the time feature and the space feature of the machine tool workshop human video segment data are extracted by using the IC3D, the dimension of the extracted feature is changed, and the full connection layer is input, and the behavior recognition is realized according to the output of the full connection layer.The beneficial effects of the application are that the IC3D network training iteration number is reduced under the condition of maintaining the C3D behavior recognition precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to recognizing human behaviors from video data. BACKGROUND

[0002] The behavior recognition technology based on traditional machine learning usually needs to manually extract features, and the application field is greatly limited. In the field of behavior recognition based on deep learning, the time feature and the space feature of the video data are extracted by using a convolution network, the video data is input into a network model, and the time feature and the space feature of the video data are automatically extracted by convolution operation. Compared with the traditional method, the convolution network significantly improves the feature extraction efficiency of the video data and reduces the feature extraction difficulty of the video data. The behavior recognition method based on deep learning gradually replaces the behavior recognition technology based on traditional machine learning. The existing C3D convolution network is only composed of a convolution layer, a pooling layer, an activation function and a full connection layer, and the extracted features are not stimulated and perceived, resulting in slow convergence and long training difficulty time. Therefore, it is necessary to improve the C3D convolution network to reduce the training time. SUMMARY

[0003] The main purpose of the present application is to improve the channel excitation module CE of ACTION-Net, embed the improved channel excitation module ICE into the C3D convolution network, and improve the C3D convolution network. The improved C3D convolution network IC3D improves the training convergence speed of the C3D convolution network and reduces the training time. The channel represents the color channel of the image, the color channel number of the color image is 3, and the color channel of the gray image is 1. The channel is the color channel of the image, the channel number of the color image is 3, and the color channel of the gray image is 1. The excitation is to increase the channel number of the image by channel convolution operation to extract the multi-channel features of the image. The output image of the feature extraction is called a feature map. The channel convolution operation increases the channel number of the image, and the channel number of the feature map after the convolution operation is 2 raised to the power of n, where n is the number of convolution kernels. When n = 5, the channel number of the feature map obtained by the convolution operation is 2 5 = 32, when n = 6, the channel number of the feature map obtained by the convolution operation is 2 6 = 64, and when n = 10, the channel number of the feature map obtained by the convolution operation is 2 10 = 1024. The improved channel excitation module ICE network structure is as shown in Figure 1

[0004] Figure 1 In the formula, Input is an input feature map, the shape of which is (N, T, C, H, W), where N represents the number of input data, T represents the number of video frames extracted from a single data, C represents the color channel of the image, H represents the height of the video frame image, and W represents the width of the video frame image.​

[0005] (a) Pooling operation: The pooling operation can be represented by equation (1), and the pooling operation obtains a feature map F1 with a shape of (N, T, C, 1, 1),

[0006] (1)

[0007] (b) 3D convolution operation: The 3D convolution operation is performed on the feature map F1, and the convolution operation can be represented by equation (2), and the convolution operation obtains a feature map F2 with a shape of (N, T, C, 1, 1),

[0008] (2)

[0009] In equation (2), K1 represents a 3D convolution operation with a convolution kernel of (1, 1, 1), and F2 represents .

[0010] (c) Shape change: The shape of the feature map F2 is changed to (N, T, C) to obtain a feature map .

[0011] (d) 1D convolution operation: The 1D convolution operation is performed on the feature map , and the 1D convolution operation can be represented by equation (3), and the convolution operation obtains a feature map F3 with an unchanged shape of (N, T, C),

[0012] (3)

[0013] In equation (3), K2 represents a 1D convolution operation, .

[0014] (e) Shape change: The shape of the feature map F3 is changed to (N, T, C, 1, 1) to obtain .

[0015] (f) 3D convolution operation: The 3D convolution operation is performed on , and the 3D convolution operation can be represented by equation (4), and the convolution operation obtains a feature map F4 with a shape of (N, T, C, 1, 1),

[0016] (4)

[0017] In equation (4), K3 represents a 3D convolution operation with a convolution kernel of (1, 1, 1), .

[0018] (g) Activation: The feature map F4 is activated by a Sigmoid activation function to obtain F5.

[0019] F5 = Sigmoid(F4) (5)

[0020] (h) Element-wise multiplication: multiply the feature map F5 with the original Input element-wise to obtain F6.

[0021] F6 = F5 O Input (6)

[0022] (i) Element-wise addition: multiply the feature map F6 with the original Input element-wise to obtain Output, then .

[0023] Output = F6 O Input (7)

[0024] The channel excitation module ICE is implemented in Python language, the ICE is embedded into the C3D convolutional network to obtain an improved IC3D convolutional network, and a structure diagram of the IC3D convolutional network is shown in Figure 2 . BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The channel excitation module ICE structure diagram shown in the summary of the application is shown.

[0026] Figure 2 The convolutional network IC3D structure diagram shown in the summary of the application is shown. DETAILED DESCRIPTION

[0027] In the exemplary embodiment of the application, the convolutional network IC3D identifies the human behavior in the machine shop, which includes the following steps:

[0028] After the improved channel excitation module ICE is embedded into the first pooling layer of the C3D convolutional network, the convolution kernel parameter is (3, 3, 3), the number of convolution kernels is 64, and the input data channel number of the channel excitation module ICE is 64 at this time.

[0029] 1. Experimental hardware configuration: dual-core Intel(R) Xeon(R) CPU Gold 5115, memory size is 32GB, 1 block Quadro P4000 graphics card, and display memory is 8GB. The software configuration is Windows10, the programming language is Python3.7, and the network is built using the Pytorch-1.7.1+cu101 framework.

[0030] 2. Parameter setting: the number of training times of all samples epochs = 120, the number of samples in each batch batchsize = 16, the initial learning rate is α = 0.0001, the momentum parameter is momentum = 0.9, and the weight decay factor is γ = 0.0005.

[0031] 3. Data set

[0032] The IC3D convolutional network is pre-trained on the disclosed data set UCF-101, which can be divided into 5 large categories of actions: sports, human-human interaction, human-object interaction, playing musical instruments and individual limb actions. The 101 categories of subdivided actions in the 5 large categories of actions include horse riding, archery and jumping, etc. The data set has a total of 13320 video segments, a total length of 27 hours, a shortest video segment length of 1.06s, a longest video segment length of 71.04s, an average length of 7.21s, a frame rate of 25fps and a resolution of 320x240. The video material is sourced from YouTube, and the video format is avi.

[0033] The engineering video data is recorded on the basis of human actions in a machine shop, and there are three action categories, namely running, dozing off and playing. The data set has a total of 138 video segments, a resolution of 1080x1920, a total length of 7 minutes and 14 seconds, an average length of 3.14s and a video format of mp4. Among them, the number of videos with a frame rate of 59fps is 51, the number of videos with a frame rate of 30fps is 37, and the number of videos with a frame rate of 30.03fps is 50.

[0034] 4. Training and identification results

[0035] After the IC3D network is trained on the UCF-101 data set, the network parameters are fine-tuned for behavior recognition in engineering video data. While maintaining an identification accuracy of 96.8%, IC3D only needs 50 iterations of training, while C3D needs 320 iterations of training, and the convergence speed of IC3D training is significantly improved.

Claims

1.A method for improving a channel excitation module, characterized in that: An improved channel excitation module (ICE) is obtained by improving a channel excitation module (CE) in an ACTION-Net algorithm, and the ICE is used to excite the channel information of the features extracted by a perception C3D convolutional network; The ACTION-Net algorithm is a convolutional network module for action recognition; The channel is a color channel of an image, and the number of color channels of a color image is 3, and the color channel of a grayscale image is 1; The excitation is to increase the number of channels of an image by channel convolution operation, so as to extract multi-channel features of the image, and the output image of feature extraction is called a feature map; The CE is a short name of a channel excitation module; The ICE is a short name of an improved channel excitation module; The method for improving the channel excitation module comprises the following steps: S11, performing a max-pooling operation on a feature map input into the channel excitation module; S12, performing a 3D convolution operation on the feature map after the pooling operation; S13, changing the shape of the feature map after the pooling operation, so that the shape of the feature map is (N, T, C), N represents the number of input data, T represents the number of video frames extracted from a single data, and C represents an image color channel; S14, performing a 1D convolution operation on the feature map after the shape change; S15, changing the shape of the feature map after the 1D convolution operation; S16, performing a 3D convolution operation on the feature map after the shape change; S17, scaling the feature map after the 3D convolution operation by a Sigmoid activation function; S18, multiplying the scaled feature map with the original input feature map element by element; S19, adding the feature map multiplied element by element with the original input feature map element by element. 2.A method for improving C3D, characterized in that: An improved C3D convolutional network (IC3D) is obtained by embedding the ICE into the C3D convolutional network, and the IC3D is used for human behavior recognition in a machine shop; The C3D is a 3D convolutional network for video feature extraction; The IC3D is a short name of an improved C3D convolutional network (Improved Convolution 3D, IC3D); The ICE is embedded into the C3D convolutional network by using Python language to program the ICE in claim 1 as a program module and adding it after the first pooling layer of the C3D convolutional network. 3.A method for human behavior recognition in a machine shop, characterized in that: The IC3D is used to extract the time features and spatial features of human video segment data in a machine shop, the dimension of the extracted features is changed, and the changed features are input into a fully connected layer, and the behavior recognition is realized according to the output of the fully connected layer; The method for human behavior recognition in a machine shop comprises the following steps: S31, training the IC3D network with a training data set to obtain network model parameters; S32, loading the network model parameters, and testing the IC3D with test data; The IC3D network is trained by using the training data set to obtain network model parameters, including the following steps: S3101, loading the public data set UCF-101, using the first 60% of the video data of the UCF-101 data set to train the network model, using the middle 20% of the video data to verify the network model, and using the last 20% of the video data to test the network model; S3102, setting the total number of sample training times epochs=120, the number of samples in each batch batchsize=16, the initial learning rate α=0.0001, the momentum parameter momentum=0.9, and the weight decay factor γ=0.0005; S3103, data preprocessing, extracting 16 frames of images in each video segment data for stacking to obtain a training data with a shape of 5; S3104, inputting the features extracted by the network into the full connection layer; S3105, performing Softmax function operation on the features output by the full connection layer to map the feature values to between 0 and 1; S3106, selecting the behavior class corresponding to the maximum value after the feature value mapping as the network training result, and saving the model parameters; Loading the model parameters and testing the IC3D by using the test data includes the following steps: S3201, loading the trained model parameters into the improved IC3D network; S3202, inputting the test video data into the IC3D network for testing; S3203、According to the formula The network model precision is evaluated, wherein P represents the test precision, x i represents the input data, y i is the label of x i , f represents the network model IC3D, f(x i ) represents the identification result of the network model IC3D on x i , and n represents the number of input data.

Images