A dual-stream network method based on compression excitation and skeleton points

Abstract

The application discloses a double-flow network method based on compression excitation and skeleton points, and the method comprises the following steps: constructing and training a spatial convolutional neural network; calculating a video optical flow, constructing and training a time flow network; freezing all weights of a backbone network of the spatial convolutional neural network except the last three convolutional layers; constructing compression excitation and skeleton point attention between the backbone network of the spatial convolutional neural network and a global average pooling layer; constructing a spatial flow network, wherein the spatial flow network comprises a YOLOX, a high-resolution network and the spatial convolutional neural network with compression excitation and skeleton point attention, and training the spatial flow network; using the trained time flow network and the spatial flow network to identify human behaviors, obtaining two network results, and performing post-fusion on the two network results to obtain a final human behavior recognition result. The application has high accuracy and good robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning and computer vision, and particularly relates to a dual-stream network method based on compression excitation and skeleton points. BACKGROUND

[0002] Human behavior recognition, as an important research direction in the field of machine learning and computer vision, has been widely used in intelligent monitoring, old people care, body sensing games and human-computer interaction scenes. Although the human behavior recognition method for simple scenes has basically matured, these methods may have performance problems when used in complex scenes. Therefore, it is necessary to further study the human behavior recognition method in complex scenes.

[0003] Studies have shown that, on the one hand, the convolutional neural network can obtain different spatial information through different sizes of convolution kernels; on the other hand, each channel of the convolution layer is a different feature detector, and processing each channel of the convolution layer in the same way may cause loss of network performance [1-2] . Hu et al. [3] proposed a compression excitation (Squeeze-and-Excitation) for recalibrating each channel of the convolution layer. The compression excitation can be simply added to the advanced network, and can work at each stage of the convolutional neural network. Howard et al. [4] In MobileNetV3, the Hard Sigmoid is used to replace the Sigmoid activation function in the compression excitation, which improves the computational efficiency and performance. Simonyan et al. [5] proposed a dual-stream network based on RGB color image stream and optical flow for human behavior recognition. Compared with the RGB color image information, the skeleton points contain information that is less sensitive to light, camera angle and background change [7] . Therefore, adding skeleton point information will help to improve the performance of the behavior recognition method. Although the behavior recognition method based on skeleton points shows excellent performance, the training of such method depends on the human behavior dataset with skeleton points, and the application scene is limited [7-8] .

[0004] In summary, the early convolutional neural network [9]The output of the backbone network is not specially processed, which can cause the network performance to be reduced when recognizing human behaviors in complex scenes. Therefore, the patent designs a double-flow network method, which can recalibrate each channel of the output of the backbone network and output the results. The backbone network refers to the part of the convolutional neural network except the last pooling layer and the fully connected layer. The patent aims to improve the performance of the behavior recognition model in complex scenes by increasing the skeleton point information, reduce the cost of human behavior dataset production, and expand the application scenarios of human behavior recognition method based on skeleton points. SUMMARY

[0005] The present application provides a double-flow network method based on compression excitation and skeleton points, which uses a post-fusion double-flow network with compression excitation and skeleton point attention for human behavior recognition, has high accuracy and good robustness, as described below:

[0006] A double-flow network method based on compression excitation and skeleton points, the method comprising:

[0007] Construct and train the spatial convolutional neural network; calculate the video optical flow, construct and train the time flow network;

[0008] Freeze all weights of the backbone network of the spatial convolutional neural network except the last three convolutional layers;

[0009] Between the backbone network of the spatial convolutional neural network and the global average pooling layer, construct compression excitation and skeleton point attention;

[0010] Construct a spatial flow network, which includes YOLOX, high-resolution network, and spatial convolutional neural network with compression excitation and skeleton point attention, and train the spatial flow network;

[0011] Use the trained time flow network and spatial flow network to recognize human behaviors, get two network results, and post-fuse the two network results to get the final human behavior recognition result.

[0012] The compression excitation and skeleton point attention between the backbone network of the spatial convolutional neural network and the global average pooling layer are:

[0013] Disconnect the connection between the backbone network of the spatial convolutional neural network and the global average pooling layer, so that the output of the backbone network is processed by the compression excitation and skeleton point attention before being input into the global average pooling layer, and then input into the fully connected layer.

[0014] The skeleton point attention is:

[0015] Project the skeleton points to a projection matrix T=(ar,c ) H×W In the method, the number of the skeletal points in each element position of the projection matrix T is counted as the value of the corresponding element;

[0016] The projection matrix T is batch standardized under a preset batch size, so that the mean and variance of the distribution of the projection matrix T at the batch level are adjusted to the learned mean and variance.

[0017] The projection matrix T of the skeletal point is (a r,c ) H×W The calculation is as follows:

[0018] The elements of the projection matrix T are calculated as follows: , wherein and represent the height and width direction coordinates of the skeletal points after projection, and the projection calculation of the skeletal points is as follows: H and W are the height and width of the feature map output by the backbone network of the spatial convolutional neural network, OH and OW are the maximum height and width that the coordinates of the skeletal points can reach before projection, and represent the coordinates of the skeletal points before projection, and b represents the number of the human body appearing in the video frame X 0 , k represents the number of the skeletal points corresponding to the human body, 0≤k≤16 and k∈N, N represents a natural number set, represents the floor function, and P F represents the set of projected skeletal points, r and c represent the rows and columns of the projection matrix T, and [] is the Iverson bracket.

[0019] The spatial stream network is constructed, the spatial stream network includes YOLOX, a high-resolution network, and a spatial convolutional neural network with compressed excitation and skeletal point attention, and the spatial stream network is trained:

[0020] The YOLOX and the high-resolution network used are both pre-trained by means of an image dataset with target frame and skeletal point label information, the YOLOX pre-trained is used to detect the position of the human body in the video frame X 0 , the target frame detected by the YOLOX is input into the high-resolution network pre-trained to detect the human skeletal points, and a total of 17 human skeletal points, i.e., nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles, are obtained in each target frame, and the set of skeletal points and are the original height and width coordinates of the skeletal points, and D represents the total number of human bodies in the video frame X 0 , i.e., the total number of the detected target frames, when D=0, it means that no human body is detected in this video frame, and N represents a natural number set;

[0021] During the training of the spatial flow network, the video frame X 0 Use random multi-scale cropping to convert to video frame X 1 , the skeleton point set P will be adjusted to the skeleton point set P*, for the video frame X 1 Use random horizontal flipping to convert to video frame X, the skeleton point set P* will be adjusted to the skeleton point set P** accordingly, and the video frame X will be converted to the video frame using standardization. Video Frame Input into the backbone network of the spatial convolutional neural network, and then the skeleton point set P** is input into the skeleton point attention to obtain a 1-channel matrix T with a shape of H×W new , the feature map output by the backbone network of the spatial convolutional neural network is input into the compressed excitation to obtain a new feature map, and the matrix T new Multiplying with the new feature map and then adding it together, we get a result whose shape is consistent with the shape of the feature map output by the backbone network of the spatial convolutional neural network. During the training process, the video frame is input into the spatial stream network, and only the weights of the spatial convolutional neural network with compressed excitation and skeleton point attention are updated, while the weights of the YOLOX and high-resolution networks are not updated.

[0022] Wherein, the video frame X 0 Converted to video frame X after random multi-scale cropping 1 , the skeleton point set P will be adjusted to the skeleton point set P* as follows:

[0023] The skeleton point set P is adjusted to the corresponding skeleton point set:

[0024] First calculate the intermediate state of the adjusted bone point coordinates That is hour, otherwise Then calculate the intermediate state of the adjusted bone point coordinates That is hour, otherwise Secondly, according to Calculate the adjusted bone point coordinates and based on Calculate the adjusted bone point coordinates Among them, ho and wo represent the height and width of the cropping position. During the training process, ho and wo take random values, and ch and cw are also random quantities, indicating that the video frame X 0 The height and width after cropping, NI is a small enough negative number to ensure that the coordinates of the skeleton points in the cropped area are less than 0 after adjustment, RH and RW are the video frame X in the random multi-scale cropping 0 The height and width to which the clipping should be scaled;

[0025] wherein the pair of video frames X 1 After conversion to video frame X with random horizontal flip, the skeleton point set P* will be adjusted to skeleton point set P**:

[0026] The relationship between the height coordinates of the skeleton point set P* and is, The relationship between the width coordinates of the skeleton point set P* and P** is adjusted to, After random horizontal flip, the video frame X 1 has a 50% probability of being horizontally flipped, if the video frame X 1 is horizontally flipped, otherwise wherein VW is the width of the video frame X 1 .

[0027] The use of the trained time flow network and the spatial flow network to identify human behaviors, the post-fusion of the results of the two networks obtains the final human behavior recognition result is:

[0028] The video frame is sequentially input into the backbone network and the compression excitation of the spatial convolutional neural network, then the skeleton point set corresponding to the video frame is input into the skeleton point attention, the results of the compression excitation and the skeleton point coordinates of the skeleton point attention are processed through the global average pooling layer and the full connection layer, and a vector output by the spatial flow network is obtained;

[0029] The optical flow frame is input into the backbone network, the global average pooling layer and the full connection layer of the time flow network, and a vector output by the time flow network is obtained; after the above two vectors are added and averaged, an averaged vector is obtained, and the dimension position value corresponding to the maximum value element of the averaged vector is the final human recognition prediction result.

[0030] The technical scheme provided by the application has the beneficial effects that:

[0031] 1、The application strengthens the feature extraction ability of different channel convolutions of the backbone network by assigning different channel weights to the feature maps of the backbone network, which helps to improve the performance of the human recognition method;

[0032] 2、The technical scheme of the application enables a human behavior dataset with lower cost and without human skeleton point labels to also play a role, that is, without training using human skeleton point labels, human skeleton point information can be introduced into the human behavior recognition method;

[0033] 3、The compression excitation and the skeleton point attention in the technical scheme of the application can not only be used in the dual-flow network, but also can be easily transplanted to other advanced methods, thereby reducing the development cost of new methods. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 a flowchart of a dual-stream network method based on compressed excitation and skeleton points;

[0035] Figure 2 a structural diagram of a dual-stream network based on compressed excitation and skeleton points;

[0036] Figure 3 a structural diagram of compressed excitation blocks and skeleton point attention;

[0037] Figure 4 a schematic diagram of training a spatial convolutional neural network and a temporal stream network;

[0038] Figure 5 a schematic diagram of training a spatial stream network. DETAILED DESCRIPTION

[0039] To make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application are described in further detail below.

[0040] Embodiment 1

[0041] As shown in the accompanying drawings, Figure 1 the embodiments of the present application provide a dual-stream network method based on compressed excitation and skeleton points, comprising the following specific steps:

[0042] S1: obtaining human behavior recognition video data to make training set and test set;

[0043] S2: constructing and training a spatial convolutional neural network;

[0044] S3: calculating video optical flow, constructing and training a temporal stream network;

[0045] S4: freezing all weights of the backbone network of the spatial convolutional neural network except the last three convolutional layers;

[0046] S5: constructing compressed excitation and skeleton point attention between the backbone network of the spatial convolutional neural network and the global average pooling layer;

[0047] S6: constructing a spatial stream network, the spatial stream network comprising YOLOX, a high-resolution network and a spatial convolutional neural network with compressed excitation and skeleton point attention, and training the spatial stream network;

[0048] S7: using the trained temporal stream network and spatial stream network to identify human behavior, obtaining two network results, and post-fusing the two network results to obtain the final human behavior recognition result.

[0049] The embodiment of the present application assigns different channel weights to the feature maps of the backbone network through the above steps S1-S7, thereby enhancing the feature extraction capability of different channel convolutions of the backbone network, and using the human skeleton point information, which is helpful to improve the performance of the human recognition method.

[0050] Embodiment 2

[0051] The scheme in embodiment 1 will be further introduced in combination with specific examples and calculation formulas. The scheme introduction is described in detail as follows:

[0052] The dual-stream network based on compression excitation and skeleton points mainly consists of a spatial stream network and a temporal stream network, as shown in Figure 2 . Among them, the spatial stream network includes a target detection model YOLOX, a human pose estimation model high-resolution network (HRNet), and a spatial convolutional neural network with compression excitation and skeleton point attention; the temporal stream network includes a temporal ResNet152 backbone network, a global average pooling layer GAP, and a fully connected layer. The compression excitation enhances the distinguishing ability of different channels of the spatial stream network, and the skeleton point attention is the core part of the spatial stream network to increase the human skeleton point information, and the detailed structures of the two are shown in Figure 3 . The steps S1-S7 in embodiment 1 set forth the method of constructing, training and using the dual-stream network based on compression excitation and skeleton points.

[0053] In step S1, the human behavior recognition video data is derived from the UCF101 human behavior recognition dataset produced by the Computer Vision Research Center of the University of Central Florida, which includes a total of 13320 videos. The UCF101 dataset is collected from the YouTube website and has great changes in camera motion, object appearance and posture, object size, viewpoint, cluttered background, lighting conditions, etc., and is a complex scene dataset. The embodiment of the present application adopts the split 1 method of the UCF101 dataset, that is, the dataset is divided into a training set including 9537 videos and a test set including 3783 videos. Before training, all video data of the UCF101 training set are extracted into video frames and saved as images, and the size of each video frame is 256x342. The advantage of converting video data into image data before training is to speed up the training model.

[0054] In step S2, the spatial convolutional neural network structure described in the embodiments of the present application uses a ResNet152 network, referred to as a spatial ResNet152 network. For the convenience of description, the embodiments of the present application refer to the part of the spatial ResNet152 network except the global average pooling layer and the fully connected layer as the spatial ResNet152 backbone network. The spatial ResNet152 backbone network is connected to the fully connected layer through the global average pooling layer. The embodiments of the present application first pre-train the spatial convolutional neural network using the ImageNet dataset, and then train in the UCF101 training set. During training, a video frame is randomly extracted from each video in the training set and input into the spatial convolutional neural network. The optimizer for training the spatial convolutional neural network uses a momentum-based stochastic gradient descent algorithm. During the training process, the momentum of the stochastic gradient descent algorithm is set to 0.9, the learning rate is 0.001, the training is 250 cycles (Epoch), the learning rate is adjusted to 0.0001 at the 100th cycle, and the learning rate is adjusted to 0.00001 at the 200th cycle. The loss function uses cross-entropy loss. The weight update of the stochastic gradient descent uses L2 norm, the weight decay value is set to 0.0005, and the batch size is set to 25. The training schematic diagram of the spatial convolutional neural network is shown in FIG. 1. Figure 4

[0055] During the training process, the video frame X 0 is input into the spatial convolutional neural network after data enhancement such as random multi-scale cropping, random horizontal flipping and standardization. Among them, the video frame X 0 is converted into the video frame X 1 after random multi-scale cropping. A random position in the set E={(0, 0), (0, 116), (32, 0), (32, 116), (16, 58), (0, 58), (32, 58), (16, 0), (16, 116), (8, 29), (8, 87), (24, 29), (24, 87)} is first selected as the cropping position of the video frame X 0 , and then a random size in the set S={(256, 256), (256, 224), (224, 256), (224, 224), (224, 192), (192, 224), (192, 192), (192, 168), (168, 192), (168, 168)} is selected as the size of the cropped video frame X 0 , and then the cropped video frame X 0 is scaled to 224x224. The video frame X 0 is converted into the video frame X ​The following formula (1) is used:

[0056]

[0057] in, is the normalized video frame The matrix of the i-th channel, H and W represent the height and width of the video frame X, both are 224, X i Represents the matrix of the i-th channel of the video frame X, i∈{0,1,2} corresponds to the R, G and B channels respectively, mean i Different values ​​are taken according to different channels, namely 0.485, 0.456 and 0.406. i Similarly, the values ​​are 0.229, 0.224 and 0.225 respectively, J∈R H×W is an all-one matrix.

[0058] The following involves each step of network training (excluding YOLOX and high-resolution network training). Unless otherwise specified, it means that data augmentation techniques such as random multi-scale cropping, random horizontal flipping, and normalization are used in the training process. The parameter settings of network training are the same as those in step S2 and are not repeated here.

[0059] Wherein, step S3 is specifically as follows:

[0060] S31: In step S2, the training video was saved as video frame images. The TV-L1 optical flow algorithm is used to generate optical flow in the width and height directions for every two frames of the same video. To speed up model training, the calculated width and height optical flows are saved as separate images. Before using the optical flow frame, the two optical flow images representing the width and height directions are stacked to form a two-channel optical flow frame.

[0061] S32: The structure of the time flow network in the embodiment of the present application is a ResNet152 network, referred to as a time ResNet152 network. For the convenience of description, the part of the time ResNet152 network except the global average pooling layer and the fully connected layer is referred to as a time ResNet152 backbone network in the embodiment of the present application. The time ResNet152 backbone network is connected with the global average pooling layer and the fully connected layer. The time flow network is initialized using the weights pre-trained on the ImageNet dataset. Random 10 two-channel optical flow frames of each video in the UCF101 training set are selected for training the time flow network. In order to match the input data, the channel number of each convolution kernel of the first layer of the time flow network is 20. The training settings of the time flow network are consistent with the training settings of the spatial convolutional neural network except the number of training cycles, the learning rate adjustment and the data standardization method. The time flow network is trained for 350 cycles, and the learning rate is reduced to 1 / 10 at the 200th cycle and the 300th cycle. In the process of training the time flow network, the calculation of the optical flow frame standardization adopts formula (1), the difference lies in that for the input 10 two-channel optical flow frames, the mean i of each channel is 0.5, and the std i of each channel is 0.226. The training schematic of the time flow network is shown in Figure 4 .

[0062] In step S4, the weights of the spatial ResNet152 backbone network except the last three convolution layers are frozen, that is, except the last fully connected layer of the spatial ResNet152 network and the last three convolution layers of the backbone network, the weights of other layers of the network are not updated during training. In the specific implementation, for the PyTorch deep learning framework, as long as the requires_grad variable of the weight is false, PyTorch will automatically not update this weight during back propagation, thereby achieving the purpose of freezing the weight.

[0063] In step S5, the specific implementation is as follows:

[0064] S51: Constructing compressed excitation. The compressed excitation can be divided into compression and excitation parts. The compression part performs global average pooling on the input. The excitation part has two convolution layers, the first convolution layer Conv1 and the second convolution layer Conv2. Then the output of Conv2 is multiplied with the input of the compression part in the channel direction, and the shape of each convolution kernel is 1x1. Among them, ReLU activation function is used after Conv1, the number of convolution kernel channels is 2048, and the number of convolution kernels is 512; HardSigmoid activation function is used after Conv2, and there are 2048 convolution kernels with 512 channels. The HardSigmoid activation function adopts the following formula (2):

[0065]

[0066] where y j is the j-th (j ∈ {1,..., j,..., L}) element of the output vector of Conv2, and L is the length of the output vector.

[0067] S52: Constructing the skeleton point attention.

[0068] The skeleton point attention involves two-step calculation. The first step projects all the skeleton points input to the skeleton point attention into a projection matrix T with the size consistent with the feature map size output by the spatial ResNet152 backbone network, i.e., a 7x7 size matrix, and then counts the number of skeleton points in each element position of the projection matrix T as the value of the corresponding element. The calculation of the skeleton point projection and the element calculation of T are shown in equations (3) to (4):

[0069]

[0070]

[0071]

[0072] where, and represent the height and width direction coordinates of the projected skeleton points, H and W are the height and width of the feature map output by the backbone network of the spatial convolutional neural network, OH and OW are the maximum height and width that the coordinates before the skeleton point projection can reach, and represent the coordinates of the skeleton points before projection, b represents the number of the actor in the video frame X 0 , k represents the number of the skeleton point of the corresponding actor, 0 ≤ k ≤ 16 and k ∈ N, N represents a set of natural numbers,

[0073] represents the floor function, P F represents the set of projected skeleton points, r and c represent the rows and columns of the projection matrix T, and [] is the Iverson bracket.

[0074] The second step calculation involved in the skeleton point attention is to perform batch normalization on the projection matrix, so that the mean and variance of the projection matrix distributed at the batch level are adjusted to the learned mean and variance. The batch normalization uses the following equation (6):

[0075]

[0076] where Tnew represents the batch-normalized projection matrix, T is the projection matrix without batch normalization, μ B and is the mean and variance of the projection matrix at the batch level, J0 is an all-one matrix, and γ and β are learnable parameters, and ε is a minimum value to prevent the denominator from being 0 during calculation.

[0077] S53: disconnect the spatial ResNet152 backbone network of the spatial convolutional neural network from the global average pooling layer, so that the output of the spatial ResNet152 backbone network is input into the global average pooling layer after being compressed and excited and processed by the skeleton point attention, and then input into the fully connected layer, as shown in Figure 5 . Thus, the compressed excitation and the skeleton point attention are constructed between the backbone network of the spatial convolutional neural network and the global average pooling layer.

[0078] In step S6, the specific steps are as follows:

[0079] S61: In step S2, the spatial convolutional neural network has been trained once in the UCF101 dataset, and the weights of the backbone network except the last three convolutional layers have been frozen in step S4. The standardized video frame is input into the spatial RestNet152 backbone network, and a feature map with a size of 7x7x2048 is obtained, and then input into the compressed excitation, and the feature map is globally averaged in the channel direction, and the feature map becomes 1x1x2048, and then input into the two convolutional layers of the compressed excitation and the corresponding activation function, and a 2048-dimensional channel weight vector vh=[hardsigmoid(y1),…,hardsigmoid(y j ),…,hardsigmoid(y 2048 )] is obtained, and finally the feature map output by the spatial RestNet152 backbone network is multiplied by the channel weight vector vh to obtain a new feature map with a shape of 7x7x2048.

[0080] S62: detecting human skeleton points from the video frame X 0 . The embodiment of the application first uses the YOLOX trained in the MMAction2 framework to detect the position of the human body in the video frame X 0 , and obtains the target frame of the human body in the video frame X 0 . Then, the high-resolution network trained in the MMAction2 framework is used to detect the human skeleton points from the obtained human target frame, and 17 human skeleton points, i.e., nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles, are detected in each target frame, and a skeleton point set And before the adjustment of the bone point coordinates, the bone points with a confidence greater than 0.3 are screened as the input of the bone point attention. The YOLOX and the high-resolution network used are trained by means of the COCO dataset with target frame and bone point label information, and do not need to be trained on UCF101.

[0081] S63: In the training process, random multi-scale cropping and random horizontal flipping are implemented on the video frame X 0 , which will change the position of the human body bone points on the video frame X 0 , so the coordinates of the detected human body bone points need to be adjusted before the input of the bone point attention. Random multi-scale cropping means that a random part of the video frame X 0 is cropped, and then the part is scaled to the set size. Correspondingly, the coordinates of the bone point set P should be adjusted to the bone point set P' by using the following formulas (7) to (10).

[0082]

[0083]

[0084]

[0085]

[0086] wherein, and represent the intermediate state of the adjusted bone point coordinates, ho and wo represent the height and width of the cropping position, ho and wo take random values in the training process, ch and cw are also random quantities, representing the height and width of the cropped video frame, NI is a small enough negative number to ensure that the coordinates of the bone points in the cropped-out area are less than 0 after adjustment, RH and RW are the height and width to which the video frame should be scaled after cropping in the random multi-scale cropping, and represent the adjusted bone point coordinates.

[0087] In the video frame X 0 , the coordinates of the human body bone point set P are adjusted to the bone point set P' by using the formulas (7) and (8) in the cropping process. (ho, wo) and (ch, cw) are randomly selected from the sets E and S respectively, and NI is-1000000 in the embodiment of the present application. Formulas (7) and (8) show that as long as the bone point is in the cropped-out area, the coordinates of the bone point are assigned to the NI value, and the bone point will be ignored in formula (5) and not processed. In the scaling process after the cropping of the video frame X 0 , the bone point set P' is scaled to the bone point set P'' by using the formulas (9) and (10). Using formula (9) and formula (10), the set of bone points P* is transformed into RH and RW are both taken as 224. Video frame X 0 After random multi-scale cropping, video frame X is transformed into 1 .

[0088] In the random horizontal flipping process, video frame X 1 has a 50% probability of being horizontally flipped, and the set of bone points P* is transformed into The correspondence between P* and P** is shown in formula (11) and formula (12):

[0089]

[0090]

[0091] wherein, VW is the width of video frame X 1 After being adjusted, the set of bone points P** is input into the bone point attention, and after formula (3) and formula (4), it becomes Then it is input into formula (5) to generate a 7x7 projection matrix T. After batch normalization, the projection matrix T becomes T new This matrix T new is multiplied by the new feature map of 7x7x2048 of the compressed excitation output and added to obtain a result.

[0092] S64: The result of multiplying and adding the compressed excitation and the output of the bone point attention is input into the global average pooling layer and the fully connected layer of the spatial convolutional neural network to obtain a 101-dimensional prediction vector. Then, the cross-entropy loss function is calculated for the gradient of each weight. After backpropagation, the update of the weights of the spatial convolutional neural network with compressed excitation and bone point attention is completed once.

[0093] wherein, the number of training cycles of the spatial convolutional neural network with compressed excitation and bone point attention is 500. At this time, the learning rate is initially 0.1, and the learning rate will be adjusted once in the 30th, 130th, 230th, 330th and 430th cycles, so that it is reduced to 1 / 10 of itself, and then the training is started. In the spatial stream network, only the spatial convolutional neural network with compressed excitation and bone point attention needs to be trained in the UCF101 training set, and the training schematic diagram is as shown in Figure 5 .

[0094] The step S7 is as follows. After the time stream network is trained in the step S3 and the spatial convolutional neural network with the compressed excitation and the skeleton point attention is trained in the step S6, the spatial stream network is constituted by the YOLOX and the high-resolution network. The video frame is sequentially input into the backbone network and the compressed excitation of the spatial convolutional neural network, and then the video frame is input into the YOLOX and the high-resolution network to obtain a skeleton point set input into the skeleton point attention. The multiplication result of the compressed excitation and the skeleton point coordinates skeleton point attention is added, and then the result is input into the global average pooling layer and the full connection layer to obtain a 101-dimensional vector. The optical flow frame is input into the time ResNet152 backbone network, the global average pooling layer and the full connection layer to obtain another 101-dimensional vector. The two vectors output by the spatial stream network and the time stream network are added and averaged to obtain an averaged vector, that is, a post-fusion vector of the network result. The dimension position value corresponding to the maximum element of the averaged vector is the final human body recognition prediction result. The human body behavior recognition process of the spatial stream network and the time stream network is as shown in the following figure. Figure 2

[0095] Embodiment 3

[0096] The feasibility of the embodiments 1 and 2 is verified by the following examples, which are described in detail as follows.

[0097] In order to evaluate the human body behavior recognition effect of the step S7, the embodiment of the present application tests 3783 videos in the UCF101 test set, and the method is as follows.

[0098] The processing method of the UCF101 test set and the training set is consistent, that is, the video frames of the test video are saved as a series of images before testing, and the calculated optical flow frames are also saved as images. Each video is uniformly sampled by 25 video frames and 25 groups of continuous 10 video frames. For the video frames and the optical flow frames, first, the standardization is performed, and then the video frames and the optical flow frames are cut into 224x224 size at five positions of (0, 0), (0, 118), (16, 60), (32, 0) and (32, 118), respectively. After that, the video frames and the optical flow frames are respectively flipped in the horizontal direction, and the same cutting operation is performed. According to the different cutting positions, 5 kinds of frames without horizontal flipping and 5 kinds of frames with horizontal flipping are obtained. All the video frames and the optical flow frames are input into the corresponding network, and the skeleton point set corresponding to the video frames is adjusted, and 250 results are obtained respectively. Finally, the average of the 500 results is taken to obtain the final human body recognition prediction result. The accuracy of the embodiment of the present application on the UCF101 test set is 91.75%, which is improved by 5.55% compared with the original dual-stream network without the compressed excitation and the skeleton point attention. [5]

[0099] ​​The compression excitation and skeleton point attention of the technical solution of the present application can be easily transplanted into a human behavior recognition method with convolutional layers, thereby introducing skeleton information and giving feature map channel weights to the transplanted method, and improving the performance of the human behavior recognition algorithm.

[0100] REFERENCES

[0101] [1]Szegedy C,Liu W,Jia Y,et al.Going deeper with convolutions[C].Proceedings of the IEEE conference on Computer Vision and Pattern Recognition,2015:1-9.

[0102] [2]Wu H,Ma X,Li Y.Convolutional networks with channel and STIPsattention model for action recognition in videos[J].IEEE Transactions onMultimedia,2019,22(9):2293-2306.

[0103] [3]Hu J,Shen L,Sun G.Squeeze-and-excitation networks[C].Proceedingsof the IEEE conference on Computer Vision and Pattern Recognition,2018:7132-7141.

[0104] [4]Howard A,Sandler M,Chu G,et al.Searching for mobilenetv3[C].Proceedings of the IEEE conference on Computer Vision and PatternRecognition,2019:1314-1324.

[0105] [5] Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos [J]. Advances in Neural Information Processing Systems, 2014, 27.

[0106] [6] Karpathy A, Toderici G, Shetty S, et al. Large-scale video classification with convolutional neural networks [C]. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2014: 1725-1732.

[0107] [7] Song Y F, Zhang Z, Shan C, et al. Constructing stronger and faster baselines for skeleton-based action recognition [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

[0108] [8] Shi L, Zhang Y, Cheng J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition [C]. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2019: 12026-12035.

[0109] [9] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C]. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016: 770-778.

[0110] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above-mentioned embodiment numbers of the application are only for description, not representing the advantages and disadvantages of the embodiments.

[0111] The above is only the preferred embodiment of the application, and is not used to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method of a dual-stream network based on compression excitation and skeleton points, characterized in that, The method comprises: building and training a spatial convolutional neural network; calculating a video optical flow, building and training a time flow network; freezing all weights of a backbone network of the spatial convolutional neural network except the last three convolutional layers; building a compressed excitation and a skeleton point attention between the backbone network of the spatial convolutional neural network and a global average pooling layer; building a spatial flow network comprising a YOLOX, a high-resolution network and the spatial convolutional neural network with the compressed excitation and the skeleton point attention, and training the spatial flow network; using the trained time flow network and the spatial flow network to identify human behaviors, obtaining two network results, and post-fusing the two network results to obtain a final human behavior recognition result; the building of the spatial flow network comprises a YOLOX, a high-resolution network and the spatial convolutional neural network with the compressed excitation and the skeleton point attention, and the training of the spatial flow network is specifically: The used YOLOX and high-resolution network are both pre-trained with image dataset with target frame and skeleton point label information, the pre-trained YOLOX is used to detect the position of human body in the video frame X 0 , the target frame detected by YOLOX is input into the pre-trained high-resolution network for human body skeleton point detection, 17 human body skeleton points of nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees and left and right ankles are obtained in each target frame, and a skeleton point set and is obtained, where H and W represent the original height and width coordinates of the skeleton points, D represents the total number of human bodies in the video frame X 0 , that is, the total number of detected target frames, when D=0, it means that no human body is detected in this video frame, and N represents a natural number set; During the spatial stream network training process, the video frame X 0 is converted into the video frame X using random multi-scale cropping 1 . The skeleton point set P is correspondingly adjusted to the skeleton point set P*, and the video frame X 1 is converted into the video frame X using random horizontal flipping. The skeleton point set P* is correspondingly adjusted to the skeleton point set P**, and the video frame X is converted into the video frame X using standardization . The video frame X is input into the backbone network of the spatial convolutional neural network, and then the skeleton point set P** is input into the skeleton point attention to obtain a 1-channel matrix T with the shape of HxW. new The feature map output by the backbone network of the spatial convolutional neural network is input into the compression excitation to obtain a new feature map. The matrix T new is multiplied by the new feature map and then added to obtain a result with the same shape as the feature map output by the backbone network of the spatial convolutional neural network.

2. The dual-stream network method based on compression excitation and skeleton points of claim 1, wherein, the building of the compressed excitation and the skeleton point attention between the backbone network of the spatial convolutional neural network and the global average pooling layer is: the connection between the backbone network of the spatial convolutional neural network and the global average pooling layer is disconnected, so that the output of the backbone network is processed by the compressed excitation and the skeleton point attention before being input into the global average pooling layer and then into the fully connected layer.

3. The method of claim 2, wherein, the skeleton point attention is: The bone points are projected to a projection matrix T=(a r,c ) H×W In the statistical projection matrix T, the number of bone points in each element position is taken as the value of the corresponding element. the projection matrix T is batch standardized under a preset batch size, so that the mean and variance of the distribution of the projection matrix T on the batch level are adjusted to the learned mean and variance, H and W are the height and width of the feature map output by the backbone network of the spatial convolutional neural network, and r and c represent the rows and columns of the projection matrix T, respectively.

4. The method of claim 3, wherein, The projection matrix T = (a r,c ) H×W The calculation of the projection matrix T = (a The elements of the projection matrix T are given by where and represent the height and width coordinates of the projected bone points, respectively, and are calculated as OH and OW are the maximum height and width that the coordinates of the bone points can reach before projection, and represent the coordinates of the bone points before projection, and b represents the number of the actor in the video frame X 0 k represents the number of the bone point corresponding to the actor, 0≤k≤16 and k∈N, N represents the set of natural numbers, represents the floor function, and P F represents the set of projected bone points, and [.] is the Iverson bracket.

5. The method of claim 4, wherein, The pair of video frames X 0 After random multi-scale cropping, the video frame X is converted to 1 The pair of video frames X 0 After random multi-scale cropping, the video frame X is converted to 1 The pair of video frames X 0 After random multi-scale cropping, the video frame X is converted to 1 The pair of video frames X 0 After random multi-scale cropping, the the skeleton point set P is adjusted to a skeleton point set: First calculate the intermediate state of the adjusted bone point coordinates That is hour, otherwise Then calculate the intermediate state of the adjusted bone point coordinates That is hour, otherwise Secondly, according to Calculate the adjusted bone point coordinates and based on Calculate the adjusted bone point coordinates Among them, ho and wo represent the height and width of the cropping position, ch and cw represent the video frame X 0 The height and width after cropping, NI is a small enough negative number to ensure that the coordinates of the skeleton points in the cropped area are less than 0 after adjustment, RH and RW are the video frame X in the random multi-scale cropping 0 The height and width to which the clipping should be scaled; The pair of video frames X 1 After random horizontal flipping, the video frame X is converted into a video frame X, and the set of bone points P* is adjusted to a set of bone points P**: The relationship of the set of skeletal points P* to the height coordinates of is adjusted to be The relationship of the set of skeletal points P* to the width coordinates of P** is adjusted to be, if the video frame X 1 is horizontally flipped, otherwise where VW is the width of the video frame X 1 .

6. The dual-stream network method based on compression excitation and skeleton points of claim 1, wherein, the using of the trained time flow network and the spatial flow network to identify human behaviors, and the post-fusion of the two network results to obtain a final human behavior recognition result is: the video frames are sequentially input into the backbone network and the compressed excitation of the spatial convolutional neural network, and then the corresponding skeleton point set of the video frames is input into the skeleton point attention, and the results of the compressed excitation and the skeleton point attention are processed through the global average pooling layer and the fully connected layer to obtain a vector output by the spatial flow network; the optical flow frames are input into the backbone network, the global average pooling layer and the fully connected layer of the time flow network to obtain a vector output by the time flow network; the two vectors are added and averaged to obtain an averaged vector, and the dimension position value corresponding to the maximum element of the averaged vector is the final human recognition prediction result.

Images