A human action recognition method based on consistent semi-supervised deep learning

Abstract

The present invention discloses a human behavior recognition method based on consistent semi-supervised deep learning, which relates to the field of computer vision. The method comprises the following steps: obtaining a labeled video set X and an unlabeled video set U to establish a training data sample set; performing video data enhancement processing on the training data sample set; building an improved 3D-ResNet18 network, constructing a loss function, training the improved 3D-ResNet18 network based on the loss function using the training data sample set, and using the optimized improved 3D-ResNet18 network to recognize human behaviors in videos. The present invention utilizes the human behavior recognition method to solve the problem that existing human behavior recognition methods lack effective data enhancement methods and thus develop relatively slowly; and the problem that existing human behavior recognition methods do not explore the temporal correlation of actions in videos, resulting in low robustness of trained models.

Description

Technical field:

[0001] The present invention relates to the field of computer vision, and in particular to a human behavior recognition method based on consistent semi-supervised deep learning. Background technology:

[0002] The purpose of video-based human behavior recognition in computer vision is to simulate human visual perception functions and accurately identify the categories of human behaviors in different environments.

[0003] Early action recognition relied heavily on manual feature extraction methods, but its limitations became increasingly prominent as the number of action categories increased. With the continuous development of convolutional neural networks (CNNs), people have designed different deep learning networks to automatically extract the spatial and temporal features of actions for classification tasks, including the following three methods: (1) Methods based on recurrent neural networks (RNNs). It is usually combined with CNNs, and by superimposing RNNs on the CNN structure, a synthetic feature representation of action space and time is obtained for classification. (2) Methods based on 2D convolution kernels. Two independent networks are established, and the spatial information (RGB image information) and temporal information (such as optical flow information) of the same action are input for training respectively. The two networks are then fused to reduce model parameters and improve recognition performance. However, this type of method is time-consuming and has low real-time performance due to its excessive reliance on optical flow as temporal information. (3) Methods based on 3D convolution kernels. A 3D convolution kernel is constructed to extract the spatiotemporal information of actions in videos, and multiple convolution kernels are used to obtain multi-layer features of actions based on weight sharing. However, too many model parameters bring challenges to calculation. In order to reduce the number of model parameters, people borrowed the design principles of 2D residual network, improved the architecture of C3D, and proposed 3D Resnet. The model reduced the number of parameters while also improving recognition accuracy.

[0004] Human movement is an important expression of our thoughts and emotions in daily life. Research findings in human action recognition have been successfully applied in fields such as intelligent surveillance, autonomous driving, and virtual reality. In recent years, with the rapid development of the short video industry, the recognition and annotation of actions in unlabeled videos has attracted widespread attention. To fully exploit the potential information of actions in unlabeled videos and reduce the resource consumption caused by manual labor, semi-supervised learning has been introduced for automatic recognition and annotation of actions in videos.

[0005] Image classification based on consistent semi-supervised deep learning largely relies on data augmentation methods (including random cropping, horizontal mirroring, vertical mirroring, contrast enhancement, and other methods) to improve model generalization capabilities. However, video classification, which extends to semi-supervised deep learning, has been relatively slow to develop due to the temporal and spatial diversity of actions and the lack of effective data augmentation methods. Horizontal flipping based on data augmentation causes action samples to become actions of other categories after enhancement; cropping methods cut out parts of the video to achieve image spatial data augmentation, but the temporal continuity of the action is lost.

[0006] Currently, most advanced methods study the temporal information of actions and design enhancement strategies including temporal consistency (sampling videos at equal intervals to obtain new sequences), scene invariance (transforming the video background), and action synonymy. These methods demonstrate strong performance in human action recognition compared to strategies such as horizontal flipping. However, when combined with the current advanced consistency-based semi-supervised deep learning recognition framework, the following two problems exist. First, in existing work that considers temporal aspects, the enhanced actions contain a certain amount of redundant information in the temporal sequence, lack descriptions of the details of the staged actions, and do not explore the temporal correlation of actions in the video, resulting in low robustness of the trained model.

[0007] In response to the above situation, designers need to design a reasonable human behavior recognition method to solve the problem that the current human behavior recognition method lacks effective data enhancement methods and develops relatively slowly, and does not explore the temporal correlation of actions in the video, resulting in the low robustness of the trained model. Summary of the invention:

[0008] In order to make up for the shortcomings of the existing technical problems, the present invention provides a human behavior recognition method based on consistent semi-supervised deep learning. The present invention realizes the description of the details of stage-by-stage actions by processing the spatial enhancement of video data and the temporal enhancement of video data, and utilizes the complete expression of multi-semantic actions to solve the problem that the existing human behavior recognition methods lack effective data enhancement methods and develop relatively slowly; in addition, the timing signal constructed by the present invention does not lose the complete action trend when extracting the fine-grained actions of the motion stages, which will help the model deepen its understanding of the detailed expression of the overall action, and solve the problem that the existing human behavior recognition methods do not explore the temporal correlation of actions in the video, resulting in the low robustness of the trained model.

[0009] The technical solutions of the present invention are as follows:

[0010] A human behavior recognition method based on consistent semi-supervised deep learning includes the following steps:

[0011] (1) Obtain a labeled video set X and an unlabeled video set U, and obtain small batches of frequency sets X' and U' from X and U respectively as training data sample sets;

[0012] (2) Performing video data enhancement processing on the training data sample set, including video data spatial enhancement and video data temporal enhancement;

[0013] (3) Build an improved 3D-Resnet18 network, which includes 17 convolutional layers and one fully connected layer;

[0014] (4) Construct the loss function L1=L s ; Among them, the loss function L s is the supervisory signal used to calculate the cross entropy loss between the true label and the predicted probability;

[0015] (5) Load the improved 3D-Resnet18 network with initialized network parameters, based on the loss function L s , use the training data sample set X' to train the network and calculate L s The loss value, that is, the loss value of the loss function L1, the first loss value is used as the initial loss value, and the current loss value is compared with the previous loss value. If the current loss value is less than the previous loss value, the network parameters are updated using the stochastic gradient descent algorithm. The above optimization process is repeated until the loss value no longer decreases and the network reaches the fit under the current iteration, and the optimized improved 3D-Resnet18 network is obtained;

[0016] (6) Construct the loss function L2=L s +λ d L d ; Among them, the loss function L d is a temporal signal used to calculate the Jensen-Shannon entropy divergence between action predictions after temporal enhancement of video data; d is the timing signal L d The weight of

[0017] (7) Load the improved 3D-Resnet18 network optimized in step (5);

[0018] Based on the loss function L s , use the training data sample set X' to train the improved 3D-Resnet18 network and calculate L s The loss value;

[0019] Based on the loss function L d , use the training data sample set (X', U') to train the improved 3D-Resnet18 network and calculate L d The loss value;

[0020] According to the loss function constructed in step (6), calculate the L2 loss value, take the first L2 loss value as the initial loss value, compare the current L2 loss value with the previous L2 loss value, and if the current L2 loss value is less than the previous L2 loss value, use the stochastic gradient descent algorithm to update the network parameters until the L2 loss value no longer decreases, and the model reaches the fit under the current iteration, thus obtaining the optimized improved 3D-Resnet18 network;

[0021] (8) Construct the loss function L = L s +λ u L u +λ d L d , where L u is a pseudo supervisory signal used to calculate the cross entropy loss between the spatial enhancement prediction category of the video data and the temporal enhancement prediction probability of the video data for unlabeled samples, λ u is the pseudo supervisory signal L u The weight of

[0022] (9) Load the improved 3D-Resnet18 network optimized in step (7);

[0023] Based on the loss function L s , use the training data sample set X' to improve the 3D-Resnet18 network for training and calculate L s The loss value;

[0024] Based on the loss function L u , use the training data sample set U' to train the improved 3D-Resnet18 network and calculate L u The loss value;

[0025] Based on the loss function L d , use the training data sample set (X', U') to train the improved 3D-Resnet18 network and calculate L d The loss value;

[0026] According to the loss function constructed in step (8), calculate the loss value of L3, take the first loss value of L3 as the initial loss value, compare the current loss value of L3 with the loss value of the previous L3, and if the current loss value of L3 is less than the loss value of the previous L3, use the stochastic gradient descent algorithm to update the network parameters until the loss value no longer decreases and the network reaches the fit under the current iteration, thus obtaining the optimized improved 3D-Resnet18 network;

[0027] (10) Load the improved 3D-Resnet18 network optimized in step (9) to perform human behavior recognition on the video that requires behavior recognition.

[0028] The spatial enhancement of the video data: The video is composed of a video sequence F. Starting from the m-th frame, N frames are extracted at the e frame rate to obtain a coarse-grained representation x = [f M , from the m-th frame, N frames are extracted at the e frame rate to obtain a coarse-grained representation x = [f m , f m+e , f m+2e , …, f m+(N-1)e of the actions in the video; The coarse-grained representation x of the video actions is spatially enhanced with probability P to obtain a spatially enhanced representation α(x) of the video data. The spatial enhancement includes horizontal flipping of the image and random cropping of the image;

[0029] The temporal enhancement processing of the video data can obtain the previous-action expression and the subsequent-action expression of the fine-grained action;​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Use the improved 3D-Resnet18 network for training to obtain the predicted probability of each video belonging to its corresponding label Use the cross entropy loss function to calculate the probability of the recognition model prediction and the true category The cross entropy loss between:

[0036]

[0037] In step (7), the timing signal L is calculated d The loss values are as follows:

[0038] For video x∈{X',U'}, perform temporal enhancement on video x to obtain the pre-action expression β of fine-grained action pre (x) and the post-action representation β of fine-grained actions post (x); β pre (x), β post (x) Use the improved 3D-Resnet18 network for training to obtain the predicted probability P(β pre (x)), P(β post (x)), calculate the Jensen-Shannon entropy divergence between action predictions after temporal enhancement of video data:

[0039] P(β avg (x))=(P(β post (x))+P(β pre (x))) / 2 (2)

[0040]

[0041] L d =L KL (P(β pre (x)),P(β avg (x)))+L KL (P(β post (x)),P(β avg (x)))(5).

[0042] In step (9), the pseudo supervisory signal L is calculated u The loss values are as follows:

[0043] Select a small batch of videos from the unlabeled video set U For unlabeled videos, Perform spatial enhancement processing on video data to obtain spatial enhancement expression of video data Video Perform temporal enhancement processing on video data to obtain the front-end action expression of fine-grained action and post-action expression of fine-grained actions

[0044] Will Use the improved 3D-Resnet18 network for training to obtain the predicted probability of each video belonging to its corresponding label

[0045] x j c The preceding action expression of fine-grained action and post-action expression of fine-grained actions The convolutional layer extracts the previous action feature H1 and the next action feature H2 respectively, and fuses the previous action feature H1 and the next action feature H2 to obtain the fused feature H, H = H1 + H2. The fused feature H is input into the fully connected layer for classification to obtain the predicted probability.

[0046] Use pseudo labeling technology to obtain the threshold T of action category c t (c), when the maximum predicted probability Exceeding the predefined threshold T t (c) When the corresponding category as the predicted category; otherwise,

[0047]

[0048] calculate The predicted probability after fusion with fine-grained action features The cross entropy loss between u ,

[0049]

[0050] The pseudo-labeling technique is used to set the threshold T t (c), comprising the following steps:

[0051] For the small batch video set selected in step (4.3) Video Sample Perform spatial enhancement processing on video data to obtain spatial enhancement expression of video data Video Perform temporal enhancement processing on video data to obtain the front-end action expression of fine-grained action and post-action expression of fine-grained actions Will Use the improved 3D-Resnet18 network for training and obtain the prediction probability respectively

[0052] Calculate the mean predicted probability of coarse and fine-grained actions

[0053]

[0054] Statistical probability of measurement The maximum value is greater than the threshold τ and the predicted category is the number of categories c σ t (c), where the threshold τ is set:

[0055]

[0056] Where I is an indicator function, which counts as 1 when the conditions in the brackets are met;

[0057] Learning effect t (c) Normalization Convert the model's learning effect evaluation of each action to between (0-1):

[0058]

[0059] Use the nonlinear convex function M(x)=x / (2-x) to fit the convergence trend of the model and get the prediction The corresponding thresholds of each action are obtained;

[0060] In order to reduce the noise data input of the model, the upper threshold τ is set min and the lower threshold τ max ;

[0061] Threshold for comparative evaluation and the minimum threshold τ min The maximum threshold between the two is selected as the threshold T of the action at the current moment. t (c)

[0062]

[0063] like Re-comparison evaluation threshold and the maximum threshold T that has appeared in history max (c) Compare and select the maximum threshold between the two as the threshold T t (c)

[0064]

[0065] During the training of the improved 3D-Resnet18 network, the training round is EPOCH, and one round contains STEP training; the initial learning rate is η0;

[0066] If the current total loss is less than the total loss obtained in the previous training, the stochastic gradient descent algorithm is used to update the network parameters; otherwise, the network parameters are not updated and the optimized improved 3D-Resnet18 network is obtained; within the EPOCH*STEP cycle, the cosine decay function is used to achieve dynamic changes in the learning rate within the range of [0-η0].

[0067] In step (10), the improved 3D-Resnet18 network optimized in step (9) is loaded to perform human behavior recognition on the video that needs behavior recognition. The specific steps are as follows:

[0068] Take a video V to be predicted with a length of S, and extract the video clip with a length of s after uniform frame extraction. Randomly select x∈(0,Ss) as the starting frame, input it into the improved 3D-Resnet18 network optimized in step (9), traverse the video, and select the action category c with the largest prediction confidence: c=argmax(P(v));

[0069] Repeat the sampling of the same action five times and take the mean P of the five predicted probabilities mean (V) is taken as the final prediction result of the video V; the action category corresponding to the maximum value of the prediction result is taken as the final prediction result class: class = argmax (P mean (V)).

[0070] Compared with the prior art, the present invention has the following advantages:

[0071] 1. The present invention performs data enhancement processing on the original video to obtain coarse-grained expression and fine-grained expression of the action; it realizes the description of the details of the staged action and utilizes the complete expression of the multi-semantic action.

[0072] 2. In the process of constructing the timing signal, the present invention divides the actions in the video into preceding actions and subsequent actions in terms of timing, calculates the probability prediction results of the complete action and actions with different timings respectively, and calculates the Johnson-Shanon divergence between actions with different timings to constrain the predicted probability distribution between them. When extracting fine-grained actions of the motion stage, the complete action trend is not lost, which will help the model deepen its understanding of the detailed expression of the overall action.

[0073] 3. In the training process of the improved 3D-Resnet18 network, the present invention first uses the supervisory signal L s The network is trained to obtain sufficient knowledge under samples with real labels; then L2=L s +λ d L dTo train the network, the timing signal L d The introduction of L = L can realize the knowledge extraction of the consistent expression of the action in the video to cope with the labeling of unlabeled samples; finally, L = L s +λ u L u +λ d L d Train the network and introduce pseudo supervision signal L u , which helps to mine potential information in large amounts of unlabeled data and improve recognition performance.

[0074] 4. The present invention uses pseudo-label technology to set the threshold T t (c) During the network training process, unlabeled samples are selected and combined with action consistency learning. A loose and strict parallel curriculum learning strategy is adopted to set the threshold. The number of actions predicted by the statistical model for the unlabeled data is used to evaluate the learning effect of the action, and the corresponding threshold is set to help the action learn better. During the model training process, statistics are taken for each category whose unlabeled sample prediction results exceed the dynamic threshold to evaluate the learning effect of different actions. Loose conditions are added in the early stage of training to prevent the input of too much noise data, and strict conditions are added in the later stage of training to prevent poor estimation results due to data imbalance. In addition, in order to avoid the influence of the stage-by-stage difference representation of the action in the video on recognition and evaluation, the expression of the action in the video is comprehensively evaluated, and the prediction results of the coarse-grained expression and the fine-grained expression are combined to evaluate the effect of the dynamic threshold. Description of the drawings:

[0075] Figure 1 A bar chart showing the recognition accuracy and corresponding thresholds of the semi-supervised model for different actions in UCF101 with a labeling rate of 50%.

[0076] Figure 2 A bar chart showing the recognition accuracy and corresponding thresholds of the semi-supervised model for different actions in HMDB51 with a labeling rate of 50%.

[0077] Figure 3 A bar chart showing the recognition accuracy and corresponding thresholds of the semi-supervised model for different actions in Kinetic100 with a labeling rate of 10%.

[0078] Figure 4 A line graph showing the recognition rates of supervised learning and semi-supervised learning for UCF101 at different labeling rates.

[0079] Figure 5 A line graph showing the recognition rates of supervised learning and semi-supervised learning for HMDB51 at different labeling rates.

[0080] Figure 6A line graph showing the recognition rates of Kinetic100 under supervised learning and semi-supervised learning at different labeling rates. Specific implementation method:

[0081] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0082] A human behavior recognition method based on consistent semi-supervised deep learning includes the following steps:

[0083] (1) Obtain a labeled video set X and an unlabeled video set U, and obtain small batches of frequency sets X' and U' from X and U respectively as training data sample sets;

[0084] (2) Performing video data enhancement processing on the training data sample set, including video data spatial enhancement and video data temporal enhancement;

[0085] (3) Build an improved 3D-Resnet18 network as a human behavior recognition model. The improved 3D-Resnet18 network includes 17 convolutional layers and one fully connected layer;

[0086] (4) Construct the loss function L1=L s ; Among them, the loss function L s is the supervisory signal used to calculate the cross entropy loss between the true label and the predicted probability;

[0087] (5) Load the improved 3D-Resnet18 network with initialized network parameters, based on the loss function L s , use the training data sample set X' to train the network and calculate L s The loss value, that is, the loss value of the loss function L1, the first loss value is used as the initial loss value, and the current loss value is compared with the previous loss value. If the current loss value is less than the previous loss value, the network parameters are updated using the stochastic gradient descent algorithm. The above optimization process is repeated until the loss value no longer decreases and the network reaches the fit under the current iteration, and the optimized improved 3D-Resnet18 network is obtained;

[0088] (6) Construct the loss function L2=L s +λ d L d ; Among them, the loss function L d is a temporal signal used to calculate the Jensen-Shannon entropy divergence between action predictions after temporal enhancement of video data; d is the timing signal L d The weight of

[0089] (7) Load the improved 3D-Resnet18 network optimized in step (5);

[0090] Based on the loss function L s , use the training data sample set X' to train the improved 3D-Resnet18 network and calculate L s The loss value;

[0091] Based on the loss function L d , use the training data sample set (X', U') to train the improved 3D-Resnet18 network and calculate L d The loss value;

[0092] According to the loss function constructed in step (6), calculate the L2 loss value, take the first L2 loss value as the initial loss value, compare the current L2 loss value with the previous L2 loss value, and if the current L2 loss value is less than the previous L2 loss value, use the stochastic gradient descent algorithm to update the network parameters until the L2 loss value no longer decreases, and the model reaches the fit under the current iteration, thus obtaining the optimized improved 3D-Resnet18 network;

[0093] (8) Construct the loss function L = L s +λ u L u +λ d L d , where L u is a pseudo supervisory signal used to calculate the cross entropy loss between the spatial enhancement prediction category of the video data and the temporal enhancement prediction probability of the video data for unlabeled samples, λ u is the pseudo supervisory signal L u The weight of

[0094] (9) Load the improved 3D-Resnet18 network optimized in step (7);

[0095] Based on the loss function L s , use the training data sample set X' to improve the 3D-Resnet18 network for training and calculate L s The loss value;

[0096] Based on the loss function L u , use the training data sample set U' to train the improved 3D-Resnet18 network and calculate L u The loss value;

[0097] Based on the loss function L d , use the training data sample set (X', U') to train the improved 3D-Resnet18 network and calculate L dLoss value;

[0098] According to the loss function constructed in step (8), calculate the loss value of L3. Take the first loss value of L3 as the initial loss value. Compare the current loss value of L3 with the previous loss value of L3. If the current loss value of L3 is less than the previous loss value of L3, use the stochastic gradient descent algorithm to update the network parameters until the loss value of L3 no longer decreases. At this time, the network reaches the fitting under the current iteration, and an optimized improved 3D-Resnet18 network is obtained.

[0099] (10) Load the optimized improved 3D-Resnet18 network in step (9) to perform human behavior recognition on the video to be recognized.

[0100] The spatial enhancement of the video data: The video is composed of a video sequence F. Starting from the m-th frame, extract N frames at the e frame rate to obtain a coarse-grained representation x = [f M of the action in the video. Starting from the m-th frame, extract N frames at the e frame rate to obtain a coarse-grained representation x = [f m , f m+e , f m+2e , …, f m+(N-1)e of the action in the video. Perform spatial enhancement on the coarse-grained representation x of the video action with a probability P (P = 0.5) to obtain the spatially enhanced representation α(x) of the video data. The spatial enhancement includes horizontal flipping of the image and random cropping of the image.

[0101] The temporal enhancement processing of the video data can obtain the previous-action expression of the fine-grained action and the post-action expression of the fine-grained action.

[0102] The previous-action expression of the fine-grained action: From the video sequence F = [f1, f2,..., f M , we extract n frames at the v1 frame rate, where n < N, and then extract N - n frames at the v2 frame rate, with v1 > v2, to obtain the previous-action expression β pre (x) of the fine-grained action.

[0103] The post-action expression of the fine-grained action: From the video sequence F = [f1, f2,..., f M , we extract n frames at the v2 frame rate, and then extract N - n frames at the v1 frame rate, with v1 > v2, to obtain the post-action expression β post (x) of the fine-grained action.

[0104] The improved 3D-Resnet18 network includes 17 convolutional layers, and the last layer is a fully connected layer. Among the convolutional layers from the 2nd to the 16th layer, the Leaky-ReLU function is used instead of the ReLU function, and Dropout is added after the fully connected layer to alleviate the overfitting problem of the model.

[0105] In step (5), the supervisory signal L is calculated s The loss values are as follows:

[0106] Select a small batch of videos from the labeled video set X For videos with tags, For video The corresponding label for the video Perform spatial enhancement processing on video data to obtain spatial enhancement expression of video data

[0107] Will Use the improved 3D-Resnet18 network for training to obtain the predicted probability of each video belonging to its corresponding label The cross entropy loss function is used to calculate the probability p(α(x i c )) and the true category The cross entropy loss between:

[0108]

[0109] In step (7), the timing signal L is calculated d The loss values are as follows:

[0110] For video x∈{X',U'}, perform temporal enhancement on video x to obtain the pre-action expression β of fine-grained action pre (x) and the post-action representation β of fine-grained actions post (x); β pre (x), β post (x) Use the improved 3D-Resnet18 network for training to obtain the predicted probability P(β pre (x)), P(β post (x)), calculate the Jensen-Shannon entropy divergence between action predictions after temporal enhancement of video data:

[0111] P(β avg (x))=(P(β post (x))+P(β pre (x))) / 2 (2)

[0112]

[0113] L d =L KL (P(β pre (x)),P(β avg (x)))+L KL(P(β post (x)),P(β avg (x)))(5).

[0114] In step (9), the pseudo supervisory signal L is calculated u The loss values are as follows:

[0115] Select a small batch of videos from the unlabeled video set U For unlabeled videos, Perform spatial enhancement processing on video data to obtain spatial enhancement expression of video data Video Perform temporal enhancement processing on video data to obtain the front-end action expression of fine-grained action and post-action expression of fine-grained actions

[0116] Will Use the improved 3D-Resnet18 network for training to obtain the predicted probability of each video belonging to its corresponding label

[0117] Video The preceding action expression of fine-grained action and post-action expression of fine-grained actions The convolutional layer extracts the previous action feature H1 and the next action feature H2 respectively, and fuses the previous action feature H1 and the next action feature H2 to obtain the fused feature H, H = H1 + H2. The fused feature H is input into the fully connected layer for classification to obtain the predicted probability.

[0118] Use pseudo labeling technology to obtain the threshold T of action category c t (c), when the maximum predicted probability Exceeding the predefined threshold T t (c) When the corresponding category as the predicted category; otherwise,

[0119]

[0120] calculate The predicted probability after fusion with fine-grained action features The cross entropy loss between u ,

[0121]

[0122] The pseudo-labeling technique is used to set the threshold T t (c), comprising the following steps:

[0123] For the small batch video set selected in step (4.3) Video Sample Perform spatial enhancement processing on video data to obtain spatial enhancement expression of video data Video Perform temporal enhancement processing on video data to obtain the front-end action expression of fine-grained action and post-action expression of fine-grained actions Will Use the improved 3D-Resnet18 network for training and obtain the prediction probability respectively

[0124] Calculate the mean predicted probability of coarse and fine-grained actions

[0125]

[0126] Statistical probability of measurement The maximum value is greater than the threshold τ and the predicted category is the number of categories c σ t (c), where the threshold τ is set:

[0127]

[0128] Where I is an indicator function, which counts as 1 when the conditions in the brackets are met;

[0129] Learning effect t (c) Normalization Convert the model's learning effect evaluation of each action to between (0-1):

[0130]

[0131] Use the nonlinear convex function M(x)=x / (2-x) to fit the convergence trend of the model and get the prediction The corresponding thresholds of each action are obtained;

[0132] In order to reduce the noise data input of the model, the upper threshold τ is set min and the lower threshold τ max ;

[0133] Threshold for comparative evaluation and the minimum threshold τ min The maximum threshold between the two is selected as the threshold T of the action at the current moment. t (c)

[0134]

[0135] like Re-comparison evaluation threshold and the maximum threshold T that has appeared in history max (c) Compare and select the maximum threshold between the two as the threshold T t (c)

[0136]

[0137] During the training of the improved 3D-Resnet18 network, the training round is EPOCH, and one round contains STEP training; the initial learning rate is η0;

[0138] If the current total loss is less than the total loss obtained in the previous training, the stochastic gradient descent algorithm is used to update the network parameters; otherwise, the network parameters are not updated and the optimized improved 3D-Resnet18 network is obtained; within the EPOCH*STEP cycle, the cosine decay function is used to achieve dynamic changes in the learning rate within the range of [0-η0].

[0139] In step (10), the improved 3D-Resnet18 network optimized in step (9) is loaded to perform human behavior recognition on the video that needs behavior recognition. The specific steps are as follows:

[0140] Take a video V to be predicted with a length of S, and extract the video clip with a length of s after uniform frame extraction. Randomly select x∈(0,Ss) as the starting frame, input it into the improved 3D-Resnet18 network optimized in step (9), traverse the video, and select the action category c with the largest prediction confidence: c=argmax(P(v));

[0141] Repeat the sampling of the same action five times and take the mean P of the five predicted probabilities mean (V) is taken as the final prediction result of the video V; the action category corresponding to the maximum value of the prediction result is taken as the final prediction result class: class = argmax (P mean (V)).

[0142] Experimentation and Evaluation

[0143] The datasets used in the method of the present invention are UCF101, HMDB51, and Kinetics. The number of samples in the three datasets, training set, and test set are shown in Table 1.

[0144] UCF101 has 101 action categories, each with approximately 130 videos, including 100 training videos and 30 test videos. The action types include interactive actions (human-object, human-human), body movements, instrument playing, and sports, showing great diversity. The dataset has three splits, and split 1 was used for all experiments.

[0145] HMDB51 has 51 action categories, each with approximately 100 videos, including 70 training videos and 30 test videos. These action types include facial movements, body movements, and interactive movements, posing a high level of complexity and a significant challenge. The dataset has three split methods, and split 1 was used in all experiments.

[0146] Kinetics has 400 action categories, but some have significant differences in distribution. To fairly compare different algorithms, we selected 100 categories with relatively uniform action distribution for experiments, called kinetic-100.

[0147]

[0148] Table 1

[0149] The evaluation criteria are similar to those of most video classification methods. The test set videos are sampled multiple times and the average of the predictions is calculated as the final result. Due to memory limitations, the test set data is uniformly sampled five times to obtain 5 clips. The model's classification performance is evaluated using Top-1 Accuracy and Top-5 Accuracy.

[0150] The number of samples in the training set and test set of different datasets

[0151] (1) Experimental environment configuration and parameter setting

[0152] Table 2 shows the main environment and configuration of the experiment on the PC, including the specific versions of software and hardware, and two RTX 2080Ti graphics cards on the server.

[0153]

[0154] Table 2

[0155] Based on the data in Table 1, an improved 3D ResNet-18 was built. Leaky ReLU (p=0.2) was used instead of the ReLU function in convolutional layers 2-16. A Dropout layer with a dropout rate of 0.5 was added after the fully connected layers to prevent overfitting and variance shift. Furthermore, given the powerful effect of knowledge distillation, a ResNet-18 model pre-trained on ImageNet was loaded. The spatial information of actions obtained through knowledge distillation helped the model accurately classify certain actions that strongly rely on spatial information. During training, 16 video clips were obtained from the video in a recursive manner and randomly cropped to 112*112 pixels. The computational power for a single video clip was 8.33 GFLOPs, and the model had 33.23M parameters. Each batch of labeled and unlabeled samples consisted of 8 video sequences. The final input size of the network was 8*16*3*112*112.

[0156] In this paper, the initial learning rate is 0.02, and the cosine decay strategy is adopted. The SGD optimizer uses a momentum of 0.9 and a -4 The weight decay training data. The initialization threshold τ is set to 0.95, and the lower threshold τ min is 0.5, the upper threshold τ max It is 0.95, and it is stipulated that the three data sets are trained on labeled data in the first 100 epochs (video data spatial enhancement), and then the temporal supervision signal training is introduced (video data temporal enhancement). After the model reaches fitting, unlabeled data is added for training.

[0157] For the videos in the training set, two partitioning methods are used to achieve data balance and imbalance. To achieve data balance, the N videos in category C are divided into N*P labeled data and N*(1-P) unlabeled data (P is the ratio).

[0158] To achieve data imbalance to validate the curriculum learning strategy, the training set is randomly split rather than sampling all classes in equal proportions. Unless otherwise stated, the standard balanced dataset split method is used.

[0159] (2) The impact of different supervisory signals on model recognition performance

[0160] The effects of different supervisory signals on the model recognition performance are tested, as shown in Table 3. s Under supervision, a single performance of a labeled action is analyzed; the supervision signal L s Add timing signal L under supervision dAfter that, we further explored the temporal relevance of actions and formed a semantic understanding of the detailed expression of actions. On the UCF101 dataset with a 5% tagging rate, the performance was improved by 1.3%. s +L u L introduced under supervision d , the model still shows a "preference" for fine-grained expression of actions, and on the HMDB51 dataset with a 40% labeling rate, the performance is improved by 2.6%. This successfully verifies the experimental motivation, that is, paying attention to the multi-semantic synonymous expression of actions in videos is very important for understanding the true category of actions.

[0161]

[0162] Table 3

[0163] (3) The impact of different enhancement methods on recognition rate

[0164] In a consistent semi-supervised learning framework, we compared the prediction results of the model using a single strong augmentation action and feature fusion. Table 4 shows the recognition performance of each dataset under different augmentation methods.

[0165] a) Front Action Prediction (FAP): Use the model to predict the previous action and use the prediction results to calculate the loss of unlabeled data.

[0166] b) BackAction Prediction (BAP): Use the model to predict the subsequent actions and calculate the loss of unlabeled data.

[0167] c) Decision Fusion Prediction (DFP): Use the model to predict the previous and next actions, and fuse the prediction results at the decision level to calculate the unlabeled loss.

[0168] d) Feature Fusion Prediction (FFP): The model is used to extract features of the previous and subsequent actions, which are then fused and input into the next layer for prediction.

[0169] Table 4 shows that after temporal enhancement of video data and feature fusion, the soft labels are closer to the true predictions of unlabeled samples, providing the model with a more accurate pseudo-supervision signal. This has advantages for fine-grained representation of actions, and the recognition rates on all three datasets are consistently higher than those obtained using temporal enhancement of video data alone. This method can be extended to any effective data augmentation-based method to achieve even better recognition results. Experiments revealed that feature fusion is not effective in the early stages of the model. Only after the model has established sufficient ability to assess action categories does feature fusion significantly improve. To avoid this problem, the model is first trained using a supervisory signal, and then trained using both the supervisory signal and the temporal signal.

[0170]

[0171] Table 4

[0172] (4) Pseudo-labeling of video courses

[0173] Figure 1-3 The semi-supervised model's recognition accuracy for some actions and the corresponding dynamic thresholds are visualized. Figure 1 Indicates the recognition accuracy and corresponding threshold of the semi-supervised model for different actions based on UCF101 with a 50% labeling rate. Figure 2 Indicates the recognition accuracy and corresponding threshold of the semi-supervised model for different actions based on HMDB51 with a 50% labeling rate. Figure 3 The figure shows the recognition accuracy and corresponding thresholds of the semi-supervised model for different actions based on Kinetic100 with a 10% labeling rate. It can be found that the model always wants to set a relatively low threshold for actions with low recognition accuracy to help the model learn the unlabeled samples of the corresponding actions; on the other hand, for action categories with high recognition accuracy, a higher dynamic threshold is set to reduce the input of noisy data, and at the same time, it is hoped that the model will reduce the deterioration of the learning effect of other categories due to classification errors. On UCF101 and HMDB51, it can be used on the Kinetic100 dataset.

[0174] (V) Comparison with supervised learning methods

[0175] The algorithms are compared under the same backbone network and experimental settings. The semi-supervised learning algorithms evaluated by setting different proportions of labeled data are compared with the supervised baseline. The recognition rates of supervised learning and semi-supervised learning on the three datasets with different labeling rates are shown in the figure below. Figure 4-6 As shown, Figure 4 represents the recognition rate of supervised learning and semi-supervised learning of UCF101 at different labeling rates, Figure 5 represents the recognition rate of supervised learning and semi-supervised learning of HMDB51 at different labeling rates, Figure 6 Indicates the recognition rate of supervised learning and semi-supervised learning of Kinetic100 at different labeling rates. Figure 4-6 It can be seen that after incorporating unlabeled data, the model can tap into the potential information in the large amount of unlabeled data, significantly improving recognition performance. In particular, on the HMDB51 dataset with a 40% labeling rate, recognition performance increased significantly (+10.5%).

[0176] (6) Comparison with other methods

[0177] The present invention is compared with the current advanced semi-supervised learning methods, including the image classification methods MeanTeacher[1], PseudoLabel[2], SD[3], S4L[4], UPS[5], and the video-based methods VideoSSL[6], ActorCutMix[7], MvPL[8], LTG[9]. The training strategies and views used by each method are detailed, as well as the recognition accuracy (%) under different proportions of labeled data (% Label) (- indicates that the method was not tested under this condition), as shown in Table 5. It can be seen from Table 5 that the present invention achieves the best performance on the RGB view using the network model after ImageNet distillation, and exceeds some methods using multiple views (2.5% higher on UCF101 with a labeling rate of 5% and 1.6% higher on a labeling rate of 50%). It is worth noting that the proposed method outperforms the MvPL[8] method using three views by 2.5% on the UCF101 dataset with a 5% labeling rate. Therefore, it is believed that MvPL does not adopt an advanced consistent semi-supervised framework and does not consider the temporal interdependence and correlation of actions. LTG[9] achieves optimal recognition performance by building three consistent semi-supervised frameworks based on the complementary information of three views.

[0178]

[0179] Table 5

[0180] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0181] References:

[0182] [1]A.Tarvainen and H.Valpola,"Mean teachers are better role models:Weight-averaged consistency targets improve semi-supervised deep learningresults,"in Advances in neural information processing systems,2017,vol.30,pp.1195-1204.

[0183] [2]D.H.Lee,"Pseudo-Label:The Simple and Efficient Semi-SupervisedLearning Method for Deep Neural Networks,"inICML,2013,vol.3,p.896.

[0184] [3]R.Girdhar,D.Tran,L.Torresani,and D.Ramanan,"Distinit:Learningvideo representations without a single labeled video,"in Proceedings of theIEEE / CVFInternational Conference on Computer Vision,2019,pp.852-861.

[0185] [4]X.Zhai,A.Oliver,A.Kolesnikov,and L.Beyer,"S4l:Self-supervisedsemi-supervised learning,"in Proceedings of the IEEE / CVF InternationalConference on Computer Vision,2019,pp.1476-1485.

[0186] [5]M.N.Rizve,K.Duarte,Y.S.Rawat,and M.Shah,"In defense of pseudo-labeling:An uncertainty-aware pseudo-label selection framework for semi-supervised learning,"in 9th International Conference on LearningRepresentations,2021.

[0187] [6]L.Jing,T.Parag,Z.Wu,Y.Tian,and H.Wang,"Videossl:Semi-supervisedlearning for video classification,"in Proceedings of the IEEE / CVF WinterConference onApplications ofComputer Vision,2021,pp.1110-1119.

[0188] [7]Y.Zou,J.Choi,Q.Wang,and J.-B.Huang,"Learning representationalinvariances for data-efficient action recognition,"arXiv preprint arXiv:2103.16565,2021.

[0189] [8]B.Xiong,H.Fan,K.Grauman,and C.Feichtenhofer,"Multiview pseudo-labeling for semi-supervised learning from video,"in Proceedings of the IEEE / CVF International Conference on Computer Vision,2021,pp.7209-7219.

[0190] [9]J.Xiao et al.,"Learning from Temporal Gradient for Semi-supervisedAction Recognition,"arXivpreprint arXiv:2111.13241,2021.

Claims

1. A human behavior recognition method based on consistent semi-supervised deep learning, characterized in that: The following steps are involved: (1) Obtain a labeled video set X and an unlabeled video set U, and obtain small batches of frequency sets X' and U' from X and U respectively as training data sample sets; (2) Performing video data enhancement processing on the training data sample set, including video data spatial enhancement and video data temporal enhancement; (3) Build an improved 3D-Resnet18 network, which includes 17 convolutional layers and one fully connected layer; (4) Construct the loss function L1=L s ; Among them, the loss function L s is the supervisory signal used to calculate the cross entropy loss between the true label and the predicted probability; (5) Load the improved 3D-Resnet18 network with initialized network parameters, based on the loss function L s , use the training data sample set X' to train the network and calculate L s The loss value, that is, the loss value of the loss function L1, is used as the initial loss value. The current loss value is compared with the previous loss value. If the current loss value is less than the previous loss value, the network parameters are updated using the stochastic gradient descent algorithm. The optimization process is repeated until the loss value no longer decreases and the network reaches the fit under the current iteration, thus obtaining the optimized improved 3D-Resnet18 network. (6) Construct the loss function L2=L s +λ d L d ; Among them, the loss function L d is a temporal signal used to calculate the Jensen-Shannon entropy divergence between action predictions after temporal enhancement of video data; d is the timing signal L d The weight of (7) Load the improved 3D-Resnet18 network optimized in step (5); Based on the loss function L s , use the training data sample set X' to train the improved 3D-Resnet18 network and calculate L s The loss value; Based on the loss function L d , use the training data sample set (X', U') to train the improved 3D-Resnet18 network and calculate L d The loss value; According to the loss function constructed in step (6), calculate the L2 loss value, take the first L2 loss value as the initial loss value, compare the current L2 loss value with the previous L2 loss value, and if the current L2 loss value is less than the previous L2 loss value, use the stochastic gradient descent algorithm to update the network parameters until the L2 loss value no longer decreases, and the model reaches the fit under the current iteration, thus obtaining the optimized improved 3D-Resnet18 network; (8) Construct the loss function L = L s +λ u L u +λ d L d , where L u is a pseudo supervisory signal used to calculate the cross entropy loss between the spatial enhancement prediction category of the video data and the temporal enhancement prediction probability of the video data for unlabeled samples, λ u is the pseudo supervisory signal L u The weight of (9) Load the improved 3D-Resnet18 network optimized in step (7); Based on the loss function L s , use the training data sample set X' to improve the 3D-Resnet18 network for training and calculate L s The loss value; Based on the loss function L u , use the training data sample set U' to train the improved 3D-Resnet18 network and calculate L u The loss value; Based on the loss function L d , use the training data sample set (X', U') to train the improved 3D-Resnet18 network and calculate L d The loss value; According to the loss function constructed in step (8), the loss value of L3 is calculated, the first loss value of L3 is used as the initial loss value, and the current loss value of L3 is compared with the loss value of the previous L3. If the current loss value of L3 is less than the loss value of the previous L3, the network parameters are updated using the stochastic gradient descent algorithm until the loss value of L3 no longer decreases. The network reaches the fit under the current iteration, and the optimized improved 3D-Resnet18 network is obtained. (10) Load the improved 3D-Resnet18 network optimized in step (9) to perform human behavior recognition on the video that requires behavior recognition.

2. The human behavior recognition method based on consistent semi-supervised deep learning according to claim 1, characterized in that: The video data space enhancement: the video is composed of a video sequence F, from the video sequence F = [f1, f2, ..., f M ], starting from frame m, extract N frames at a frame rate of e, and obtain the coarse-grained expression of the action in the video x = [f m , f m+e , f m+2e ,…,f m+(N-1)e ]; spatially enhance the coarse-grained expression x of the video action with probability P to obtain the spatially enhanced expression α(x) of the video data. Spatial enhancement includes horizontal image flipping and random image cropping; The video data temporal enhancement processing can obtain the pre-action expression of fine-grained actions and the post-action expression of fine-grained actions; Pre-temporal action expression of fine-grained actions: From the video sequence F = [f1, f2,..., f M , we extract n frames at a frame rate of v1, where n < N, and then extract N - n frames at a frame rate of v2, with v1 > v2, to obtain the pre-temporal action expression β pre (x); Post-action expression of fine-grained action: From the video sequence F = [f1,f2,...,f M ], we extract n frames at v2 frame rate, and then extract Nn frames at v1 frame rate, v1>v2, and obtain the post-action expression β of fine-grained action post (x).

3. The human behavior recognition method based on consistent semi-supervised deep learning according to claim 1, characterized in that: The improved 3D-Resnet18 network includes 17 convolutional layers, the last of which is a fully connected layer. In the convolutional layers from layers 2 to 16, the Leaky-ReLU function is used instead of the ReLU, and Dropout is added after the fully connected layer to alleviate the overfitting problem of the model.

4. The human behavior recognition method based on consistent semi-supervised deep learning according to claim 2, characterized in that: In step (5), the supervisory signal L is calculated s The loss values are as follows: Select a small batch of videos from the labeled video set X For videos with tags, For video The corresponding label for the video Perform spatial enhancement processing on video data to obtain spatial enhancement expression of video data Will Use the improved 3D-Resnet18 network for training to obtain the predicted probability of each video belonging to its corresponding label Use the cross entropy loss function to calculate the probability of the recognition model prediction and the true category The cross entropy loss between:

5. The human behavior recognition method based on consistent semi-supervised deep learning according to claim 2, characterized in that: In step (7), the timing signal L is calculated d The loss values are as follows: For video x∈{X',U'}, perform video data temporal enhancement on video x to obtain the pre-action expression β of fine-grained action pre (x) and the post-action representation β of fine-grained actions post (x); β pre (x), β post (x) Use the improved 3D-Resnet18 network for training to obtain the predicted probability P(β pre (x)), P(β post (x)), calculate the Jensen-Shannon entropy divergence between action predictions after temporal enhancement of video data: P(β avg (x))=(P(β post (x))+P(β pre (x))) / 2 (2) L d =L KL (P(β pre (x)),P(β avg (x)))+L KL (P(β post (x)),P(β avg (x)))(5).

6. The human behavior recognition method based on consistent semi-supervised deep learning according to claim 2, characterized in that: In step (9), the pseudo supervisory signal L is calculated u The loss values are as follows: Select a small batch of videos from the unlabeled video set U For unlabeled videos, Perform spatial enhancement processing on video data to obtain spatial enhancement expression of video data Video Perform temporal enhancement processing on video data to obtain the front-end action expression of fine-grained action and post-action expression of fine-grained actions Will Use the improved 3D-Resnet18 network for training to obtain the predicted probability of each video belonging to its corresponding label Video The preceding action expression of fine-grained action and post-action expression of fine-grained actions The convolutional layer extracts the previous action feature H1 and the next action feature H2 respectively, and fuses the previous action feature H1 and the next action feature H2 to obtain the fused feature H, H = H1 + H2. The fused feature H is input into the fully connected layer for classification to obtain the predicted probability. Use pseudo labeling technology to obtain the threshold T of action category c t (c), when the maximum predicted probability Exceeding the predefined threshold T t (c) When the corresponding category as the predicted category; otherwise, calculate The predicted probability after fusion with fine-grained action features The cross entropy loss between them is used as the pseudo supervisory signal L u , 7. The human behavior recognition method based on consistent semi-supervised deep learning according to claim 6, characterized in that: The pseudo-labeling technique is used to set the threshold T t (c), comprising the following steps: For the small batch video set selected in step (4.3) Video Sample Perform spatial enhancement processing on video data to obtain spatial enhancement expression of video data Video Perform temporal enhancement processing on video data to obtain the front-end action expression of fine-grained action and post-action expression of fine-grained actions Will Use the improved 3D-Resnet18 network for training and obtain the prediction probability respectively Calculate the mean predicted probability of coarse and fine-grained actions Statistical probability of measurement The maximum value is greater than the threshold τ and the predicted category is the number of categories c σ t (c), where the threshold τ is set: Where I is an indicator function, which counts as 1 when the conditions in the brackets are met; Learning effect t (c) Normalization Convert the model's learning effect evaluation of each action to between (0-1): Use the nonlinear convex function M(x)=x / (2-x) to fit the convergence trend of the model and get the prediction The corresponding thresholds of each action are obtained; In order to reduce the noise data input of the model, the upper threshold τ is set min and the lower threshold τ max ; Threshold for comparative evaluation and the minimum threshold τ min The maximum threshold between the two is selected as the threshold T of the action at the current moment. t (c) like Re-comparison evaluation threshold and the maximum threshold T that has appeared in history max (c) Compare and select the maximum threshold between the two as the threshold T t (c) 8. The human behavior recognition method based on consistent semi-supervised deep learning according to claim 1, characterized in that: During the training of the improved 3D-Resnet18 network, the training round is EPOCH, and one round contains STEP training; the initial learning rate is η0; If the current total loss is less than the total loss obtained in the previous training, the stochastic gradient descent algorithm is used to update the network parameters; otherwise, the network parameters are not updated and the optimized improved 3D-Resnet18 network is obtained; within the EPOCH*STEP cycle, the cosine decay function is used to achieve dynamic changes in the learning rate within the range of [0-η0].

9. The human behavior recognition method based on consistent semi-supervised deep learning according to claim 1, characterized in that: In step (10), the improved 3D-Resnet18 network optimized in step (9) is loaded to perform human behavior recognition on the video that needs behavior recognition. The specific steps are as follows: Take a video V to be predicted with a length of S, and extract the video clip with a length of s after uniform frame extraction. Randomly select x∈(0,Ss) as the starting frame, input it into the improved 3D-Resnet18 network optimized in step (9), traverse the video, and select the action category c with the largest prediction confidence: c=argmax(P(v)); Repeat the sampling of the same action five times and take the mean P of the five predicted probabilities mean (V) is taken as the final prediction result of the video V; the action category corresponding to the maximum value of the prediction result is taken as the final prediction result class: class = argmax (P mean (V)).

Images