Behavior recognition method, electronic device and storage medium

Abstract

The present invention discloses a behavior recognition method, an electronic device and a storage medium, wherein the behavior recognition method comprises: obtaining labeled and unlabeled data, and preprocessing the obtained labeled and unlabeled data respectively; constructing a large model and a small model, and using the large model to perform distillation training on the small model; the large model adopts a first encoder and decoder structure, and includes five branches, namely a text feature branch, an audio feature branch, a video frame prediction branch, a classifier branch and a video sequence comparison learning branch; the small model distillation part is only composed of a second encoder, and the parameters of the second encoder are trained by distillation of the large model; the large model and / or the small model obtained by training are used to recognize the behavior. Multimodal information such as audio, text and video frames is used to generate optimization targets to supervise the model, so that the pre-trained network has a more balanced feature extraction capability, which can be efficiently applied to downstream subtasks.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a behavior recognition method. Background Art

[0002] Behavior recognition has a wide range of applications in scenarios such as smart security and smart campuses. Behavior recognition includes human posture, human-object interaction, and human-human interaction. The current mainstream technologies include video classification, spatiotemporal behavior detection, etc. Behavior recognition needs to consider temporal information while considering spatial information. Many categories in behavior recognition have a low probability of occurrence (such as violent behaviors such as fighting, people falling, traffic accidents, etc.), and the problem of too few samples leads to poor model generalization ability. At the same time, as networks such as Vision Transformer surpass CNN, due to factors such as the large number of parameters and large model capacity, training is very difficult, and the training set data volume is very high, which further increases the difficulty of specific tasks in the field of behavior recognition. At present, many methods use unsupervised pre-training to solve this problem, but there are problems such as a single data set and low utilization of effective data information, which leads to the pre-training model being applicable only to a few downstream tasks.

[0003] Many current unsupervised pre-training methods only focus on a single task, such as image restoration (such as Masked Autoencoders), which may result in the extracted features being unrepresentative and too single, unable to bring out all the information of the data used, and ultimately resulting in only improvements on limited tasks. At the same time, most current methods use a single image instead of a video sequence for unsupervised training, which also results in the pre-trained model not being better adapted to the behavior recognition task. Summary of the invention

[0004] In order to solve the problems of complex and diverse behavior recognition tasks, small number of samples, and high data volume requirements of the model, the present invention proposes a method of unsupervised training on large-scale data. This method can allow the pre-trained model to obtain a more balanced feature extraction capability, which is convenient for subsequent downstream tasks. That is, by fine-tuning the network with small samples, higher accuracy and generalization ability can be achieved, which can greatly reduce the cost of data annotation and the difficulty of algorithm implementation.

[0005] The technical solution of the present invention is:

[0006] The present invention first provides a behavior recognition method, comprising:

[0007] Acquire labeled and unlabeled data, and preprocess the acquired labeled and unlabeled data respectively;

[0008] Construct a large model and a small model, use the large model to perform distillation training on the small model, and obtain two types of models, large and small, that can be used for downstream tasks in one training; during training, the inputs of the large model and the small model are the same, and the optimization target of the small model is the output of the large model; the large model adopts a first encoder and decoder structure, which includes five branches, namely, a text feature branch, an audio feature branch, a video frame prediction branch, a classifier branch, and a video sequence comparison learning branch. The input data of all branches of the large model are image sequences and share the parameters of the encoder. The outputs of the text feature branch, the audio feature branch, the classifier branch, and the video sequence comparison learning branch are the outputs of the encoder part, and the video frame prediction branch additionally includes a decoder part for realizing the prediction of video frames; the distillation part of the small model is only composed of the encoder, and the parameters of the second encoder are trained through the distillation of the large model;

[0009] The trained large model and / or small model is used to identify the behavior.

[0010] Preprocess the acquired labeled and unlabeled data, including:

[0011] Merge the labeled data and open source commercially available data into a segment of S1 seconds, containing a summary dataset of C categories;

[0012] The unlabeled data is divided into segments of S2 seconds in length, and it is determined whether there is a corresponding subtitle file in the segment timestamp; if there is a subtitle file, its corresponding natural language feature expression is obtained through the language model; if it has corresponding audio, its corresponding audio feature expression is obtained through the audio model; if there is a title, and S2 / the entire video length is >50%, the title text is passed through the text processing model to obtain its corresponding natural language feature expression.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] 1. The present invention makes full use of effective information in unsupervised video data, such as audio, title subtitles, annotation information, and video frame continuity, and processes this information separately through multiple task branches. The use of multiple branches can improve the performance of the large model, so that the large model can obtain better feature representation and generalization capabilities, and reduce the data acquisition and annotation costs of downstream tasks. The performance of the large model has been improved, and the knowledge distillation method is used to supervise and guide the small model, so that the performance of the small model is naturally improved. Multimodal information such as audio, text, and video frames is used to generate optimization targets (labels or pseudo-labels) to supervise the model, so that the pre-trained network has a more balanced feature extraction capability, which can be efficiently applied to downstream subtasks, and the requirements of downstream tasks for the amount of training data can be reduced by 2-3 orders of magnitude, greatly reducing the difficulty of algorithm implementation.

[0015] 2. One-time training can support multiple tasks, such as video interpolation, video super-resolution, video classification, spatiotemporal behavior detection, etc. 3. Through specific data preprocessing methods, it can efficiently process labeled and unlabeled data and be used for semi-supervised training. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the model training method of the present invention;

[0017] Figure 2 It is a principle block diagram of the model of the present invention. DETAILED DESCRIPTION

[0018] This embodiment provides a behavior recognition method, and the training of the recognition model used by the recognition method uses both labeled data and unlabeled data, which is called semi-supervised training. Because currently, in addition to private data sets, there are relatively many commercially available open source data sets with manual annotations, such as Google Ava, Kinetics 700, etc.

[0019] By efficiently combining unsupervised and supervised learning through multi-task learning, data can be fully and efficiently used to improve the pre-trained model's ability to represent various underlying or abstract features, thereby reducing the data requirements of downstream tasks while improving their generalization capabilities.

[0020] The behavior recognition method of the present invention, see Figure 1 ,include:

[0021] Acquire labeled and unlabeled data, and preprocess the acquired labeled and unlabeled data respectively;

[0022] Build a large model and a small model, use the large model to perform distillation training on the small model, and obtain two types of models, large and small, that can be used for downstream tasks in one training; the large model uses an encoder and decoder structure, which contains five branches, namely, a text feature branch, an audio feature branch, a video frame prediction branch, a classifier branch, and a video sequence contrast learning branch. The input data of all branches of the large model are image sequences and share the parameters of the encoder. The outputs of the text feature branch, the audio feature branch, the classifier branch, and the video sequence contrast learning branch are the outputs of the encoder part. The video frame prediction branch additionally includes a decoder part for realizing the prediction of video frames; the distillation part of the small model is only composed of the encoder; the input of the small model is consistent with the large model, and the optimization target of the small model is the output of the encoder part of the large model;

[0023] The large and / or small models obtained through training are used to identify the behavior. That is, one of the large model and the small model can be used alone, or both can be used.

[0024] In one embodiment, the acquired labeled and unlabeled data are preprocessed, including:

[0025] For labeled data:

[0026] Merge your own labeled data and open source commercially available data into a segment of S1 (S1 is generally 3-10) seconds, containing a summary data set of C (C is an integer greater than or equal to 0) categories.

[0027] For unlabeled data acquisition:

[0028] A large amount of video data is obtained by crawling the Internet, which may also include audio tracks (Mandarin is preferred), subtitle files (Chinese subtitles are preferred), titles / tags, etc.

[0029] The acquired video data is divided into segments of S2 (S2 is generally 3-10 seconds), and it is determined whether there is a corresponding subtitle file in the segment timestamp. If so, the corresponding natural language feature expression is obtained through the language model BERT (pre-trained), and the feature dimension scale is D1 (to be consistent with the BERT output dimension, D1=768). If it has corresponding audio (if there are multiple audio tracks, Mandarin is preferred), the corresponding audio feature expression is obtained through the audio model (AST), and the feature dimension scale is D2 (D1=D2). If there is a title, and S2 / the entire video duration is >50%, the title text is passed through the text processing model to obtain its corresponding natural language feature expression, and the feature dimension scale is D3 (D3=D1).

[0030] If the video is short and the title exists, the title text is used as the text feature. If the video is long and there is a corresponding subtitle file, the subtitle text is used as the text feature. If the text feature or audio feature does not exist or does not meet the requirements, the corresponding feature is marked as -1 (i.e., it does not exist).

[0031] The specific structures of the large model and the small model are as follows Figure 2 shown.

[0032] The overall structure of the large model is an encoder-decoder structure, which includes a text feature branch, an audio feature branch, a video frame prediction branch, a classifier branch, and a video sequence comparison learning branch. The large model encoder has only one output, and the subsequent multiple task branches use this encoder output as input.

[0033] The distillation part of the small model is only composed of the encoder, named the encoder small model. The input of the small model is consistent with the large model. The optimization target of the small model is the output of the large model encoder. The encoder of the small model has a similar structure to the encoder of the large model (for example, only the width, depth or hidden layer feature dimension is different). The small model no longer needs other branches to supervise the training because it already has the large model as a supervisory guide.

[0034] The encoder and decoder use a Transformer-based network, and CNN, LSTM and other networks can also be used.

[0035] Text feature branch:

[0036] Text features have a high information density. Although this method does not manually clean and annotate the text and contains a large amount of noisy data, the massive amount of data on the Internet can still allow the algorithm model to extract valuable information and map it to the model parameters. The extraction of text features uses a pre-trained fixed-parameter BERT model to extract text information such as titles / video descriptions and subtitles corresponding to video clips offline. The loss function uses KL divergence, and the loss is named text feature loss (loss text).

[0037] Audio feature branch:

[0038] Audio is not limited to speech, but refers to all sounds. Although the amount of audio information is less than that of video images, it often contains a lot of effective information. Therefore, audio information is also very important for behavior recognition / video classification. For example, hearing "keyboard sound" can be associated with "sitting in front of a computer to work", hearing "engine roar" can be associated with "driving a vehicle", and "gunshots" and "quarrels" are also related to "violent behavior", so audio features can be used as supervisory information to further extract effective information from unlabeled video data.

[0039] The audio features are extracted offline using a pre-trained fixed-parameter AST model to extract the audio in the time-corresponding video clips. The audio features are only used as supervisory information, and the loss function uses KL divergence. The loss is named audio feature loss (loss audio).

[0040] Video frame prediction branch:

[0041] This branch is unsupervised learning and is the core branch of this method. This branch still exists when audio, text, and annotation tags do not exist. This branch borrows from the training method of BERT, which masks part of the text in the sentence and predicts the masked words through the context. This branch uses a similar idea to input a video sequence, mask its key frames, and predict the key frames as targets. In order to increase the difficulty of prediction and enable the network to learn effective features, the key frames and the frames on both sides need to be separated by a certain length of time (for example, 0.5 seconds on both sides, not just adjacent frames). At the same time, it is necessary to ensure that there is no scene / shot switching, and the samples of shot / scene switching are filtered out by the pysceandetect tool. The L2 loss function is used to measure the accuracy of video frame prediction, and the loss is named video frame loss (loss L2).

[0042] Classifier branch:

[0043] This branch is supervised training, which is consistent with the general behavior classification or video classification method and uses cross entropy loss. The main purpose of this branch is to make full use of the labeled data to improve the representation ability of the model. The loss is named cross entropy loss.

[0044] Video sequence contrastive learning branch:

[0045] This branch is unsupervised learning based on contrastive learning and is the core branch of this method. This branch still exists when audio, text, and annotation tags do not exist. First, the input video clip is randomly sampled in both the time domain and the spatial domain as sampling 1, and then the same video clip or other video clips are sampled a second time in both the time domain and the spatial domain as sampling 2, and features are extracted through an encoder model with shared parameters. If sampling 1 and sampling 2 are sampled from the same video clip, then their semantic information should be similar, otherwise they are not similar. The contrastive loss function is used to measure the features of the two samples, and the loss is named contrastive learning loss.

[0046] Small model distillation branch part:

[0047] Since unsupervised training contains a large amount of noisy data, the small model has fewer parameters and poor noise resistance, it is difficult to achieve ideal results by directly training the small model. Therefore, this method uses the knowledge distillation method to train the small model to ensure that the small model is similar to the large model. This allows the small model to better reduce the redundancy of the large model with limited accuracy loss, "replicate" the knowledge of the large model through relatively few parameters, and train the large model and the small model to meet different task requirements. By ensuring that the large model and the small model have similar structures (for example, only the width, depth or hidden layer feature dimensions are different), the large model can be better used to distill the small model.

[0048] The present invention adopts the method of joint training of small models and large models to perform knowledge distillation, with the aim of improving the accuracy of the small model. Even if the large model has converged, the distribution of its output will change with more iterations. Although this change will not improve the accuracy of the large model, it is equivalent to using multiple different large models to train the small model, which will make the features extracted by the small model more robust.

[0049] The structure of the small model only contains the encoder part, named encoder small model. The input of the small model is consistent with the large model. The optimization target of the small model is the output of the encoder part of the large model. The loss is measured by L2 Loss, and the loss is named distillation loss (loss L2 small).

[0050] The unsupervised and supervised branches are efficiently combined through multi-task learning, and all losses are finally weighted and summarized, loss = A1*loss_text + A2*loss_audio + A3*loss_crossentroy + A4*loss_L2 + A5*loss_contrastive + A6*loss_L2_small (where A1, A2, A3, A4, A5 and A6 are weighting coefficients. A weighting coefficient greater than 0 indicates that this branch exists. Its value can be adjusted according to the importance of different tasks).

[0051] Model training method:

[0052] The video clips were sampled by random sampling with uniform distribution, and random cropping and color jitter were performed in the spatial domain. The optimizer used stochastic gradient descent (SGD), and the initial basic learning rate and weight decay parameters of the network were 0.01 and 0.00005 respectively. Four GPU server nodes were used, each with 8 NVIDIA GPUs. Distributed data parallelism was used and training was randomly initialized. The batch size was set to 128, and a total of 200 rounds were iterated. The first 10 epochs used warmup to gradually increase the learning rate to 0.01, and the learning rate was reduced by half every 40 rounds.

[0053] Pre-trained model application:

[0054] The encoder part can be used for pre-training models for tasks such as behavior recognition / video classification, spatiotemporal behavior detection, etc. On platforms with sufficient computing power, a large model structure and its pre-training parameters can be considered. On platforms with high computing power and power consumption requirements, a small model and its pre-training parameters can be considered. The encoder-decoder part can also be applied to other fields, such as video interpolation, video super-resolution, etc.

[0055] The above method allows the pre-trained model to obtain more balanced feature extraction capabilities through various labeled and unlabeled supervision information, which is convenient for subsequent downstream tasks, that is, by fine-tuning the network with small samples, higher accuracy and generalization can be achieved. It can effectively solve the problem of fewer samples in some behavior recognition tasks, such as violent behaviors such as fighting, falls, traffic accidents, etc.

[0056] An embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the behavior recognition method of the above embodiment.

[0057] An embodiment of the present invention also provides a readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the behavior recognition method of the above embodiment.

Claims

1. A behavior recognition method, characterized in that: include: Acquire labeled and unlabeled data, and preprocess the acquired labeled and unlabeled data respectively; Build a large model and a small model, use the large model to perform distillation training on the small model, and obtain two types of models, large and small, that can be used for downstream tasks in one training; during training, the inputs of the large model and the small model are the same, and the optimization target of the small model is the output of the large model; the large model adopts the first encoder and decoder structure, which contains five branches, namely text feature branch, audio feature branch, video frame prediction branch, classifier branch, and video sequence comparison learning branch; The text feature branch extracts text features from the image sequence input by the large model through the first encoder; The audio feature branch extracts audio features from the image sequence input by the large model through the first encoder; The video frame prediction branch extracts the spatiotemporal motion state features in the image sequence input by the large model through the first encoder, and predicts the spatiotemporal motion state features of the video frames deleted from the input image sequence through the decoder; the classifier branch extracts the abstract behavior category features in the image sequence input by the large model through the first encoder; the video sequence contrast learning branch extracts the homology discrimination ability features in the image sequence input by the large model through the first encoder; The small model distillation part is only composed of the second encoder, and the parameters of the second encoder are trained by the large model distillation; The trained large model and / or small model is used to identify the behavior.

2. The behavior recognition method according to claim 1, characterized in that: Preprocess the acquired labeled and unlabeled data, including: Merge the labeled data and open source commercially available data into a segment of S1 seconds, containing a summary dataset of C categories; The unlabeled data is divided into segments of length S2 seconds, and it is determined whether there is a corresponding subtitle file in the segment timestamp; if a subtitle file exists, its corresponding natural language feature expression is obtained through the language model; if the video segment has corresponding audio, the audio feature expression corresponding to the video segment is obtained through the audio model; if there is a title, and S2 / the entire video length is > 50%, the title text is passed through the text processing model to obtain the natural language feature expression corresponding to the title text.

3. The behavior recognition method according to claim 1, characterized in that: When the large model trains the small model, the loss used for back propagation to correct the weight coefficients in the large and small models is: loss = A1*loss_text + A2*loss_audio + A3*loss_crossentroy + A4*loss_L2 +A5*loss_contrastive + A6*loss_L2_small Where A1, A2, A3, A4, A5 and A6 are weighted coefficients. A weighted coefficient greater than 0 indicates that this branch exists. loss_text is the loss of the text feature branch. loss_audio is the loss of the audio feature branch. loss_crossentroy is the loss of the classifier branch. loss_L2 is the loss of the video frame prediction branch. loss_contrastive is the loss of the video sequence contrast learning branch. loss_L2_small is the distillation loss of the small model.

4. The behavior recognition method according to claim 1, characterized in that: The loss function of the text feature branch adopts KL divergence.

5. The behavior recognition method according to claim 1, characterized in that: The loss function of the audio feature branch adopts KL divergence.

6. The behavior recognition method according to claim 1, characterized in that: The video frame prediction branch uses the L2 loss function to measure the video frame prediction accuracy.

7. The behavior recognition method according to claim 1, characterized in that: The classifier branch adopts cross entropy loss.

8. The behavior recognition method according to claim 1, characterized in that: The video sequence contrastive learning branch adopts a contrastive loss function.

9. An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the behavior recognition method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the behavior recognition method as described in any one of claims 1 to 8.

Images