Semi-supervised learning video behavior identification method based on Soft-Teach
Through the semi-supervised learning method based on Soft-Teacher, combined with local feature learning and multi-head attention module, the problem of low time and accuracy of video behavior recognition model during data annotation and training is solved, and more efficient video behavior is achieved Identification and model generalization capabilities.
Patent Information
- Application Number
- CN202411069987.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-19
AI Technical Summary
Existing video behavior recognition models take a long time and have low accuracy during data annotation and training, especially the problems of excessive training time and low recognition accuracy caused by unsupervised learning methods.
Using the semi-supervised learning method based on Soft-Teacher, the video sample data is divided into labeled and labelless data, and iterative training is used to use the student model and the teacher model, and the local feature learning module and the multi-head attention calculation module are used to generate pseudo-labels. Perform comparative learning to reduce dependence on labeled data and improve model generalization capabilities.
Effectively reduce the data annotation workload of the video behavior recognition model, improve the recognition performance and generalization ability of the model, reduce the dependence on the annotated data, avoid the problem of difficult model convergence and low recognition accuracy, and improve the video behavior recognition accuracy.
Smart Images

Figure CN118982869A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video processing, and in particular to a semi-supervised learning video behavior recognition method based on Soft-Teacher. Background Art
[0002] Behavior recognition plays a vital role in many application scenarios such as intelligent monitoring, video sequence analysis and human-computer interaction. In recent years, with the continuous development of computer science and technology, the field of artificial intelligence has also flourished, which has greatly promoted the intelligence of computers and enabled computers to complete some simple tasks such as image classification. Behavior recognition is a research hotspot in the fields of control science and computer science. As a key technology for machines to understand the world and human behavior, it has also attracted widespread attention from academia and industry in recent years. It has a great social demand in the fields of smart elderly care and smart city security construction. At the same time, since most of the information that people can access is dynamic, the video field is more in line with real life than the image field.
[0003] With the continuous optimization of behavior recognition models, its accuracy and speed are constantly improving, but the current video data set is difficult to produce, and the labeling and positioning of data requires a lot of time and manpower, which greatly restricts the development of the video field. Unsupervised or semi-supervised learning methods can solve this problem to a certain extent, but since unsupervised learning does not require labels to guide learning, clustering or K-nearest neighbor methods are used to classify and judge data, which will result in low accuracy or too long training time.
[0004] The semi-supervised learning method combines supervised learning and unsupervised learning, that is, it uses most of the unlabeled data and a small amount of labeled data for training. It can not only reduce the model's strong dependence on labeled data, thereby reducing the time and labor cost of data annotation, but also use part of the labeled data to train the model, thereby ensuring the training speed and accuracy of the model.
[0005] Therefore, it is an urgent problem for those skilled in the art to propose a semi-supervised learning video behavior recognition method based on Soft-Teacher to solve the difficulties existing in the prior art. Summary of the invention
[0006] In view of this, the present invention provides a semi-supervised learning video behavior recognition method based on Soft-Teacher to solve the problem of heavy video data labeling tasks, thereby restricting the development of video behavior recognition models.
[0007] In order to achieve the above object, the present invention adopts the following technical solution:
[0008] The semi-supervised learning video behavior recognition method based on Soft-Teacher includes:
[0009] Obtain video sample data, and divide the video sample data into labeled data and unlabeled data;
[0010] The labeled data and unlabeled data are input into the trained video behavior recognition model to obtain the final classification result of the video behavior category.
[0011] In the above method, optionally, the video behavior recognition model includes: a model to be trained and an exponential average exponential model of the model to be trained;
[0012] The model to be trained is defined as the student model, the exponential average exponential model of the model to be trained is defined as the teacher model, and the teacher model is the sum of multiple iterative versions of the model to be trained.
[0013] In the above method, optionally, the student model includes: a feature extractor and a classifier;
[0014] Feature extractor, used to process labeled data and unlabeled data to obtain feature maps;
[0015] The classifier is used to classify the feature map obtained by the feature extractor to obtain the final classification result;
[0016] The final classification results include: vectors for supervised learning and vectors for semi-supervised learning.
[0017] In the above method, optionally, the feature extractor includes: a local feature learning module and a plurality of multi-head attention calculation modules connected in sequence;
[0018] The local feature learning module consists of three parts. The first part includes: 1×1×1 three-dimensional convolution, batch normalization and activation function; the second part includes: 1×3×3 deep convolution, batch normalization and activation function; the third part includes: 1×1×1 three-dimensional convolution and batch normalization.
[0019] In the above method, optionally, the labeled data and the unlabeled data are mapped to a high-dimensional feature space through a three-dimensional convolution of size 1×1×1 in the first part, and then the nonlinear information of the features is increased through batch normalization and activation function;
[0020] The feature map processed in the first part is then processed by the single-channel 1×3×3 deep convolution, batch normalization and activation function in the second part to obtain the feature map after single-channel feature extraction;
[0021] The feature map after single-channel feature extraction obtained after processing in the second part is then processed by 1×1×1 three-dimensional convolution and batch normalization in the third part. The convolution operation is used to map the feature map after single-channel feature extraction to a low-dimensional space to extract key features, and finally obtain a feature map containing local feature information.
[0022] The above method optionally inputs a feature map containing local feature information into multiple multi-head attention calculation modules connected in sequence to obtain a spatiotemporal feature map.
[0023] In the above method, optionally, the classifier includes: a sequence pooling module and a multi-layer perceptron module;
[0024] The spatiotemporal feature map obtains the weight of the current spatiotemporal feature map through the sequence pooling module, and multiplies the weight of the current spatiotemporal feature map with the current spatiotemporal feature map to obtain a weighted spatiotemporal feature map;
[0025] The weighted spatiotemporal feature map is input into the multi-layer perceptron module to obtain the final classification result.
[0026] The above method optionally inputs the unlabeled data into the teacher model and then outputs a pseudo label, compares the pseudo label with the vector used for semi-supervised learning, and compares the pseudo label with the vector used for semi-supervised learning to obtain a trained recognition model.
[0027] It can be seen from the above technical solution that, compared with the prior art, the present invention provides a semi-supervised learning video behavior recognition method based on Soft-Teacher, which has the following beneficial effects:
[0028] 1) To address the problem of spatial information loss in Transformer calculations, a local feature learning module is introduced to learn local feature information, which can be used to better learn spatial information in the future; 2) The features containing local feature information are then input into the multi-scale visual Transformer to obtain a feature map using a multi-head attention mechanism, so that the feature map contains more contextual information, and the final output feature map is used for behavior category prediction, which can then predict the behavior category of the characters in the video, such as whether they are riding a bicycle or a horse; 3) The multi-scale visual Transformer model is used as a student model for continuous optimization training, and the exponential average version of the multi-scale visual Transformer is used as a teacher model to generate pseudo labels for semi-supervised learning training, reducing the workload of annotating video samples, which is beneficial to the promotion and actual deployment of the model, while improving the generalization ability of the model; and the introduction of the teacher model can guide the training of the model to be trained, avoid the problem of difficult model convergence and low recognition accuracy, and improve the recognition performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0030] Figure 1 This is a network architecture diagram of the video behavior recognition model disclosed in the present invention;
[0031] Figure 2 This is a network architecture diagram of the local feature learning module disclosed in the present invention;
[0032] Figure 3 This is a diagram of the classifier network architecture disclosed in the present invention;
[0033] Figure 4 A schematic diagram of the result of behavior recognition of a person in a video disclosed in one embodiment of the present invention;
[0034] Figure 5 A schematic diagram of the results of behavior recognition of a person in a video disclosed in another embodiment of the present invention. DETAILED DESCRIPTION
[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0036] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0037] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.
[0038] The present invention discloses a semi-supervised learning video behavior recognition method based on Soft-Teacher, comprising:
[0039] Obtain video sample data, and divide the video sample data into labeled data and unlabeled data;
[0040] The labeled data and unlabeled data are input into the trained video behavior recognition model to obtain the final classification result of the video behavior category.
[0041] For further information, see Figure 1 As shown, the video behavior recognition model includes: a model to be trained and an exponential average exponential model of the model to be trained;
[0042] The model to be trained is defined as the student model, the exponential average exponential model of the model to be trained is defined as the teacher model, and the teacher model is the sum of multiple iterative versions of the model to be trained.
[0043] Furthermore, the student model includes: a feature extractor and a classifier;
[0044] Feature extractor, used to process labeled data and unlabeled data to obtain feature maps;
[0045] The classifier is used to classify the feature map obtained by the feature extractor to obtain the final classification result;
[0046] The final classification results include: vectors for supervised learning and vectors for semi-supervised learning.
[0047] Furthermore, the feature extractor includes: a local feature learning module and a plurality of multi-head attention calculation modules connected in sequence;
[0048] See also Figure 2 As shown in the figure, the local feature learning module includes three parts. The first part includes: 1×1×1 three-dimensional convolution, batch normalization and activation function; the second part includes: 1×3×3 depth convolution, batch normalization and activation function; the third part includes: 1×1×1 three-dimensional convolution and batch normalization.
[0049] Furthermore, the labeled data and unlabeled data are mapped to a high-dimensional feature space through a 3D convolution of size 1×1×1 in the first part, which is used for subsequent feature extraction based on high-dimensional tensors, so that global information can be extracted and the problem of information loss in low-dimensional conditions can be avoided. Then, batch normalization and activation functions are used to increase the nonlinear information of the features.
[0050] The feature map processed in the first part is then processed by the single-channel 1×3×3 deep convolution, batch normalization and activation function in the second part to obtain the feature map after single-channel feature extraction; it is different from the conventional convolution where one convolution kernel is responsible for all channels, but on the contrary, one convolution kernel is only responsible for one channel, which can be used to extract features and has a lower computational cost than conventional convolution;
[0051] The feature map after single-channel feature extraction obtained after processing in the second part is then processed by 1×1×1 three-dimensional convolution and batch normalization in the third part. The convolution operation is used to map the feature map after single-channel feature extraction to a low-dimensional space to extract key features, and finally obtain a feature map containing local feature information.
[0052] Therefore, the local feature learning module only calculates the local feature information for each frame space, and then obtains a feature map containing the local feature information.
[0053] Furthermore, the feature map containing local feature information is input into multiple multi-head attention calculation modules connected in sequence to obtain a spatiotemporal feature map.
[0054] Specifically, the feature map containing local feature information is then input into the multi-head attention calculation module of the multi-scale visual transformer to calculate the spatiotemporal feature map for the final video features. The overall calculation process of the multi-scale visual transformer is as follows:
[0055] First, the 5-dimensional input feature map is converted into a vector form, where a 3-dimensional convolution is used to convert the feature map from 5 dimensions to 3 dimensions, that is, Convert to
[0056] The input vector is calculated through three learnable weight matrices to obtain three vectors, namely the query vector, the key vector and the value vector. The calculation process can be simply expressed as:
[0057] Q=XW Q
[0058] K=XW K
[0059] V=XW V
[0060] The attention feature map is calculated based on the three vectors obtained, and the calculation process can be expressed as:
[0061]
[0062] The attention calculation module in the architecture is a multi-head attention calculation mechanism, which is equivalent to performing the attention calculation process multiple times. The overall calculation process of a module can be expressed as:
[0063] X1=MHA(LN(X))+X
[0064] Block(X)=MLP(LN(X1))+X1
[0065] Among them, MHA stands for multi-head attention calculation, LN is layer normalization, which performs normalization operation on the layer, and MLP is the fully connected layer to obtain the final feature map, that is, the spatiotemporal feature map.
[0066] Furthermore, the classifier includes: a sequence pooling module and a multi-layer perceptron module;
[0067] The spatiotemporal feature map obtains the weight of the current spatiotemporal feature map through the sequence pooling module, and multiplies the weight of the current spatiotemporal feature map with the current spatiotemporal feature map to obtain a weighted spatiotemporal feature map;
[0068] The weighted spatiotemporal feature map is input into the multi-layer perceptron module to obtain the final classification result.
[0069] For details, see Figure 3 As shown, the sequence pooling module includes: layer normalization, average pooling and activation function; the multi-layer perceptron module includes: layer normalization and linear layer.
[0070] The overall calculation process can be expressed as:
[0071]
[0072]
[0073] Among them, g represents the average pooling technology, and the final weighted feature vector is represented by Z. Then, the feature vector Z is mapped to the feature space of the true label through a multi-layer perceptron to obtain the final prediction vector.
[0074] The above is the prediction process of the student model. It outputs two vectors in total, one is the prediction vector for supervised learning, which is finally compared with the real label by cross entropy; the other is the vector for semi-supervised learning, which is compared with the pseudo label of the teacher model.
[0075] Furthermore, the labeled data is input into the teacher model and a pseudo-label is output, and the pseudo-label is compared with the vector used for semi-supervised learning to obtain a trained video behavior recognition model.
[0076] Specifically, the specific steps of semi-supervised learning are as follows:
[0077] The input data includes labeled data and unlabeled data. The ratio of labeled data to unlabeled data is selected according to the characteristics of the downstream tasks. The unlabeled data will be input into the student model and the teacher model for semi-supervised learning and training, and the labeled data will be input into the student model for supervised learning and training.
[0078] There are two types of unlabeled data. One is the data that has been randomly erased and has higher recognition difficulty and is input into the student model for prediction. The other is the data that has been simply cropped and is input into the teacher model.
[0079] For the construction of the teacher model, the present invention adopts the exponential average index value of the student model weight to obtain the teacher model, because the weight value obtained by the exponential average index is more stable and reliable. Among them, the calculation method of the weight update of the teacher model is as follows:
[0080] v t =β·v t-1 +(1-β)·θ t
[0081] Among them, v t The weights obtained after a complete training of the entire current dataset. Since the early weights are unstable and not reliable, the early β is 0.9, and after the weights are stable, β is 0.999.
[0082] For the construction of pseudo-label loss, JS divergence is used to calculate. JS divergence is based on KL divergence. KL divergence is used to measure the difference between two distributions. JS divergence is a "symmetric KL divergence" proposed for the asymmetry problem of KL divergence. It can avoid the problem that KL divergence is meaningless when the two distributions are far apart and completely irrelevant. The calculation method is as follows:
[0083]
[0084] Considering that the prediction values of the teacher model are more stable and more credible, the necessity of aligning the teacher model with the student model should be reduced, so the unequal JS loss is proposed:
[0085]
[0086] Finally, the loss of the entire model should consist of two parts: one is supervised learning loss and the other is semi-supervised learning loss:
[0087] Loss=Loss sup +Loss semi
[0088] The supervised learning loss can be expressed as:
[0089]
[0090] Among them, x and y represent the predicted value of the model and the true label value corresponding to the input video, respectively, and the similarity of the two results is calculated by cross entropy loss.
[0091] The present invention proposes a video behavior recognition framework based on semi-supervised learning, wherein for the video recognition model, a local feature learning module is also proposed, so as to overcome the problem that the Transformer calculation will lose the spatial bias, and then optimize the training of the Transformer; a classifier module including sequence pooling is introduced, which mainly performs a pooling operation on the data of the entire sequence, so as to better extract the key information in the sequence for the final prediction; for the semi-supervised learning framework, the student model inputs a set of randomly erased video frames, and the teacher model inputs a set of randomly cropped video frames, wherein the teacher model is the exponential average exponential value of the student model weight, which is more stable and accurate, and the pseudo-label generated by the teacher model is finally used for comparison calculation with the prediction of the student model, and the JS divergence that can avoid the disappearance of the gradient is introduced here for calculation. Finally, the whole framework includes a semi-supervised learning part and a supervised learning part, which can well reduce the dependence on labeled data while supervised learning can maintain the reliability of training.
[0092] According to the content disclosed in the present invention, the recognition performance and generalization ability of the video behavior recognition model can be improved; in addition, due to the introduction of the semi-supervised learning training method, the cost of the preliminary preparation of video behavior recognition data can be reduced, which is conducive to the promotion and practical application deployment of the model; and due to the guidance of the teacher model, the problem of difficult convergence of the model and low recognition accuracy in unsupervised learning can be solved.
[0093] The video behavior recognition model of the present invention can effectively recognize the behavior of the characters in the video, and further processing can be performed based on the prediction results of the model for the character behavior, such as classroom behavior management or warning of dangerous behavior of the elderly and children at home. In a specific embodiment of the present invention, the model's results of character behavior recognition in the video refer to Figure 4 and Figure 5 As shown, Figure 4 The middle behavior is identified as high jump. Figure 5The behavior recognition result in the middle is riding a bicycle.
[0094] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0095] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A semi-supervised learning video behavior recognition method based on Soft-Teacher, characterized in that: Includes the following: Obtain video sample data, and divide the video sample data into labeled data and unlabeled data; The labeled data and unlabeled data are input into the trained video behavior recognition model to obtain the final classification result of the video behavior category.
2. The method for semi-supervised learning video behavior recognition based on Soft-Teacher according to claim 1, characterized in that: The video behavior recognition model includes: a model to be trained and an exponential average exponential model of the model to be trained; The model to be trained is defined as the student model, the exponential average exponential model of the model to be trained is defined as the teacher model, and the teacher model is the sum of multiple iterative versions of the model to be trained.
3. The method for semi-supervised learning video behavior recognition based on Soft-Teacher according to claim 2, characterized in that: The student model includes: feature extractor and classifier; Feature extractor, used to process labeled data and unlabeled data to obtain feature maps; The classifier is used to classify the feature map obtained by the feature extractor to obtain the final classification result; The final classification results include: vectors for supervised learning and vectors for semi-supervised learning.
4. The method for semi-supervised learning video behavior recognition based on Soft-Teacher according to claim 3 is characterized in that: The feature extractor includes: a local feature learning module and a plurality of multi-head attention calculation modules connected in sequence; The local feature learning module consists of three parts. The first part includes: 1×1×1 three-dimensional convolution, batch normalization and activation function; the second part includes: 1×3×3 deep convolution, batch normalization and activation function; the third part includes: 1×1×1 three-dimensional convolution and batch normalization.
5. The method for semi-supervised learning video behavior recognition based on Soft-Teacher according to claim 4, characterized in that: The labeled data and unlabeled data are mapped to a high-dimensional feature space through a 3D convolution of size 1×1×1 in the first part, and then the nonlinear information of the features is increased through batch normalization and activation function; The feature map processed in the first part is then processed by the single-channel 1×3×3 deep convolution, batch normalization and activation function in the second part to obtain the feature map after single-channel feature extraction; The feature map after single-channel feature extraction obtained after processing in the second part is then processed by 1×1×1 three-dimensional convolution and batch normalization in the third part. The convolution operation is used to map the feature map after single-channel feature extraction to a low-dimensional space to extract key features, and finally obtain a feature map containing local feature information.
6. The method for semi-supervised learning video behavior recognition based on Soft-Teacher according to claim 5, characterized in that: The feature map containing local feature information is input into multiple multi-head attention calculation modules connected in sequence to obtain a spatiotemporal feature map.
7. The method for semi-supervised learning video behavior recognition based on Soft-Teacher according to claim 6, characterized in that: The classifier includes: sequence pooling module and multi-layer perceptron module; The spatiotemporal feature map obtains the weight of the current spatiotemporal feature map through the sequence pooling module, and multiplies the weight of the current spatiotemporal feature map with the current spatiotemporal feature map to obtain a weighted spatiotemporal feature map; The weighted spatiotemporal feature map is input into the multi-layer perceptron module to obtain the final classification result.
8. The method for semi-supervised learning video behavior recognition based on Soft-Teacher according to claim 6, characterized in that: After inputting the unlabeled data into the teacher model, the pseudo-label is output, and the pseudo-label is compared with the vector used for semi-supervised learning to obtain a trained video behavior recognition model.
Citation Information
Patent Citations
Optical field high-resolution deconvolution method and system based on convolutional neural network
CN110441271A
Radar radiation source identification method based on DCNN and Transform
CN114296041A
Steel plate surface defect detection method and system based on semi-supervised learning
CN116863274A
Target detection test time domain adaptation method for image damage
CN118015423A
Weak supervision video violence detection method based on time domain enhancement and comparative learning
CN118015507A
Cited By
Unsupervised power video action positioning method, system and equipment and storage medium
CN119888566A
