Small sample video identification method and system

Abstract

The invention provides a small sample video recognition method and system, and belongs to the field of sample video behavior recognition, and the method comprises the steps: S1, carrying out the preprocessing of a video, and obtaining a support set and a query set; s2, constructing a small sample video behavior recognition model which comprises a backbone network, a self-supervised motion learning module and a contrast learning time modeling module; s3, inputting the support set and the query set into a backbone network to obtain a support feature set and a query feature set; s4, inputting the support feature and query feature set into a self-supervised motion learning module for processing to obtain a support and query motion dynamic feature set; s5, capturing global time context information from appearance and motion dynamic angles by using a contrast learning time modeling module; and finally, performing a classification task on the selected frame by using a model obtained by metric learning on the enhanced support feature set and the query feature set. According to the method, video identification under a small sample is realized, and the method is closer to application in a real identification scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of small-sample video behavior recognition, and particularly relates to a small-sample video recognition method and system. Background Art

[0002] Action recognition, as a key sub-task in the field of video understanding, aims to accurately classify video sequences containing human actions. Compared with tasks in the image field, behavior recognition in the video field faces greater challenges because human actions exhibit dynamic changes in the time dimension. In recent years, with the increasing improvement of large-scale datasets in the video field and the rapid development of deep learning technology, significant progress has been made in behavior recognition research. However, traditional deep learning-based behavior recognition methods rely on a large amount of labeled video data as the basic support, which leads to high labor costs. In addition, the generalization ability of current traditional deep learning algorithms is limited, and even if the datasets are similar, it is difficult for pre-trained models to be effectively transferred to other dataset tasks.

[0003] In practical industrial applications, in specific fields such as national defense and military, medical diagnosis and treatment, etc., it is difficult to obtain a sufficient number of labeled video data due to reasons such as data sensitivity or high acquisition difficulty. Directly applying deep learning methods to train on a small-scale dataset is extremely likely to cause the problem of model overfitting. Although some researchers have tried to expand the dataset through simple data augmentation techniques such as rotation and cropping, experimental results show that the effects of these methods are still not satisfactory. Although the neural network model adopted by deep learning simulates the working mechanism of human brain neurons, there are still significant differences compared with human learning ability. Humans can quickly and accurately recognize new object categories through a small number of samples, or even a single image, without relying on large-scale training data. Driven by the dual needs of reducing the dependence on large-scale datasets and pursuing human-brain-like learning ability, the research on behavior recognition of small-sample data has gradually become an emerging hot spot in this field, providing new ideas and directions for solving existing problems.

[0004] Small-sample video behavior recognition, as an important research direction in the field of computer vision, its theoretical basis comes from small-sample learning. Small-sample learning is usually formalized as an N-way, K-shot learning task framework. Among them, "way" represents the number of categories sampled from the training data, and "shot" represents the total number of samples contained in each category.

[0005] Taking the small-sample image classification task as an example, when setting the number of sampled categories N = 4 and the total number of samples K = 1, its training sampling process is as follows: First, randomly select N categories from all the categories in the dataset as the training categories, such as Figure 1As shown, the extracted categories include Category 1, Category 2, Category 3, and Category 4, corresponding to the samples with differently colored squares on the left side of the figure. Next, among the total samples of the selected categories, K samples are randomly selected. These samples together constitute the support set required for training, and the total number of samples in this support set is K×N. Finally, X samples (X = 1 in the figure, corresponding to the sample with a "?" on the left side) are selected from the remaining unselected samples in the selected training categories to form the query set. In this system, all samples in the support set carry label information, while the samples in the query set are in an unlabeled state. The core task of few-shot learning is to measure the similarity between the query set samples and the support set samples, and then achieve accurate classification of the query set samples. Each such sampling process is called an "episode".

[0006] Transferring to the few-shot video action recognition task, its goal is to classify unlabeled query videos into one of the action categories with a limited number of samples in the support set. Since a video consists of a series of temporally ordered video frames, accurate recognition of human behaviors in videos requires comprehensive utilization of context video frame information for in-depth understanding, which makes few-shot video action recognition a highly challenging task. Action recognition, as a key branch in the field of video understanding, demonstrates important application value in many real-world scenarios. However, traditional deep learning-based action recognition methods highly rely on large-scale data, and the data transfer ability of the model is limited, severely restricting the expansion of its application scenarios. In contrast, the research on few-shot video action recognition methods can better adapt to the actual situation of scarce data and is expected to significantly improve the recognition accuracy. Therefore, in-depth research on few-shot-based action recognition methods not only has important theoretical significance but also has non-negligible practical value for promoting the application of computer vision technology in actual scenarios.

[0007] Currently, the research methods for this task can be mainly classified into two major categories: generation-based and metric learning-based:

[0008] The core idea of generation-based methods is to expand the sample size of the dataset through data generation techniques. Specifically, this method uses models such as generative adversarial networks (GANs) or variational autoencoders (VAEs) to generate new virtual samples based on the distribution characteristics of the existing small number of samples, thereby increasing the diversity and quantity of training data. The essential purpose of this method is to provide more learning information for the model by enriching the training data, and then improve the accuracy of action recognition under few-shot conditions. For example, Mishra et al. proposed an innovative zero-shot and few-shot video action recognition method, which is based on generative adversarial networks (GANs) to generate new action samples to address the challenges brought by data scarcity to training, and can help the model better learn action patterns and reduce overfitting problems caused by insufficient samples.

[0009] The method based on metric learning constructs an effective feature representation space by borrowing the concept of meta - learning. This method first extracts and transforms the features of the samples in the support set and the query set, so that similar behavior samples are close in the feature vector space, while samples of different behaviors are far apart. Then, by measuring the distance (such as Euclidean distance, cosine distance, etc.) between the query set samples and the support set samples in this space, the behavior category to which the query sample belongs is determined. Currently, metric learning has become the mainstream idea in the field of few - shot video behavior recognition, and many advanced methods are designed based on this. Taking the HyRSM++ method as an example, it is an improvement based on HyRSM (a basic architecture). HyRSM++ introduces a hybrid relationship module and a time - set matching metric strategy. The former can make full use of the relevant relationships in the episodic tasks and across videos, so as to learn task - specific feature representations. The latter measures the distance between the query set and the support set videos from the perspective of set matching. After enhancing the features through these two modules, HyRSM++ further enhances the distinguishability between different behavior features, thus achieving excellent performance in the few - shot video behavior recognition task and reaching the current optimal level.

[0010] Although the above - mentioned methods have achieved great success, in the field of few - shot video behavior recognition, previous studies generally have the problem of insufficient mining of key motion dynamics and temporal context information. When traditional algorithms process videos, they cannot accurately analyze the dynamic changes of actions over time, nor can they effectively utilize the global temporal context information, resulting in difficult improvement of recognition performance, poor accuracy and robustness in the few - shot case. Summary of the Invention

[0011] To solve the above - mentioned technical problems, the present invention provides a few - shot video recognition method, which includes the following steps:

[0012] Step S1: Pre - process the video to obtain a support set with class labels and a query set without class labels. Both the support set and the query set are composed of video frames with time series;

[0013] Step S2: Construct a few - shot video behavior recognition model. The few - shot video behavior recognition model includes a backbone network, a self - supervised motion learning module, and a contrastive learning temporal modeling module. Among them, the self - supervised motion learning module includes: a motion generator and a motion learner; the contrastive learning temporal modeling module contains two branches: an appearance branch and a motion branch, and each branch contains a temporal context modeling module;

[0014] Step S3: Input the support set and the query set into the few - shot video behavior recognition model. After being processed by the backbone network, a support feature set and a query feature set are obtained;

[0015] Step S4: Input the support feature set and the query feature set into the self-supervised motion learning module for processing to obtain a support motion dynamic feature set and a query motion dynamic feature set;

[0016] Step S5: Input the support feature set and the query feature set into the appearance branch to perform temporal dimension modeling and aggregate temporal context information; input the support motion dynamic feature set and the query motion dynamic feature set into the motion branch to perform temporal dimension modeling and aggregate temporal context information; then use a contrastive learning function to further promote samples of the same class to have similar temporal context information from the perspectives of the original video and motion dynamics; use a video frame-level metric function and a query set classifier for classification to obtain the final classification result of the query set.

[0017] Beneficial effects:

[0018] 1. The present invention designs a self-supervised motion learning module, including a motion generator and a motion learner, to extract video motion information in a self-supervised learning manner. This module has significant computational efficiency and reduces the memory requirement for pre-computed optical flow.

[0019] 2. The present invention designs a contrastive learning temporal modeling module, which enriches the temporal representation, integrates the global temporal context through two new contrastive losses, and at the same time encourages samples of the same class to have similar temporal context information from the perspectives of the original video and motion dynamics, improving the accuracy of discrimination and matching.

[0020] 3. The present invention conducts comprehensive experiments on three challenging video behavior datasets. The few-shot video behavior recognition method proposed by the present invention always outperforms existing methods and achieves state-of-the-art performance under various few-shot experimental settings. Description of the Drawings

[0021] Figure 1 Schematic diagram of the few-shot method in the existing method;

[0022] Figure 2 Schematic diagram of the process of a few-shot video recognition method of the present invention;

[0023] Figure 3 Schematic diagram of the structure of a few-shot video behavior recognition model;

[0024] Figure 4 Schematic diagram of the process of an optical flow extractor;

[0025] Figure 5 Block diagram of the structure of a few-shot video recognition system of the present invention. Detailed Embodiments

[0026] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0027] Embodiment 1

[0028] As Figure 2 shown, a small-sample video recognition method provided by an embodiment of the present invention includes the following steps:

[0029] Step S1: Preprocess the video to obtain a support set with class labels and a query set without class labels. Both the support set and the query set are composed of video frames with time series.

[0030] Step S2: Build a small-sample video behavior recognition model. The small-sample video behavior recognition model includes a backbone network, a self-supervised motion learning module, and a contrastive learning time modeling module. Among them, the self-supervised motion learning module includes: a motion generator and a motion learner; the contrastive learning time modeling module contains two branches: an appearance branch and a motion branch, and each branch contains a time context modeling module.

[0031] Step S3: Input the support set and the query set into the small-sample video behavior recognition model. After being processed by the backbone network, a support feature set and a query feature set are obtained.

[0032] Step S4: Input the support feature set and the query feature set into the self-supervised motion learning module for processing to obtain a support motion dynamic feature set and a query motion dynamic feature set.

[0033] Step S5: Input the support feature set and the query feature set into the appearance branch to perform time dimension modeling and aggregate time context information; input the support motion dynamic feature set and the query motion dynamic feature set into the motion branch to perform time dimension modeling and aggregate time context information; then use the contrastive learning function to further promote the samples of the same class to have similar time context information from the perspectives of the original video and motion dynamics; use the video frame-level metric function and the query set classifier for classification to obtain the final classification result of the query set.

[0034] In one embodiment, the above step S1: Preprocess the video to obtain a support set with class labels and a query set without class labels. Both the support set and the query set are composed of video frames with time series, specifically including:

[0035] Support set consists of It consists of video samples of different categories; the query set contains a video sample to be classified ; each video sample and is represented as a tensor of dimension , where , and represent the number of sampled frames, the original video height, and the original video width used to construct the video feature representation, respectively.

[0036] In an embodiment of the present invention, the N-way 1-shot experiment setting is selected as an example. The support set consists of video samples of different categories, while the query set contains 1 video sample to be classified. The core objective of few-shot video action recognition is to accurately classify the video sample to be classified into one of the categories in the support set.

[0037] In one embodiment, the above step S2: constructing a few-shot video action recognition model. The few-shot video action recognition model includes a backbone network, a self-supervised motion learning module, and a contrastive learning temporal modeling module. Among them, the self-supervised motion learning module includes: a motion generator and a motion learner; the contrastive learning temporal modeling module contains two branches: an appearance branch and a motion branch, and each branch contains a temporal context modeling module;

[0038] As Figure 3 shown, it is a schematic structural diagram of the few-shot video action recognition model.

[0039] In one embodiment, the above step S3: constructing a backbone network based on Resnet-50, which is used to simultaneously extract and process the spatial features and temporal features of the video to obtain a support feature set and a query feature set , where , , C, h, and w represent the number of channels, the feature height, and the width respectively, and specifically include:

[0040] The few-shot video action recognition model of the present invention first uses the backbone network constructed based on Resnet-50 to encode the video frame sequences in the support set and the query set; that is, the dynamic image sequences of the support set and the query set videos are used as the input of the backbone network Resnet-50, and the support feature set and the query feature set are output.

[0041] In one embodiment, the above-mentioned step S4: input the support feature set and the query feature set into the self-supervised motion learning module for processing to obtain the support motion dynamic feature set and the query motion dynamic feature set, specifically including:

[0042] Step S41: Input and into the motion learner of the self-supervised motion learning module. First, through spatial 3D convolution and temporal 3D convolution, the spatio-temporal local information of the support feature set and the query feature set output by the backbone network Resnet-50 is enhanced. Among them, spatial 3D convolution can capture the detailed information in the spatial dimension of the video, such as the changes in the actions of people at different positions; temporal 3D convolution focuses on mining the dynamic information in the temporal dimension, such as the sequence and rhythm changes of actions; through these two convolution operations, the feature set can contain richer spatio-temporal information, laying a solid foundation for subsequent processing; then, through an optical flow extractor, a three-channel feature map is obtained, where two channels represent the displacement map of the corresponding feature points between two adjacent frames , and the other channel represents which represents the confidence map corresponding to the displacement map. Connect and to obtain a three-channel per-pixel optical flow map; pass the three-channel per-pixel optical flow map through 1 spatial 2D convolution to convert it into motion feature information with the same number of channels C, and then through 7 channel-upgraded 2D convolutions to obtain the motion dynamics feature , ;

[0043] In the embodiment of the present invention, 1 spatial 2D convolution and 7 channel-upgraded 2D convolutions adopt depthwise separable convolutions. Compared with two-dimensional convolutions, depthwise separable convolutions have significantly improved computational efficiency, making them an important tool for optimizing performance. To enhance the generalization ability of the self-supervised motion learning module, in each depthwise separable convolution layer, the embodiment of the present invention adopts Batch Normalization and ReLU activation functions. Therefore, the motion dynamics feature is obtained. The process of motion information reconstruction aims to interpret the semantics of displacement and confidence through convolutional layers, so as to learn the motion feature information of specific tasks.

[0044] Such as Figure 4As shown in the figure, a schematic diagram of the optical flow extractor process is presented. The spatio-temporal local information of the support feature set and the query feature set after being enhanced by two 3D convolutions is input into the optical flow extractor to obtain the corresponding optical flow field and its related confidence map. The optical flow field can visually display the movement direction and speed of objects in the video, while the confidence map reflects the reliability of the optical flow information. The optical flow information undergoes spatial relationship modeling and channel upsampling operations. Spatial relationship modeling aims to deeply analyze the spatial interrelationships of the optical flow information, highlighting the motion semantic information related to specific tasks. For example, in the action recognition task, it focuses on highlighting the motion parts related to the target action; channel upsampling can, to a certain extent, suppress irrelevant noise, making the optical flow information more pure and accurate.

[0045] Step S42: Input and into the motion generator of the self-supervised motion learning module. The process is similar to step S31, except that it does not go through spatial 3D convolution and temporal 3D convolution, and directly obtains the low-level displacement map and the low-level confidence map through 1 optical flow extractor. The connected low-level three-channel per-pixel optical flow map is passed through 1 spatial 2D convolution and 7 channel-upgraded 2D convolutions to obtain the low-level motion dynamic feature , ;

[0046] Step S43: Construct the self-supervised loss function , and :

[0047] ;

[0048] ;

[0049] .

[0050] Different from the above motion learner, the motion generator mainly supervises the optical flow information modeling of the motion learner at the video optical flow feature level and the video semantic level. At the video optical flow feature level, the motion generator compares the optical flow features extracted by the motion learner with the ideal feature pattern to guide the optimization of the feature extraction process; at the video semantic level, the motion generator starts from a higher-level semantic perspective to ensure that the information extracted by the motion learner conforms to the semantic logic of the behavior. Through this self-supervised mechanism, the motion learner can more robustly learn high-level motion semantic information, and while maintaining a lightweight design, effectively suppress various irrelevant noise interferences, improving the performance and efficiency of the model.

[0051] In the field of small-sample video behavior recognition, many existing research methods have certain limitations. Some methods rely too much on the attention mechanism for spatial and temporal alignment, which makes the model lack the ability to capture motion information. Because the attention mechanism mainly focuses on the attention to key regions or moments, it is difficult to comprehensively and deeply explore the rich motion details between consecutive frames in the video, thus affecting the accuracy and effectiveness of behavior recognition. In view of this, the present invention innovatively proposes a brand-new method for extracting motion features between frames - the self-supervised motion learning module. The core goal of this module is to effectively and efficiently model motion dynamics, so as to provide stronger support for small-sample video behavior recognition.

[0052] The self-supervised motion learning module of the present invention mainly consists of two major parts: a motion generator and a motion learner. The motion generator guides the motion learner in a unique self-supervised manner, enabling the two to jointly optimize operations from the time-series data of the query set and the support set, and accurately extract task-specific motion dynamic information. This joint optimization mode can make full use of the motion information in limited samples and avoid information omission and misjudgment. Specifically speaking, the self-supervised motion learning module significantly enhances the integration of optical flow motion information in the process of small-sample video behavior recognition through three key steps. The first step is the calculation of motion optical flow features. Through a series of previous operations, the optical flow features containing rich motion information are accurately calculated. The second step is the reconstruction of motion information. The calculated optical flow features are integrated and optimized to better reflect the real motion situation in the video. The third step is the self-supervised guidance of motion information. Using the supervision of the motion generator, the motion learner is guided to continuously adjust and improve the learning and utilization of motion information, thereby enhancing the overall effect of small-sample video behavior recognition.

[0053] The self-supervised motion learning module shows certain advantages in extracting motion dynamic information, and can effectively capture the motion changes between video frames, providing an important feature basis for behavior recognition. However, for complex and variable video content, relying solely on motion dynamic information is often not enough to comprehensively understand the behavior patterns in the video. Because the behaviors in the video not only include the motion of objects or people, but also involve the changes in appearance features in the time dimension and the context connections between different behaviors.

[0054] In order to further improve the model's ability to model temporal context information and thus more accurately recognize behaviors in small-sample videos, the present invention innovatively proposes a contrastive learning temporal modeling module. The design concept of this module is to fully integrate appearance information and motion dynamic information to achieve more powerful temporal context modeling and more accurate video frame-level matching results.

[0055] Appearance information encompasses static features such as the shape, color, and texture of objects or people in a video, which also play a crucial role in action recognition. For example, different clothing colors or the appearance forms of objects may be associated with specific actions. Motion dynamic information, on the other hand, reflects the action changes and movement trajectories of objects or people. Combining the two can more comprehensively describe the action scenarios in the video.

[0056] The contrastive learning time modeling module optimizes the model performance through contrastive learning. Specifically, it aims to minimize the visual similarity loss of videos in the same class, which means the model can better capture the common features of the same action class in different videos, even if these videos vary in shooting angles, lighting conditions, etc. At the same time, it maximizes the video visual similarity loss of videos in different classes, enabling the model to clearly distinguish the differences between different action classes and avoid confusion. In this way, the module achieves robust and effective global time context modeling, can grasp the information changes of the video in the time dimension as a whole, and can perform accurate matching at the video frame level to improve the accuracy of action recognition.

[0057] In one embodiment, the above step S5: Input the support feature set and the query feature set into the appearance branch for time dimension modeling and aggregating time context information; input the support motion dynamic feature set and the query motion dynamic feature set into the motion branch for time dimension modeling and aggregating time context information; then use the contrastive learning function to further promote the samples in the same class to have similar time context information from the perspectives of the original video and motion dynamics; use the video frame-level metric function and the query set classifier for classification to obtain the final classification result of the query set, specifically including:

[0058] Step S51: In the appearance branch, input and into the time context modeling module in the contrastive learning time modeling module. The spatial information is first aggregated by spatial average pooling operation, and then the channel information is mixed and enhanced by a multi-layer perceptron to obtain discriminative spatially aggregated features ;

[0059] Then, input after time compression operation , lightweight time context modeling DWConv, and time expansion operation to construct enhanced time features and time context features :

[0060] ;

[0061] ;

[0062] Among them, , represents layer normalization; represents a non-linear activation function; TGAP represents temporal global average pooling, is the temporal context feature aggregated from the enhanced temporal features ;

[0063] The present invention respectively constructs features with temporal context modeling efficiently through operations such as temporal compression lightweight temporal context modeling DWConv and temporal expansion and so on. Notably, compared with ordinary convolution, DWConv not only performs temporal context modeling along the temporal dimension in the latent space, but also significantly reduces the computational cost, improves the computational efficiency, and reduces the memory usage.

[0064] Step S52: In the motion branch, for the motion dynamic feature adopt the method of step S41 to obtain the enhanced motion temporal feature ;

[0065] Step S53: Adopt a contrastive learning method including positive sample pairs and negative sample pairs to promote frame-level matching, so as to comprehensively perceive the temporal context information. Therefore, two contrastive learning loss functions are constructed to improve the perception ability of frame features to the global temporal context, and their definitions are as follows:

[0066] ;

[0067] ;

[0068] Among them, is the concatenation operation along the temporal dimension;

[0069] and are respectively the appearance feature and the motion dynamic feature after being enhanced by the temporal context modeling module in the query feature set in the appearance branch and the query motion feature set in the motion branch; and are from and aggregated temporal context features;

[0070] and are in the training episode task, the appearance feature and the motion dynamic feature after being enhanced by the temporal context modeling module from the videos belonging to the same category as in the support set; and is the temporal context feature aggregated from and ;

[0071] and are the enhanced appearance feature and motion dynamic feature through the temporal context modeling module in the support set;

[0072] Since the aggregated temporal context feature of the original video appearance feature and motion information is crucial for achieving accurate frame-level matching. Therefore, the method proposed in the present invention adopts a contrastive learning method including positive sample pairs and negative sample pairs to promote frame-level matching, so as to comprehensively perceive the temporal context information. The model regards the same-category context temporal features in the motion branch and appearance branch of the support set and query set from the same video as positive sample pairs, while all other combinations form negative sample pairs.

[0073] is used to compare the connected temporally enhanced appearance feature and motion feature before and after temporal global average pooling (TGAP); while is used to compare the temporally enhanced appearance branch feature and motion branch feature before and after temporal global average pooling (TGAP). The objective of the above contrastive learning loss function is to amplify the cumulative similarity of all positive sample pairs while weakening the cumulative similarity of negative sample pairs, so as to explicitly promote the integration of local features into the overall global temporal context information.

[0074] Step S54: Classify the query video sample using the frame-level metric function and the query set category classifier;

[0075] The enhanced support feature set and the enhanced query feature set The metric distance between them is expressed as:

[0076] ;

[0077] where is a frame-level metric function;

[0078] Similarly, the distance between the motion feature of the support set and the motion feature of the query set is expressed as:

[0079] ;

[0080] The total distance between the support set video and the query set video is expressed as:

[0081] ;

[0082] Construct the overall combined loss function :

[0083] ;

[0084] ;

[0085] ;

[0086] wherein, is a balancing factor;

[0087] is the cross-entropy loss based on the true label for the total distance between the support set and the query set ; c represents any class label in the support set;

[0088] is the cross-entropy loss of the query set class classifier; is a hyperparameter used to balance the auxiliary loss ; is a learnable weight;

[0089] represents the 5 loss functions defined above, .

[0090] To verify the effectiveness of the present invention, under the 5-way, 1-shot experimental setting, Table 1 shows the experimental results on three datasets, namely HMDB51, SSV2-Full, and Kinetics, as follows:

[0091] Table 1 Comparison of experimental results of the method of the present invention in the 5-way, 1-shot case

[0092] Under the 5-way, 5-shot experimental setting, the experimental results of the method of the present invention on three datasets, namely HMDB51, SSV2-Full, and Kinetics, are as follows:

[0093] Table 2 Comparison of experimental results of the method of the present invention in the 5-way, 5-shot case

[0094] It can be seen from Table 1 and Table 2 that the experimental results of the method of the present invention on each dataset are significantly better than the existing methods.

[0095] Example 2

[0096] As Figure 5 shown, the embodiment of the present invention provides a few-shot video recognition system, including the following modules:

[0097] A preprocessing module 61, configured to preprocess a video to obtain a support set with class labels and a query set without class labels, where both the support set and the query set are composed of video frames with time series;

[0098] A small-sample video behavior recognition model construction module 62, configured to construct a small-sample video behavior recognition model. The small-sample video behavior recognition model includes a backbone network, a self-supervised motion learning module, and a contrastive learning temporal modeling module. Among them, the self-supervised motion learning module includes a motion generator and a motion learner; the contrastive learning temporal modeling module contains two branches: an appearance branch and a motion branch, and each branch contains a temporal context modeling module;

[0099] A support and query feature set construction module 63, configured to input the support set and the query set into the small-sample video behavior recognition model, and after being processed by the backbone network, obtain a support feature set and a query feature set;

[0100] A self-supervised motion learning module 64, configured to input the support feature set and the query feature set into the self-supervised motion learning module for processing to obtain a support motion dynamic feature set and a query motion dynamic feature set;

[0101] A contrastive learning temporal modeling module 65, configured to input the support feature set and the query feature set into the appearance branch to perform temporal dimension modeling and aggregate temporal context information; input the support motion dynamic feature set and the query motion dynamic feature set into the motion branch to perform temporal dimension modeling and aggregate temporal context information; then use a contrastive learning function to further promote samples of the same class to have similar temporal context information from the perspectives of the original video and motion dynamics; use a video frame-level metric function and a query set classifier for classification to obtain the final classification result of the query set.

[0102] A small-sample video recognition device includes one or more electronic devices, and one or more of the electronic devices are configured to implement the small-sample video recognition method, system, and device.

[0103] An electronic device includes: one or more processors; a memory, configured to store one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors implement the small-sample video recognition method, system, and device.

[0104] A computer-readable storage medium stores executable instructions thereon, and when the instructions are executed by a processor, the processor implements the small-sample video recognition method, system, and device.

[0105] A non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor implements a small-sample video recognition method, system, and apparatus.

[0106] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A small-sample video recognition method, characterized in that, Including: Step S1: Preprocess the video to obtain a support set with class labels and a query set without class labels. Both the support set and the query set are composed of video frames with time series. Step S2: Build a few-shot video action recognition model. The few-shot video action recognition model includes a backbone network, a self-supervised motion learning module, and a contrastive learning temporal modeling module. Among them, the self-supervised motion learning module includes a motion generator and a motion learner; the contrastive learning temporal modeling module contains two branches: an appearance branch and a motion branch, and each branch contains a temporal context modeling module. Step S3: Input the support set and the query set into the few-shot video action recognition model. After being processed by the backbone network, a support feature set and a query feature set are obtained. Step S4: Input the support feature set and the query feature set into the self-supervised motion learning module for processing to obtain a support motion dynamic feature set and a query motion dynamic feature set. Step S5: Input the support feature set and the query feature set into the appearance branch to perform temporal dimension modeling and aggregate temporal context information; input the support motion dynamic feature set and the query motion dynamic feature set into the motion branch to perform temporal dimension modeling and aggregate temporal context information; then use a contrastive learning function to further promote samples of the same class to have similar temporal context information from the perspectives of the original video and motion dynamics; use a video frame-level metric function and a query set classifier for classification to obtain the final classification result of the query set.

2. The small-sample video recognition method according to claim 1, characterized in that The specific content of Step S1: Preprocess the video to obtain a support set with class labels and a query set without class labels. Both the support set and the query set are composed of video frames with time series, includes: Support Set Depend on The query set consists of video samples of different categories; Contains a video sample to be classified ; Each video sample and It is represented as a dimension A tensor of 、 and They represent the number of sample frames, original video height, and original video width used to construct video feature representation.

3. The small sample video recognition method according to claim 2, characterized in that: The specific content of Step S3: Input the support set and the query set into the few-shot video action recognition model. After being processed by the backbone network, a support feature set and a query feature set are obtained, includes: Build a backbone network based on Resnet-50 to simultaneously extract and process the spatial and temporal features of the video, and obtain a support feature set and a query feature set , where , , where C, h, and w represent the number of channels, feature height, and width respectively.

4. The small-sample video recognition method according to claim 3, wherein The specific content of Step S4: Input the support feature set and the query feature set into the self-supervised motion learning module for processing to obtain a support motion dynamic feature set and a query motion dynamic feature set, includes: Step S41: Take and as the input to the motion learner of the self-supervised motion learning module. First, perform 3D spatial convolution and 3D temporal convolution, and then pass through an optical flow extractor to obtain a three-channel feature map, where two channels represent the displacement map of corresponding feature points between two adjacent frames , and the other channel represents the confidence map corresponding to the displacement map. Connect and to obtain a three-channel per-pixel optical flow map; pass the three-channel per-pixel optical flow map through 1 2D spatial convolution to convert it into motion feature information with the same number of channels C, and then pass through 7 2D convolutional layers for channel boosting to obtain motion dynamic features , ; Step S42: Input and into the motion generator of the self-supervised motion learning module. The process is similar to that of Step S31, except that it does not go through the spatial 3D convolution and the temporal 3D convolution. Instead, it directly obtains the low-level displacement map and the low-level confidence map . The concatenated low-level three-channel per-pixel optical flow map is passed through one spatial 2D convolution and seven channel-upgraded 2D convolutions to obtain the low-level motion dynamic feature , ; Step S43: Construct a self-supervised loss function , and : ； ； 。 5. The small sample video recognition method according to claim 4, characterized in that: The specific content of Step S5: Input the support feature set and the query feature set into the appearance branch to perform temporal dimension modeling and aggregate temporal context information; input the support motion dynamic feature set and the query motion dynamic feature set into the motion branch to perform temporal dimension modeling and aggregate temporal context information; then use a contrastive learning function to further promote samples of the same class to have similar temporal context information from the perspectives of the original video and motion dynamics; The specific content of using a video frame-level metric function and a query set classifier for classification to obtain the final classification result of the query set, includes: Step S51: In the appearance branch, and are input into the time context modeling module in the contrastive learning time modeling module. First, the spatial information is aggregated by spatial average pooling operation, and then the channel information is mixed and enhanced by a multi-layer perceptron to obtain discriminative spatially aggregated features ; Then, perform a time compression operation , lightweight time context modeling DWConv, and time expansion operation to construct enhanced temporal features and temporal context features : ； ； Among them, , represents layer normalization; represents a non-linear activation function; TGAP represents temporal global average pooling, is the temporal context feature aggregated from the enhanced temporal features obtained. Step S52: In the motion branch, for the motion dynamic features Adopt the method in Step S41 to obtain the enhanced motion time features ; Step S53: Adopt a contrastive learning method including positive sample pairs and negative sample pairs to promote frame-level matching, so as to comprehensively perceive temporal context information. Therefore, two contrastive learning loss functions are constructed to improve the perception ability of frame features for the global temporal context, and their definitions are as follows: ； ； Among them, is a splicing operation along the time dimension; and are the appearance features and motion dynamic features enhanced by the time context modeling module from the query feature set in the appearance branch and the query motion feature set in the motion branch, respectively; and are the time context features aggregated from and ; and are the appearance features and motion dynamic features enhanced by the time context modeling module from videos belonging to the same category in the support set during the training episode task; and are the time context features aggregated from and ;​ and are the enhanced appearance features and motion dynamic features supported by the centralized time context modeling module; Step S54: Classify the query video sample using the frame-level metric function and the query set class classifier; Enhanced support feature set and the enhanced query feature set The metric distance between them is expressed as: ； Among them, is a frame-level metric function; Similarly, the motion feature of the support set and the motion feature of the query set are represented as the distance between them: ； The total distance between the support set video and the query set video is expressed as: ； Construct the overall combined loss function : ； ； ； wherein, is a balance factor; is the cross-entropy loss based on the true labels for the total distance between the support set and the query set where c represents any class label in the support set; is the cross - entropy loss of the query set category classifier; is a hyperparameter used to balance the auxiliary loss ; is a learnable weight; represent the five loss functions defined above, .

6. A small-sample video recognition system, characterized in that, including the following modules: A preprocessing module for preprocessing the video to obtain a support set with class labels and a query set without class labels, both the support set and the query set are composed of video frames with time series; A module for constructing a few-shot video behavior recognition model for constructing a few-shot video behavior recognition model, the few-shot video behavior recognition model includes a backbone network, a self-supervised motion learning module and a contrastive learning temporal modeling module, wherein, the self-supervised motion learning module includes: a motion generator and a motion learner; the contrastive learning temporal modeling module contains two branches: an appearance branch and a motion branch, and each branch contains a temporal context modeling module; A module for constructing support and query feature sets for inputting the support set and the query set into the few-shot video behavior recognition model, and obtaining a support feature set and a query feature set after being processed by the backbone network; A self-supervised motion learning module for inputting the support feature set and the query feature set into the self-supervised motion learning module for processing to obtain a support motion dynamic feature set and a query motion dynamic feature set; A contrastive learning temporal modeling module for inputting the support feature set and the query feature set into the appearance branch to perform temporal dimension modeling and aggregate temporal context information; inputting the support motion dynamic feature set and the query motion dynamic feature set into the motion branch to perform temporal dimension modeling and aggregate temporal context information; then using the contrastive learning function to further promote the samples of the same class to have similar temporal context information from the perspectives of the original video and motion dynamics; using the video frame-level metric function and the query set classifier for classification to obtain the final classification result of the query set.

7. A small-sample video recognition device, characterized in that including one or more electronic devices, wherein the one or more electronic devices are used to implement the method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, including: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, Stored thereon are executable instructions, which when executed by a processor cause the processor to implement the method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium, characterized in that, Stored thereon is a computer program, which when executed by a processor implements the steps of the method according to any one of claims 1 to 5.

Images