A video action recognition method based on zero-shot learning

Abstract

The present invention discloses a video action recognition method based on zero-sample learning, and proposes a new instance-level video attribute as a semantic feature. The semantic feature utilizes the spatial information and temporal information of the video, and can provide a more accurate semantic description for the video, so as to better connect the visible category with the invisible category, and achieve a higher recognition accuracy rate on the invisible category. At the same time, in order to better optimize the visual-semantic joint space, the present invention designs a new contrast loss function-the visual-instance-level video attribute contrast loss function. The contrast loss function considers both the relationship between the visual representation and semantic representation of different video individuals, and the relationship between the visual representations of different categories of videos, so that the visual-semantic joint space has better alignment and uniformity. These two characteristics make the trained model perform better on invisible categories.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and in particular to a video action recognition method based on zero-sample learning. Background Art

[0002] The types of actions to be identified in massive video data are increasing, such as running, playing basketball, etc. At the same time, the same batch of video data may have action labels of different dimensions. Faced with such complex scenarios, it is costly to annotate a sufficient number of training samples. How to quickly and efficiently identify actions in videos to help with subsequent video review and user recommendations has become an urgent problem to be solved.

[0003] The success of deep learning relies on a large amount of training data, which makes the trained model lack the ability to generalize and transfer to unseen categories. The introduction of zero-shot learning effectively solves the problem of training unseen categories. Its goal is to classify data of unseen categories by learning the visual and semantic features of visible categories. Zero-shot learning enables machines to have the ability to transfer knowledge. It does not need to train samples of new categories, which can reduce the tedious manual labeling process and avoid retraining classifiers. It is more in line with the problem of massive category classification in real life.

[0004] The paradigm of zero-shot learning is to construct a joint space of visual features and semantic features. In the training phase, the visual features of the visible categories are aligned with the corresponding semantic features in the joint space; in the testing phase, for each visual feature to be tested, the nearest neighbor search (NNS) algorithm is used to find the semantic feature that best matches it, so as to obtain the category to which it belongs.

[0005] The core problem in zero-shot learning is how to construct appropriate semantic features to achieve the migration process from visible categories to invisible categories. In the field of zero-shot action recognition, common semantic features can be divided into the following three categories:

[0006] (1) Manually labeled attributes. The labeler defines a set of attributes in advance. If a certain type of video has the corresponding attribute, the corresponding attribute is marked as 1, otherwise it is marked as 0. For the same type of video, its manually labeled attributes are discrete vectors composed of 0 and 1. Although manually labeled attributes can better connect visible categories and invisible categories, it has the following two disadvantages: a) The attributes are manually labeled and may have strong subjective intentions, and cannot accurately express the semantic information of the video; b) For different data, there may be different attribute sets, that is, the manual attributes of different data may be different, and cannot be directly migrated to any data for use, and the scalability is poor.

[0007] (2) Word embedding vector. Considering the two shortcomings of manually annotated attributes, researchers began to think about a simpler and more effective method - encoders in natural language processing. For each action name, a pre-trained encoder such as Word2Vec, Glove, and BERT can be used to obtain its corresponding word vector. Benefiting from the huge pre-trained corpus, the word vector can be directly used as a semantic attribute to connect the visible category with the invisible category. Although the word embedding vector is simple and easy to use, it cannot accurately measure the relationship between different actions, and does not utilize the information of the video itself, and its performance on the invisible category is limited.

[0008] (3) Object attributes in the video. Considering the shortcomings of word embedding vectors, researchers considered using the information of the video itself - the objects in the video as semantic features. Specifically, frames of a fixed length are sampled from the video, and each frame is sent to a target detector or object classifier to obtain the object with the highest probability of appearing. This object information can be encoded to obtain semantic features, thereby connecting the visible category with the invisible category. This type of feature is currently widely used and has good performance on the invisible category.

[0009] Although object attributes in videos have achieved good performance, they only utilize the spatial information in the video (i.e., the object attributes in each frame), ignore the temporal information of the video (i.e., the action relationship of objects between different frames), and cannot provide accurate semantic features for the video. Summary of the invention

[0010] In view of the shortcomings of object attributes in the existing video technology, the present invention provides a video action recognition method based on zero-sample learning, and proposes a new video attribute as a semantic feature. This attribute considers both the spatial information (objects, scenes) and temporal information (motion, interaction) of the video, and can more accurately describe the semantic features of the video, thereby better connecting the visible category with the invisible category, and having a higher recognition accuracy on the invisible category. At the same time, a new contrast loss function is designed based on the properties of the semantic feature. This loss function makes the visual-semantic joint space have better alignment and uniformity, so that the trained model has a higher recognition accuracy on the invisible category.

[0011] In order to achieve the above object, the present invention provides the following technical solutions:

[0012] A video action recognition method based on zero-sample learning includes the following steps:

[0013] S1. For visible category video data, visual features are extracted through a visual encoder; instance-level captured content is extracted through a video capturer, and the instance-level captured content includes the spatial information and temporal information of the video. The captured content features corresponding to the captured content are obtained through the encoder SBERT in natural language processing;

[0014] S2. For the category name and category description at the category level, the corresponding category name features and category description features are obtained through the encoder SBERT in natural language processing, and the captured content features, category name features and category description features are input into the relation distillation module. The relation distillation module generates visual-instance-level video attributes for each video by learning the co-occurrence relationship between the captured content features at the instance level, the category name features at the category level and the category description features;

[0015] S3, optimizing the visual-semantic joint space according to the visual-instance level video attribute contrast loss function until the model converges to obtain a trained zero-shot action recognition model;

[0016] S4. Evaluate the performance of the trained zero-shot action recognition model on the unseen category video data. There is no intersection between the visible category video data set and the unseen category video data set.

[0017] Furthermore, in step S1, the visible category video data is represented as Among them, x n Refers to the original video clip, y n ∈S refers to the category name corresponding to the video clip, S = {1, ..., K} refers to the category set of visible categories, there are K categories of videos in total, and N refers to the number of videos in the visible category.

[0018] Furthermore, in step S1, for each video data x n , the output of the video capture device is Among them, ts l Indicates the start time of the event, te l Indicates the end time of the event, c l Represents the description statement of the event, L represents the video data x n , the video capturer outputs description statements of L events.

[0019] Furthermore, the captured content features corresponding to the L events are averaged by the encoder SBERT in natural language processing to obtain the captured content features corresponding to each video captured content, as shown in formula (2):

[0020]

[0021] Among them, c lA descriptive statement that represents an event.

[0022] Furthermore, the generation process of the visual-instance level video attributes in step S2 is:

[0023] Get the category name y through the encoder SBERT in natural language processing n And the category description des n The corresponding features are shown in formula (3):

[0024]

[0025] Among them, h n is the class name feature, d n Describe characteristics for the categories;

[0026] will capture the content features c n , category name feature h n And the category description feature d n Input to the relation distillation module to obtain the instance-level video attribute a n , as shown in formula (4):

[0027] a n =MultiHead(c n ,h n ,h n )+MultiHead(c n , d n , d n ) (4)

[0028] Among them, MultiHead(Q, K, V) refers to the multi-head attention mechanism, which splices h self-attention modules to obtain information in different subspaces. The calculation process of the multi-head attention mechanism is shown in formula (5):

[0029]

[0030] Among them, the first formula represents the calculation process of the self-attention module, the second formula represents the calculation process of the multi-head attention mechanism, and the third formula represents the calculation process of each head in the multi-head attention. Q represents the query value (Query), K represents the key value (Key), and K T is the transpose of K, V represents value, W O , W i Q , W i K , W i V is a learnable parameter and concat is a concatenation operation.

[0031] Furthermore, in step S3, for the visual feature v of the video n , the visual representation f(v n ); for the instance-level video attribute a of the video n , and the semantic representation g(a n ); after passing through the two projectors, the visual features and instance-level video attributes are mapped into a joint visual-semantic space.

[0032] Furthermore, in the visual-semantic joint space, the visual representation and semantic representation of the same video are brought closer, while the visual representation and semantic representation of different videos are pulled further away; the visual representation of the same type of video data is brought closer, while the visual representation of different types of video data is pulled further away, as shown in formula (6):

[0033]

[0034] Among them, τ is the temperature coefficient, l refers to the category of each video data, f(v k ) represents the visual representation corresponding to the kth sample in the batch data, and N is the batch size.

[0035] Further, in step S4, the video data of the invisible category is represented as M refers to the number of videos in the invisible category, and the category set of the invisible category is There are T category videos in total.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The video action recognition method based on zero-sample learning provided by the present invention proposes a new instance-level video attribute as a semantic feature. The semantic feature utilizes the spatial information and temporal information of the video, and can provide a more accurate semantic description for the video, thereby better connecting the visible category with the invisible category, and achieving a higher recognition accuracy on the invisible category. Specifically, the co-occurrence relationship between the instance-level captured content and the category name and category description at the category level is learned through the relationship distillation module, thereby obtaining the instance-level video attribute. The instance-level video attribute includes both the personality of the video (derived from the video capture) and the characteristics of the category to which the video belongs (derived from the category name and category description).

[0038] At the same time, in order to better optimize the visual-semantic joint space, the present invention designs a new contrast loss function - the visual-instance level video attribute contrast loss function. This contrast loss function considers both the relationship between the visual representation and semantic representation of different video individuals and the relationship between the visual representations of different categories of videos, so that the visual-semantic joint space has better alignment and uniformity. These two characteristics make the trained model perform better on unseen categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0040] Figure 1 A flowchart of a video action recognition method based on zero-sample learning provided by an embodiment of the present invention.

[0041] Figure 2 A network structure diagram of a video action recognition method based on zero-sample learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to better understand the technical solution, the method of the present invention is described in detail below with reference to the accompanying drawings.

[0043] The video action recognition method based on zero-sample learning provided by the embodiment of the present invention has the following process: Figure 1 As shown, the network structure is as Figure 2 As shown, the specific steps include:

[0044] S1. For visible category video data, visual features are extracted through a visual encoder; instance-level captured content is extracted through a video capturer, and the instance-level captured content includes the spatial information and temporal information of the video. The captured content features corresponding to the captured content are obtained through the encoder SBERT in natural language processing;

[0045] S2. For the category name and category description at the category level, the corresponding category name features and category description features are obtained through the encoder SBERT in natural language processing, and the captured content features, category name features and category description features are input into the relation distillation module. The relation distillation module generates instance-level video attributes for each video by learning the co-occurrence relationship between the instance-level captured content features, the category name features and the category description features at the category level;

[0046] S3, optimizing the visual-semantic joint space according to the visual-instance level video attribute contrast loss function until the model converges to obtain a trained zero-shot action recognition model;

[0047] S4. Evaluate the performance of the trained zero-shot action recognition model on the unseen category video data. There is no intersection between the visible category video data set and the unseen category video data set.

[0048] The parameter settings are as follows:

[0049] During the training phase, only video data of visible categories can be used Among them, x n Refers to the original video clip, y n ∈S refers to the category name corresponding to the video clip, S = {1, ..., K} refers to the category set of visible categories, there are K categories of videos in total, and N refers to the number of videos in the visible category.

[0050] During the testing phase, the trained model will be used on video data of unseen categories. The test is conducted on , M refers to the number of videos in the invisible category, and the category set of the invisible category is There are T category videos in total.

[0051] It should be noted that the category set S of the visible category and the category set S of the invisible category There should be no intersection. Specifically, the word vector distance between the visible category and the invisible category needs to be greater than a certain distance, as shown in formula (1).

[0052]

[0053] Where θ is the distance threshold.

[0054] In this scheme, the original visible category dataset uses Kinetics700, which contains 700 action categories and a total of 536,469 video data. The invisible category dataset uses UCF101 and HMDB51. For UCF101, it contains 101 action categories and a total of 13,320 video data; for HMDB51, it contains 51 action categories and a total of 6,767 video data. In order to ensure that the category set S of the visible category is consistent with the category set S of the invisible category There is no intersection. In this embodiment, the distance threshold θ is set to 0.05, and the cleaned visible category dataset Kinetics662 is obtained, which contains 662 action categories and a total of 501614 video data.

[0055] The overall process steps of the embodiment of the present invention are as follows:

[0056] S1. For visible category video data Input it into the visual encoder to get the visual feature v n , input it into the video capturer to obtain the instance-level captured content, and input it into the encoder SBERT in natural language processing to obtain the corresponding captured content feature c n .

[0057] S2, the category name y n And the category description n Input to the encoder SBERT in natural language processing to obtain the corresponding category name feature h n And the category description feature d n , c n 、h n and n Input to the relation distillation module to obtain the instance-level video attribute a n .

[0058] The instance-level video attribute generation process is:

[0059] For each video data (x n ,y n ), and input it into the video capturer to obtain the captured content at the instance level. Specifically, for each video data x n , the output of the video capture device is Among them, ts l Indicates the start time of the event, te l Indicates the end time of the event, c l Represents the description statement of the event, L represents the video data x n , the video capturer will output description statements of L events. The description statement output by the video capture contains both the spatial information of the video (objects, scenes) and the temporal information of the video (movement, interaction). For example, for a video of the "push-ups" category, the description statement output by the video capture is "a boy kneeling on the floor, his arms moving up and down", which contains both spatial information (boy, on the floor) and temporal information (kneeling, arms moving up and down), and can provide more accurate semantic features for the video. In order to comprehensively consider the information of L events, this embodiment averages the features corresponding to the L events to obtain the features corresponding to each video capture content, as shown in formula (2).

[0060]

[0061] However, due to the limitations of the video capturer, there may be some noise in the description sentences output by the video capture, which will affect the performance on unseen categories. For example, for videos in the "makeup" category, the description sentences output by the video capture are prone to "there is a person sitting in front of the camera". These broad sentences have limited relationship with the characteristics of the category itself, making the difference between different categories smaller.

[0062] Therefore, in order to generate semantic features that retain both the personality of the video (capture content) and the category characteristics, this scheme designs a relation distillation module, which learns the description sentence at the instance level and the category name y at the category level. n And the category description des n The co-occurrence relationship between them is used to generate instance-level video attributes for each video.

[0063] First, get the category name y through SBERT n And the category description des n The corresponding features are shown in formula (3).

[0064]

[0065] c n 、h n and n Input to the relation distillation module to obtain the instance-level video attribute a n , as shown in formula (4).

[0066] a n =MultiHead(c n ,h n ,h n )+MultiHead(c n , d n , d n ) (4)

[0067] Among them, MultiHead(Q, K, V) refers to the multi-head attention mechanism, which splices h self-attention modules to obtain information in different subspaces. The calculation process of the multi-head attention mechanism is shown in formula (5).

[0068]

[0069] Among them, Q represents query value (Query), K represents key value (Key), V represents value (Value), and W represents O , W i Q , W i K , W i Vis a learnable parameter and concat is a concatenation operation.

[0070] Here we get a n It contains both the spatiotemporal information in the video and the characteristics of the category itself, which can provide more accurate semantic features for the video and better connect the visible category with the invisible category.

[0071] S3, optimize the visual (v) based on the visual-instance level video attribute contrast loss function n )-semantics(a n ) joint space to obtain a trained zero-shot action recognition model.

[0072] In order to optimize the visual-semantic joint space, this paper designs a new contrast loss function based on the properties of the proposed instance-level video attributes. n , the visual representation f(v n ); for the instance-level video attribute a of the video n , and the semantic representation g(a n ). After passing through the two projectors, the visual features and instance-level video attributes are mapped into the visual-semantic joint space. In the joint space, the visual representation and semantic representation of the same video are brought closer, and the visual representation and semantic representation of different videos are pulled farther apart. At the same time, in order to ensure that the joint space has better alignment and uniformity, the present invention considers the relationship between visual representations of different categories. Specifically, in the joint space, the visual representation of the same type of video data is brought closer, and the visual representation of different types of video data is pulled farther apart, as shown in formula (6):

[0073]

[0074] Where τ is the temperature coefficient and l refers to the category of each video data. k ) represents the visual representation corresponding to the kth sample in the batch data, and N represents the batch size.

[0075] The contrast loss function first appeared in self-supervised learning and is often used to learn discriminative representations that can be transferred to downstream tasks. In self-supervised learning, the input model is a positive pair and a negative pair. The positive pair is brought closer in space and the negative pair is moved further away. The core work is how to construct positive and negative pairs from the data set. The representative work SimCLR constructs positive pairs by performing data augmentation on the data. Its contrast loss function is shown in formula (7):

[0076]

[0077] Among them, (f(v n ), f(v j )) is a positive example pair, v j Indicates that v n Corresponding positive examples. Some existing zero-shot action recognition methods also use contrastive loss functions, which introduce category supervision information. Specifically, the visual representation is brought closer to the semantic representation of its corresponding category in space, and the visual representation is moved further away from the semantic representation of other categories. The contrastive loss function is shown in formula (8):

[0078]

[0079] Where S is the set of visible categories. n v n The corresponding semantic information, s k is the semantic information of each category in S. It should be noted that for these methods, videos of the same category share the same semantic features, which ignore the differences between different individuals of the same category of videos. The present invention proposes a new instance-level video attribute as a semantic feature. The instance-level video attribute includes both the personality of the video (derived from video capture) and the characteristics of the category to which the video belongs (derived from the category name and category description), which can provide a more accurate semantic description for the video. Therefore, when designing the contrast loss function, the present invention brings the visual representation and semantic representation of the same video closer, and distances the visual representation and semantic representation of different videos; at the same time, it brings the visual representation of the same category of video data closer, and distances the visual representation of different categories of video data, as shown in formula (6). Such a design enables the joint space to have better alignment and uniformity, and its performance on the UCF101 and HMDB51 datasets has reached the industry-leading level.

[0080] S4. Video data in unseen categories Evaluate the model performance on.

[0081] In zero-sample action recognition, the core work is to construct accurate semantic features to connect visible categories with invisible categories. The present invention fully considers the spatial information (objects, scenes) and temporal information (motion, interaction) of the video through the video capture module, thereby providing more accurate semantic features. At the same time, the present invention designs a new contrast loss function to make the visual-semantic joint space have better alignment and uniformity. These two characteristics make the trained model perform better on invisible categories.

[0082] In summary, the present invention provides a simple and effective zero-sample video action recognition method, and the performance of this scheme on the UCF101 and HMDB51 datasets has reached the industry-leading level, as shown in Table 1.

[0083] Table 1 Performance test results on UCF101 and HMDB51 datasets

[0084] method Semantic Information Training Dataset UCF Accuracy Training Dataset HMDB Accuracy WGAN MA UCF 37.5％ - - OD MA UCF 38.3％ - - DAZSL MA UCF 48.9％ - - Act2Vec W UCF 22.1％ HMDB 23.5％ TARN W UCF 23.2％ HMDB 19.5％ WGAN W UCF 25.8％ HMDB 29.1％ OD W UCF 26.9％ HMDB 30.2％ Object2act W UCF 30.3％ HMDB 15.6％ TS-GCN W UCF 34.2％ HMDB 23.2％ SAOE W UCF 40.4％ - - PSGNN W UCF 43.0％ HMDB 32.6％ E2E W Kinetics 48.0％ Kinetics 31.2％ E2E(aug) W Kinetics 49.2％ Kinetics 32.6％ ER ED UCF 51.8％ HMDB 35.3％ CLASTER W UCF 52.7％ HMDB 42.6％ The present invention I A Kinetics 63.3％ Kinetics 41.1％

[0085] Among them, MA refers to manually annotated attributes, W refers to word embedding vectors, ED refers to object attributes integrated into the video based on the word embedding vectors, and IA refers to instance-level video attributes proposed in the present invention.

[0086] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video action recognition method based on zero-shot learning, It is characterized in that The following steps are involved: S1. For visible category video data, visual features are extracted through a visual encoder; The video capturer is used to extract the captured content at the instance level. The captured content at the instance level includes the spatial information and temporal information of the video. The encoder SBERT in natural language processing is used to obtain the captured content features corresponding to the captured content. S2. For the category name and category description at the category level, the corresponding category name features and category description features are obtained through the encoder SBERT in natural language processing, and the captured content features, category name features and category description features are input into the relation distillation module. The relation distillation module generates visual-instance-level video attributes for each video by learning the co-occurrence relationship between the captured content features at the instance level, the category name features at the category level and the category description features; The generation process of visual-instance level video attributes is: Get the category name y through the encoder SBERT in natural language processing n And the category description n The corresponding features are shown in formula (3): Among them, h n is the class name feature, d n Describe characteristics for the categories; will capture the content features c n , category name feature h n And the category description feature d n Input to the relation distillation module to obtain the instance-level video attribute a n , as shown in formula (4): a n =MultiHead(c n ,h n ,h n )+MultiHead(c n ,d n ,d n ) (4) Among them, MultiHead(Q,K,V) refers to the multi-head attention mechanism, which splices h self-attention modules to obtain information in different subspaces. The calculation process of the multi-head attention mechanism is shown in formula (5): Among them, the first formula represents the calculation process of the self-attention module, the second formula represents the calculation process of the multi-head attention mechanism, and the third formula represents the calculation process of each head in the multi-head attention. Q represents the query value, K represents the key value, and K T is the transpose of K, V represents value, is a learnable parameter, and concat is a concatenation operation; S3, optimizing the visual-semantic joint space according to the visual-instance level video attribute contrast loss function until the model converges to obtain a trained zero-shot action recognition model; S4. Evaluate the performance of the trained zero-shot action recognition model on unseen category video data, where the category set of the visible category does not overlap with the category set of the unseen category.

2. The video action recognition method based on zero-sample learning according to claim 1, It is characterized in that In step S1, the visible category video data is represented as Among them, x n Refers to the original video clip, y n ∈S refers to the category name corresponding to the video clip, S = {1,…,K} refers to the category set of visible categories, there are K categories of videos in total, and N refers to the number of videos in the visible category.

3. The video action recognition method based on zero-sample learning according to claim 1, It is characterized in that In step S1, for each video data x n , the output of the video capture device is Among them, ts l Indicates the start time of the event, te l Indicates the end time of the event, c l Represents the description statement of the event, L represents the video data x n , the video capturer outputs description statements of L events.

4. The video action recognition method based on zero-sample learning according to claim 3, It is characterized in that The captured content features corresponding to the L events are averaged through the encoder SBERT in natural language processing to obtain the captured content features corresponding to each video captured content, as shown in formula (2): Among them, c l A descriptive statement that represents an event.

5. The video action recognition method based on zero-sample learning according to claim 1, It is characterized in that In step S3, for the visual feature v of the video n , the visual representation f(v n ); for the instance-level video attribute a of the video n , and the semantic representation g(a n ); after passing through the two projectors, the visual features and instance-level video attributes are mapped into a joint visual-semantic space.

6. The video action recognition method based on zero-sample learning according to claim 5, It is characterized in that In the visual-semantic joint space of step S3, the visual representation and semantic representation of the same video are brought closer, while the visual representation and semantic representation of different videos are pulled further away; the visual representation of the same type of video data is brought closer, while the visual representation of different types of video data is pulled further away, as shown in formula (6): Among them, τ is the temperature coefficient, l refers to the category of each video data, f(v k ) represents the visual representation corresponding to the kth sample in the batch data, and N is the batch size.

7. The video action recognition method based on zero-sample learning according to claim 1, It is characterized in that In step S4, the video data of the invisible category is represented as M refers to the number of videos in the invisible category, and the category set of the invisible category is There are T category videos in total.

Images