Zero-shot action recognition method based on human skeleton data and semantic association

Abstract

The application is a zero-shot behavior recognition method based on human skeleton data and semantic association, comprising the following steps: step 1: using a trained skeleton feature extraction network to extract visual features from the skeleton data; step 2: constructing an adversarial generation network model and a classifier network; step 3: training the adversarial generation network model, and the feature generation network generates visual feature samples corresponding to visible class labels; step 4: inputting the semantic features of the unvisible class data into the visual feature samples corresponding to the unvisible class labels of the feature generation network, and training the classifier network; step 5: when recognizing the behavior samples of the unvisible class, first, the visual features are extracted from the behavior samples by using the skeleton feature extraction network, and then the visual features are input into the classifier network to obtain the classification result. The application effectively utilizes the semantic association between each word in the phrase when representing semantics, improves the problem that the generalization ability of the generator in the zero-shot behavior recognition is low, and improves the recognition rate of the unvisible class after the training of the classifier.

Description

Technical Field

[0001] This invention belongs to the field of video understanding technology, specifically relating to a zero-shot behavior recognition method based on human skeleton data and semantic association. Background Technology

[0002] Behavior recognition technology has significant applications in video supervision and human-computer interaction. It has seen considerable development in recent years, especially after the introduction of graph convolutional techniques, making skeleton-based behavior recognition a new hot topic in the field. However, when applying existing networks to behavior categories outside the training categories, the final recognition accuracy will drop significantly without collecting sufficient data samples for retraining. Furthermore, the inherent danger or implementation difficulty of certain behavior categories may result in high costs for data sample collection, hindering the industrial application of behavior recognition.

[0003] Zero-shot action recognition technology abstracts the aforementioned problem. It divides the categories in the dataset into visible and invisible classes. The aim is to obtain a classifier capable of classifying and recognizing invisible classes using only a training set containing visible classes, by using category labels as auxiliary information. A typical approach in zero-shot action recognition is feature generation. This method uses adversarial generation to generate a generator network that can convert the semantic features of category labels into visual features of skeleton data. This generator network can then use the semantic features of unseen class labels to obtain the corresponding visual features, solving the problem of being unable to train a classifier due to the lack of unseen class samples in zero-shot scenarios.

[0004] In the methods described above, category labels play a crucial role as auxiliary information. Unlike category labels in image recognition tasks, category labels in action recognition tasks are often composed of phrases or sentences. However, existing zero-shot action recognition work has not paid much attention to category labels. Currently, many works directly average the semantic features of each word in the phrase to obtain a one-dimensional tensor as the semantic feature of that category. This simple pooling method obscures the role of each word in the feature projection process, leading to the generator's generated results having similar intra-class and inter-class distances, resulting in low classification accuracy of the subsequent trained classifier. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a zero-shot behavior recognition method based on human skeleton data and semantic association. This method extracts visual features from skeleton data and semantic features from category labels using a pre-trained network, then inputs these features into a generative adversarial network (GAN) for training. The trained GAN then generates visual features for unseen classes to train a classifier network. This method preserves the semantic features of each word in the category labels, and the GAN extracts the semantic association features of each word in the category labels, ultimately converting them into visual features. Simultaneously, a feature assimilation loss is introduced to optimize the task objective of the GAN, thereby improving its generalization ability and mitigating the problem of similar intra-class and inter-class distances in the generated results, thus improving the recognition accuracy of the final classifier.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] This invention is a zero-shot behavior recognition method based on human skeleton data and semantic association. The recognition method includes two stages and the following steps:

[0008] Step 1: Transfer the skeleton dataset The categories in the dataset are divided into visible and invisible classes, which are used as the training set and the test set, respectively, for the skeleton dataset. The category labels in the dataset are modified, and the semantic features of words in each category label are extracted using an existing word semantic feature extraction model. A corresponding mask matrix is ​​then created for each label, using the skeleton dataset. The skeleton feature extraction network is pre-trained, and then used to extract features from the skeleton dataset. Visual features are extracted from the skeleton data for subsequent training.

[0009] Step 2: Construct the Generative Adversarial Network (GAN) model needed in Stage 1 and the classifier network needed in Stage 2. The GAN model consists of a feature generation network and a discriminator network. The feature generation network includes a semantic association extraction module and a feature projection module, which are connected in series. The discriminator network consists of two fully connected layers and one activation layer. The classifier network also consists of two fully connected layers and one activation layer. The semantic association extraction module consists of three fully connected layers, one residual connection, and one normalization layer. The feature projection module is a convolutional neural network consisting of three one-dimensional convolutional layers, two activation layers, and three normalization layers.

[0010] The generation process of the adversarial generative network model is as follows:

[0011] Step 2-1: Semantic features and random noise are concatenated and input into the feature generation network, which then outputs the corresponding visual features.

[0012] Step 2-2: From the skeleton dataset The visual features extracted from the data and the visual features generated in step 2-1 are input together into the discriminator network. The discriminator network outputs 0 or 1 to indicate whether the corresponding visual feature comes from the skeleton data or the output of the feature generation network.

[0013] Step 3: This step is stage 1. The semantic features, mask matrix and visual features obtained in step 1 are used to train the adversarial generative network model in step 2. The feature generation network generates visual feature samples corresponding to the visible class labels.

[0014] A feature assimilation loss function is introduced when training the feature generation network. This feature assimilation loss function is based on the triplet loss function. In the feature assimilation loss function, the samples with the largest distance among the visual features of the same category extracted from the skeleton data in each training batch are taken as positive samples. In each training batch, the samples with the smallest distance among the other visual features that do not belong to the same category generated by the feature generation network are taken as negative samples. The distance metric used here is Euclidean distance. The final objective function of the adversarial generative network is:

[0015]

[0016] in, It is a balancing factor used to balance the relationship between adversarial generation loss and feature assimilation loss; This represents the adversarial generative loss, and its expression is:

[0017]

[0018] In this expression, This represents the output of the discriminator network. This represents the output of the feature generation network;

[0019] The representative feature assimilation loss is expressed as follows:

[0020]

[0021] In this expression, margin is a constant greater than 0. It is a Euclidean distance metric function. The maximum distance between the generated visual features of labels of the same category and the visual features extracted from the skeleton data. It represents the minimum distance between other visual features that do not belong to the same category among the generated visual features.

[0022] Step 4: This step is stage 2. The semantic features of the invisible class data in step 1 are input into the visual feature samples corresponding to the invisible class labels of the feature generation network trained in step 3. The generated samples are used to train the classifier network. The cross-entropy loss function is used for training the classifier network in step 4.

[0023] Step 5: When identifying behavioral samples of unseen classes, first use the skeleton feature extraction network to extract visual features, and then input them into the classifier network in step 4 to obtain the classification result.

[0024] A further improvement of this invention is that the training of the adversarial generative network model is performed adversarially, specifically including the following steps:

[0025] Step 3-1: Freeze the model weight parameters of the feature generation network and update the weight parameters of the discriminator network through backpropagation;

[0026] Step 3-2: Then freeze the weight parameters of the discriminator network and update the weight parameters of the feature generation network through backpropagation;

[0027] Step 3-3: Repeat the above steps until the preset number of executions is completed.

[0028] A further improvement of the present invention is that step 5 specifically includes the following steps:

[0029] Step 5-1: Use the skeleton feature extraction network described in Step 1 to extract the corresponding visual features from the skeleton samples;

[0030] Step 5-2: Use the classifier network trained in Step 4 to perform classification and obtain the final result.

[0031] A further improvement of the present invention is that: in step 1, the skeleton dataset... Modify the category labels in the document, specifically including the following steps:

[0032] Step 1-1: Analyze the skeleton dataset in the form of executor, execution action, and execution target. The category tags in the code are patched to ensure that the maximum number of words for each category tag is similar;

[0033] Step 1-2: Use the existing word feature extraction network to extract the semantic features of each word in the label, and denote the dimension of the semantic features of each word as... When extracting semantic features, articles and prepositions in the category labels will be ignored and their corresponding semantic features will not be extracted.

[0034] Steps 1-3: For cases where the phrase length of the category labels obtained in Step 1-1 is short, zero-padding is used to fill the variable-length semantic feature matrix with a fixed length. Feature matrix ,in This represents the length after padding;

[0035] Steps 1-4: Create a mask matrix for each category label. The size of the mask matrix for each category label is... Each column corresponds to a feature matrix of fixed length. The system checks if the category label obtained in step 1-1 contains a corresponding word. If no word is found, it is stored. If there are words, store them. .

[0036] The beneficial effects of this invention are as follows: This invention utilizes visual features of unseen classes generated by a feature generation network to train a classifier network, avoiding the problem in zero-shot behavior recognition where the classifier network cannot be trained due to a lack of training samples for unseen classes. It preserves the semantic features of each word in the category label and fully utilizes the semantic information contained therein using a feature generation network, which is of great significance for zero-shot behavior recognition and related research. Furthermore, by using real features as positive samples and other generated features as negative samples, and incorporating the feature assimilation loss into the training loss function of the adversarial generative network, the network's objective function better aligns with the requirements of zero-shot learning, ultimately improving the robustness of zero-shot behavior recognition.

[0037] This invention effectively utilizes the semantic association between words when representing semantics in phrases, improves the problem of low generalization ability of generators in zero-shot behavior recognition, and enhances the recognition rate of unseen classes after classifier training. Attached Figure Description

[0038] Figure 1 This is a flowchart of the two-stage training process for the zero-shot behavior recognition method of this invention.

[0039] Figure 2 This is a flowchart of the feature generation network of the present invention. Detailed Implementation

[0040] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0041] like Figure 1The training flowchart of this invention is shown. This invention is a zero-shot behavior recognition method based on human skeleton data and semantic association. The implementation of the recognition method includes two stages, specifically including the following steps:

[0042] Step 1: Transfer the skeleton dataset The categories in the dataset are randomly divided into visible and invisible classes in a 9:1 ratio, serving as the training and test sets respectively. (This is related to the skeleton dataset.) The category labels in the dataset are modified, and the semantic features of words in each category label are extracted using an existing word semantic feature extraction model. A corresponding mask matrix is ​​then created for each label, using the skeleton dataset. The skeleton feature extraction network is pre-trained, and then used to extract features from the skeleton dataset. Visual features are extracted from the skeleton data for subsequent training. In this example, the visual features of each skeleton sample have a dimension of 1024.

[0043] Wherein: in step 1, the skeleton dataset is processed. Modify the category labels in the document, specifically including the following steps:

[0044] Step 1-1: Analyze the skeleton dataset in the form of executor, execution action, and execution target. The category tags in the code are patched to ensure that the maximum number of words for each category tag is similar;

[0045] Step 1-2: Use the existing word feature extraction network word2vec to extract the semantic features of each word in the label, and denote the dimension of the semantic features of each word as . In this embodiment, When extracting semantic features, articles in the category labels such as a, an, and the, as well as prepositions such as of and to, will be ignored and their corresponding semantic features will not be extracted.

[0046] Steps 1-3: For cases where the phrase length of the category labels obtained in Step 1-1 is short, zero-padding is used to fill the variable-length semantic feature matrix with a fixed length. Feature matrix ,in This represents the length after padding, in this embodiment. .

[0047] Steps 1-4: Create a mask matrix for each category label. The size of the mask matrix for each category label is... Each column corresponds to a feature matrix of fixed length. The system checks if the category label obtained in step 1-1 contains a corresponding word. If no word is found, the label is stored. If there are words, store them. .

[0048] Step 2: Construct the adversarial generative network model needed in Stage 1. This model consists of a feature generation network and a discriminator network. The feature generation network includes a semantic association extraction module and a feature projection module, which are connected in series. The semantic association extraction module consists of 3 fully connected layers, 1 residual connection, and 1 normalization layer. The feature projection module is a convolutional neural network consisting of 3 one-dimensional convolutional layers, 2 activation layers, and 3 normalization layers. Construct the classifier network needed in Stage 2. This discriminator network consists of 2 fully connected layers and 1 activation layer. The input and output dimensions of the fully connected layers are (1024, 512) and (512, 1), respectively.

[0049] The generation process of the adversarial generative network model is as follows:

[0050] Step 2-1: Semantic features and random noise are concatenated and input into the feature generation network, which then outputs the corresponding visual features.

[0051] Step 2-2: From the skeleton dataset The visual features extracted from the data and the visual features generated in step 2-1 are input together into the discriminator network. The discriminator network outputs 0 or 1 to indicate whether the corresponding visual feature comes from the skeleton data or the output of the feature generation network.

[0052] Step 3: This step is Phase 1 of the entire method. The semantic features, mask matrix, and visual features obtained in Step 1 are used to train the adversarial generative network model in Step 2. The feature generation network generates visual feature samples corresponding to the visible class labels.

[0053] In the training of the feature generation network, a feature assimilation loss function is incorporated. Training is conducted adversarially. In each training batch, the model weight parameters of the feature generation network are first frozen, and the weight parameters of the discriminator network are updated via backpropagation. Then, the weight parameters of the discriminator network are frozen, and the weight parameters of the feature generation network are updated via backpropagation. This process is repeated until the preset number of executions is completed.

[0054] In step 3, a feature assimilation loss function is introduced when training the feature generation network. This feature assimilation loss function is based on the triplet loss function. In this function, the samples with the largest distance among visual features of the same category extracted from the skeleton data in each training batch are taken as positive samples. In each training batch, the samples with the smallest distance among other visual features not belonging to the same category generated by the feature generation network are taken as negative samples. The distance metric used here is Euclidean distance. The final objective function of the adversarial generative network is:

[0055]

[0056] in, It is a balancing factor used to balance the relationship between adversarial generation loss and feature assimilation loss; This represents the adversarial generative loss, and its expression is:

[0057]

[0058] In this expression, This represents the output of the discriminator network. This represents the output of the feature generation network;

[0059] The representative feature assimilation loss is expressed as follows:

[0060]

[0061] In this expression, margin is a constant greater than 0. It is a Euclidean distance metric function. The maximum distance between the generated visual features of labels of the same category and the visual features extracted from the skeleton data. It represents the minimum distance between other visual features that do not belong to the same category among the generated visual features.

[0062] Step 4: This step is stage 2 of the entire method. The semantic features of the invisible class data from step 1 are input into the visual feature samples corresponding to the invisible class labels of the feature generation network trained in step 3. The generated samples are used to train the classifier network, which uses the cross-entropy loss function for training.

[0063] Step 5: When identifying behavioral samples of unseen classes, first use the skeleton feature extraction network in Step 1 to extract visual features, then input them into the classifier network in Step 4 to obtain the classification results, and calculate the classification accuracy, which is the ratio of the number of correctly classified samples to the total number of samples.

[0064] By employing the aforementioned technical solution, the category labels of the dataset are divided into visible and invisible classes, which are applied to the training and testing sets respectively. The skeleton data and category labels are input into two pre-trained networks to obtain corresponding visual and semantic features. Subsequently, these two types of features and the corresponding mask feature matrix are input into the generative adversarial network (GAN) for training. Specifically, the semantic association feature extraction module calculates the association degree between each word, and then updates the features based on a weighted calculation according to the association degree. The updated semantic features are then converted into visual features by the feature projection module. In contrast, the feature projection module focuses more on the association features between adjacent words. Furthermore, the inclusion of feature assimilation loss constrains the task objective of the feature generation network, enhancing the generalization ability of the final generator network.

[0065] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A zero-shot behavior recognition method based on human skeleton data and semantic association, the implementation of which includes two stages, characterized in that: The zero-shot behavior recognition method includes the following steps: Step 1: Transfer the skeleton dataset The categories in the dataset are divided into visible and invisible classes, which are used as the training set and the test set, respectively, for the skeleton dataset. The category labels in the dataset are modified, and the semantic features of words in each category label are extracted using an existing word semantic feature extraction model. A corresponding mask matrix is ​​then created for each label, using the skeleton dataset. The skeleton feature extraction network is pre-trained, and then used to extract features from the skeleton dataset. Visual features are extracted from the skeleton data for subsequent training. Step 2: Construct the adversarial generative network model needed in Phase 1 and the classifier network needed in Phase 2; Step 3: Train the adversarial generative network model in Step 2 using the semantic features, mask matrix and visual features obtained in Step 1. The adversarial generative network model consists of a feature generation network and a discriminator network. The feature generation network obtains visual feature samples corresponding to the visible class labels. Step 4: Input the semantic features of the invisible class data from Step 1 into the feature generation network trained in Step 3 to obtain the visual feature samples corresponding to the invisible class labels, and use the generated samples to train the classifier network; Step 5: When identifying unseen behavior samples, first use a skeleton feature extraction network to extract visual features, then input these features into the classifier network in Step 4 to obtain the classification result; where: In step 3, a feature assimilation loss function is introduced when training the feature generation network. This feature assimilation loss function is based on the triplet loss function. In this function, the samples with the largest distance among visual features of the same category extracted from the skeleton data in each training batch are taken as positive samples. In each training batch, the samples with the smallest distance among other visual features not belonging to the same category generated by the feature generation network are taken as negative samples. The distance metric used here is Euclidean distance. The final objective function of the adversarial generative network is: ， in, It is a balancing factor used to balance the relationship between adversarial generation loss and feature assimilation loss; This represents the adversarial generative loss, and its expression is: ， In this expression, This represents the output of the discriminator network. This represents the output of the feature generation network; The representative feature assimilation loss is expressed as follows: ， In this expression, margin is a constant greater than 0. It is a Euclidean distance metric function. The maximum distance between the generated visual features of labels of the same category and the visual features extracted from the skeleton data. It represents the minimum distance between other visual features that do not belong to the same category among the generated visual features; In step 2, the feature generation network includes a semantic association extraction module and a feature projection module, and the semantic association extraction module and the feature projection module are connected in series. The discriminator network consists of two fully connected layers and one activation layer, and the classifier network consists of two fully connected layers and one activation layer. The semantic association extraction module consists of 3 fully connected layers, 1 residual connection and 1 normalization layer, and the feature projection module consists of a convolutional neural network consisting of 3 one-dimensional convolutional layers, 2 activation layers and 3 normalization layers. The training of the adversarial generative network model in step 3 is carried out in an adversarial manner, specifically including the following steps: Step 3-1: In a training batch, freeze the model weight parameters of the feature generation network and update the weight parameters of the discriminator network through backpropagation. Step 3-2: Then freeze the weight parameters of the discriminator network and update the weight parameters of the feature generation network through backpropagation; Step 3-3: Repeat the above steps until the preset number of executions is completed; In step 1, the skeleton dataset is processed. Modify the category labels in the document, specifically including the following steps: Step 1-1: Analyze the skeleton dataset in the form of executor, execution action, and execution target. The category tags in the code are patched to ensure that the maximum number of words for each category tag is similar; Step 1-2: Use the existing word feature extraction network to extract the semantic features of each word in the label, and denote the dimension of the semantic features of each word as... When extracting semantic features, articles and prepositions in the category labels will be ignored and their corresponding semantic features will not be extracted. Steps 1-3: For cases where the phrase length of the category labels obtained in Step 1-1 is short, zero-padding is used to fill the variable-length semantic feature matrix with a fixed length. Feature matrix ,in This represents the length after padding; Steps 1-4: Create a mask matrix for each category label. The size of the mask matrix for each category label is... Each column corresponds to a feature matrix of fixed length. The system checks if the category label obtained in step 1-1 contains a corresponding word. If no word is found, the label is stored. If there are words, store them. .

2. The zero-shot behavior recognition method based on human skeleton data and semantic association according to claim 1, characterized in that: The generation process of the adversarial generative network model is as follows: Step 2-1: Semantic features and random noise are concatenated and input into the feature generation network, which then outputs the corresponding visual features. Step 2-2: From the skeleton dataset The visual features extracted from the data and the visual features generated in step 2-1 are input together into the discriminator network. The discriminator network outputs 0 or 1 to indicate whether the corresponding visual feature comes from the skeleton data or the output of the feature generation network.

3. The zero-shot behavior recognition method based on human skeleton data and semantic association according to claim 1, characterized in that: In step 4, the classifier network training uses the cross-entropy loss function.

Images