Zero-sample human behavior recognition method guided by multiple prompt messages

Abstract

The invention discloses a multi-prompt information guided zero-sample human behavior recognition method, which comprises the following steps of S100, constructing a strategy, and emphatically paying attention to feature information related to behavior categories in an early stage of visual feature extraction through a strategy guide model; s200, by utilizing priori knowledge among different modes in a pre-trained visual language model, through alignment with skeleton sequence features, helping the model to obtain semantic information related to behavior categories from the model; s300, through common guidance of visual and semantic prompt information, enabling the model to realize effective identification in a reasoning task without a category based on the strategy; through the common guidance of visual and semantic prompt information, the strategy enables the model to obtain better recognition performance in a reasoning task without a category, and the strategy pays attention to the human body posture feature change and visual semantic features highly related to the behavior category at the same time. Therefore, the problem that the reasoning generalization ability of the model is insufficient when no category is seen is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of model feature data processing, and in particular to a zero-sample human behavior recognition method guided by multiple prompt information. Background Art

[0002] In recent years, in the field of human action recognition, the research focus has gradually shifted from traditional fully supervised learning methods to the more challenging field of zero-shot / few-shot human action recognition. Fully supervised learning methods have achieved remarkable success in using a large amount of labeled data for model training, but in practical applications they face problems such as difficulty in obtaining data of some categories and high labeling costs. This has prompted researchers to turn their attention to zero-shot learning in order to achieve prediction of unseen categories based on prompt information without having seen new behavior category samples. In current research, a key challenge of zero-shot human action recognition is how to effectively handle unseen categories. It is generally believed in academia that the solution to this challenge requires a combination of multimodal information and advanced deep learning techniques.

[0003] There is a lack of effective fusion of multiple modal information in zero-shot human behavior recognition tasks, and the learning method based on modal isolation cannot well understand the data ambiguity caused by prompt information. In zero-shot learning, prompt information is not fully utilized, there is a lack of effective interaction and guidance between the extraction of visual modal and semantic feature information, and there is a lack of effective fusion of multiple modal information that is highly related to human behavior tasks. The model prediction results are greatly affected by prompt information, resulting in insufficient semantic understanding of the model for unseen categories.

[0004] Therefore, it is necessary to design a zero-shot human action recognition method guided by multiple cue information. Summary of the invention

[0005] The purpose of the present invention is to provide a zero-sample human behavior recognition method guided by multiple prompt information to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] The zero-sample human behavior recognition method guided by multiple prompt information includes the following steps:

[0008] S100, construct a strategy and use it to guide the model to focus on feature information related to the behavior category in the early stage of visual feature extraction;

[0009] S200, using the prior knowledge between different modalities in the pre-trained visual language model, by aligning with the skeleton sequence features, to help the model obtain semantic information related to the behavior category;

[0010] S300, through the joint guidance of visual and semantic cue information, based on this strategy, the model can achieve effective recognition in reasoning tasks of unseen categories.

[0011] According to the above technical solution, the specific working method of S200 is as follows:

[0012] Import a batch of training data (x s ,v s ,y s ), the visual category semantic guidance feature information p is extracted through the pre-trained visual language model, namely VLM ′ , and generate the skeleton feature prediction score vector p by aligning the skeleton sequence features and the category prompt information, and obtain the prediction values ​​of the K categories in the sample The calculation formula is as follows:

[0013]

[0014] In the formula, It is the probability value predicted for each category in A under the guidance of the two modal features of skeleton sequence and vision, p ′ (v|A) is the guidance score of the visual semantic feature for each category of prompt information, and p(x|A) is the prediction score of the skeleton sequence feature for each category, where:

[0015] p ′ The calculation formula for (v|A) is as follows:

[0016]

[0017] p(v1,a i ) is the probability value of predicting the visual feature v1 as the i-th category, a i is the feature vector of the text prompt information of the i-th category, p(a i ) is the probability that the i-th behavior category is the true category;

[0018] The calculation formula of p(x|A) is as follows:

[0019]

[0020] Similarly, p(v2,a i ) is the probability value of predicting the skeleton sequence feature v2 as the i-th category, p(a i ) is the probability that the i-th behavior category is the true category.

[0021] According to the above technical solution, the distance between each visual feature vector and the semantic information needs to be measured, and cosine similarity is used as the measurement method:

[0022] d(pi ,a j )=cosine(p i ,a j ) (4)

[0023] p i is the feature vector of the i-th skeleton sequence, a j is the semantic feature vector of the jth category, d(p i ,a j ) is the cosine similarity between the i-th skeleton sequence feature vector and the j-th category semantic feature vector:

[0024]

[0025] in, Indicates that under the guidance of the prompt information and p i The category of the most similar semantic feature vector, argmin j is and p i The category with the smallest similarity value with the chord in A;

[0026]

[0027] Among them, pos(p) represents the distance between the positive sample pair and the negative sample pair in the same cluster in the prediction result of the skeleton feature, B is the batch size of the data, that is, the number of categories of prompt information, is an indicator function used to indicate the visual feature vector clustering result Is it consistent with the semantic feature vector a of the jth category? j Belonging to the same cluster, the indicator function is defined as follows:

[0028]

[0029] here The value of 1 indicates with a j Belong to the same cluster, the value is 0 with a j Do not belong to the same cluster;

[0030] Define positive and negative sample pairs within the same cluster and between different clusters, and calculate their distances:

[0031]

[0032] neg(p) represents the distance between positive sample pairs and negative sample pairs in different clusters in the prediction results of the skeleton sign;

[0033] Similarly, pos(p ′) represents the distance between the positive sample pair and the negative sample pair in the same cluster in the prediction result of the visual feature, neg(p ′ ) represents the distance between positive sample pairs and negative sample pairs between different clusters in the prediction results of the visual features;

[0034] Finally, by constructing the contrast loss function loss cc To constrain the eigenvectors of different modes to align into the same space:

[0035] loss cc =pos(p)+pos(p ′ )-neg(p)-neg(p ′ ) (9)

[0036] The design of this loss function guides the model to make the features within the same cluster closer and the features between different clusters more dispersed during the optimization process, so as to achieve effective multimodal data alignment.

[0037] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0038] Through the joint guidance of visual and semantic cue information, this strategy enables the model to achieve better recognition performance in reasoning tasks of unseen categories. This strategy focuses on both changes in human posture features and visual semantic features that are highly correlated with behavior categories, thereby making up for the model's insufficient reasoning generalization ability on unseen categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a model framework diagram of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] The present invention provides a technical solution: a zero-sample human behavior recognition method guided by multiple prompt information, comprising the following steps:

[0042] S100, construct a strategy and use it to guide the model to focus on feature information related to the behavior category in the early stage of visual feature extraction;

[0043] S200, using the prior knowledge between different modalities in the pre-trained visual language model, by aligning with the skeleton sequence features, to help the model obtain semantic information related to the behavior category;

[0044] S300, through the joint guidance of visual and semantic cue information, based on this strategy, the model can achieve effective recognition in reasoning tasks of unseen categories.

[0045] Specifically, the specific working method of S200 is as follows:

[0046] Import a batch of training data (x s ,v s ,y s ), the visual category semantic guidance feature information p is extracted through the pre-trained visual language model, namely VLM ′ , and generate the skeleton feature prediction score vector p by aligning the skeleton sequence features and the category prompt information, and obtain the prediction values ​​of the K categories in the sample The calculation formula is as follows:

[0047]

[0048] In the formula, It is the probability value predicted for each category in A under the guidance of the two modal features of skeleton sequence and vision, p ′ (v|A) is the guidance score of the visual semantic feature for each category of prompt information, and p(x|A) is the prediction score of the skeleton sequence feature for each category, where:

[0049] p ′ The calculation formula of (v|A) is as follows:

[0050]

[0051] p(v1,a i ) is the probability value of predicting the visual feature v1 as the i-th category, a i is the feature vector of the text prompt information of the i-th category, p(a i ) is the probability that the i-th behavior category is the true category;

[0052] The calculation formula of p(x|A) is as follows:

[0053]

[0054] Similarly, p(v2,a i ) is the probability value of predicting the skeleton sequence feature v2 as the i-th category, p(a i ) is the probability that the i-th behavior category is the true category.

[0055] Specifically, the distance between each visual feature vector and the semantic information needs to be measured, and cosine similarity is used as the measurement method:

[0056] d(p i ,a j )=cosine(p i ,a j ) (4)

[0057] p i is the feature vector of the i-th skeleton sequence, a j is the semantic feature vector of the jth category, d(p i ,a j ) is the cosine similarity between the i-th skeleton sequence feature vector and the j-th category semantic feature vector:

[0058]

[0059] in, Indicates that under the guidance of the prompt information and p i The category of the most similar semantic feature vector, argmin j is and p i The category with the smallest similarity value with the chord in A;

[0060]

[0061] Among them, pos(p) represents the distance between the positive sample pair and the negative sample pair in the same cluster in the prediction result of the skeleton feature, B is the batch size of the data, that is, the number of categories of prompt information, is an indicator function used to indicate the visual feature vector clustering result Is it consistent with the semantic feature vector a of the jth category? j Belonging to the same cluster, the indicator function is defined as follows:

[0062]

[0063] here The value of 1 indicates with a j Belong to the same cluster, the value is 0 with a j Do not belong to the same cluster;

[0064] Define positive and negative sample pairs within the same cluster and between different clusters, and calculate their distances:

[0065]

[0066] neg(p) represents the distance between positive sample pairs and negative sample pairs in different clusters in the prediction results of the skeleton sign;

[0067] Similarly, pos(p ′) represents the distance between the positive sample pair and the negative sample pair in the same cluster in the prediction result of the visual feature, neg(p ′ ) represents the distance between positive sample pairs and negative sample pairs between different clusters in the prediction results of the visual features;

[0068] Finally, by constructing the contrast loss function loss cc To constrain the eigenvectors of different modes to align into the same space:

[0069] loss cc =pos(p)+pos(p ′ )-neg(p)-neg(p ′ ) (9)

[0070] The design of this loss function guides the model to make the features within the same cluster closer and the features between different clusters more dispersed during the optimization process, so as to achieve effective multimodal data alignment.

[0071] Example:

[0072] We evaluated the results on two multi-split datasets, NTU-60 and NTU-120, respectively, using the Generative Adversarial Network (GAN) and matching pipelines. In zero-shot learning, the experimental results are shown in the following table:

[0073]

[0074] In zero-shot learning, different datasets, different ways of dividing visible and invisible categories, and different text prompt information will have a huge impact on the verification results. Therefore, we chose the same dataset and data division method as SMIE, including the category description text enhanced by ChatGPT. At the same time, in order to compare the impact of different pipelines, we reproduced the test results of OD and AURL on the two pipelines as our baseline, and used our method on the two paradigms. The test results are shown in Table 1. The test results on NTU-120 exceed other methods including the baseline. Compared with the current best method SMIE method, on the four partition schemes of NTU-60 and NTU-12, 55 / 5, 48 / 12, 110 / 10 and 96 / 24, OD+BSkeleton is 4.18%, 13.24%, 8.58%, 8.59% higher than it respectively, and AURL+BSkeleton is 5.06%, 12.78%, 11.95%, 10.82% higher than it respectively. These results show that our method has better effect in zero-shot learning tasks.

[0075] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0076] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A zero-sample human behavior recognition method guided by multiple prompt information, characterized in that: The following steps are involved: S100, construct a strategy and use it to guide the model to focus on feature information related to the behavior category in the early stage of visual feature extraction; S200, using the prior knowledge between different modalities in the pre-trained visual language model, by aligning with the skeleton sequence features, to help the model obtain semantic information related to the behavior category; S300, through the joint guidance of visual and semantic cue information, based on this strategy, the model can achieve effective recognition in reasoning tasks of unseen categories.

2. The zero-sample human behavior recognition method guided by multiple prompt information according to claim 1 is characterized in that: The specific working method of S200 is as follows: Import a batch of training data (x s ,v s ,y s ), extract the visual category semantic guidance feature information p′ through the pre-trained visual language model, i.e. VLM, and generate the skeleton feature prediction score vector p by aligning the skeleton sequence features and the category prompt information, and obtain the prediction values ​​of K categories in the sample The calculation formula is as follows: In the formula, is the probability value predicted for each category in A under the guidance of the prompt information of the two modal features of skeleton sequence and vision, p′(v|A) is the guidance score of the visual semantic feature for each category of prompt information, and p(x|A) is the prediction score of the skeleton sequence feature for each category, where: The calculation formula of p′(v|A) is as follows: p(v1,a i ) is the probability value of predicting the visual feature v1 as the i-th category, a i is the feature vector of the text prompt information of the i-th category, p(a i ) is the probability that the i-th behavior category is the true category; The calculation formula of p(x|A) is as follows: Similarly, p(v2, a i ) is the probability value of predicting the skeleton sequence feature v2 as the i-th category, p(a i ) is the probability that the i-th behavior category is the true category.

3. The zero-sample human behavior recognition method guided by multiple prompt information according to claim 2 is characterized in that: The distance between each visual feature vector and the semantic information needs to be measured, and cosine similarity is used as the measurement method: d(p i ,a j )=cosine(p i ,a j ) (4) p i is the feature vector of the i-th skeleton sequence, a j is the semantic feature vector of the jth category, d(p i , a j ) is the cosine similarity between the i-th skeleton sequence feature vector and the j-th category semantic feature vector: in, Indicates that under the guidance of the prompt information and p i The category of the most similar semantic feature vector, argmin j Yes and p i The category with the smallest similarity value with the chord in A; Among them, pos(p) represents the distance between the positive sample pair and the negative sample pair in the same cluster in the prediction result of the skeleton feature, B is the batch size of the data, that is, the number of categories of prompt information, is an indicator function used to indicate the visual feature vector clustering result Is it consistent with the semantic feature vector a of the jth category? j Belonging to the same cluster, the indicator function is defined as follows: here The value of 1 indicates with a j Belong to the same cluster, the value is 0 with a j Do not belong to the same cluster; Define positive and negative sample pairs within the same cluster and between different clusters, and calculate their distances: neg(p) represents the distance between positive sample pairs and negative sample pairs in different clusters in the prediction results of the skeleton sign; Similarly, pos(p′) represents the distance between the positive sample pairs and the negative sample pairs in the same cluster in the prediction results of the visual features, and neg(p′) represents the distance between the positive sample pairs and the negative sample pairs in different clusters in the prediction results of the visual features; Finally, by constructing the contrast loss function loss cc To constrain the eigenvectors of different modes to align into the same space: loss cc =pos(p)+pos(p′)-neg(p)-neg(p′) (9) The design of this loss function guides the model to make the features within the same cluster closer and the features between different clusters more dispersed during the optimization process, so as to achieve effective multimodal data alignment.

Images