Noise-robust unsupervised representation learning method and device for human skeleton assisted by multimodal large model

Abstract

The present invention discloses a method and device for learning a human skeleton representation that is noise-robust under the assistance of a multimodal large model. The method includes: obtaining key frames by sampling RGB videos corresponding to a human skeleton sequence, generating action text descriptions through a multimodal large model; then encoding the human skeleton sequence and the corresponding action text descriptions into a common space, generating a more reliable learning target through a sample noise evaluation method that jointly estimates abnormal samples and information entropy, and training the model using a noise-aware contrast learning method; finally, after a certain batch of iterative training, an encoder with better performance can be obtained. Under the premise of no additional training and inference costs, the present invention utilizes the ability of the encoder itself to distinguish text descriptions that are not related to actions, so that the model learns a more reasonable similarity representation, while better improving the performance of the encoder in downstream tasks.

Description

Technical Field

[0001] The present invention relates to the field of unsupervised representation learning of human skeleton, and in particular to a noise-robust unsupervised representation learning method and device for human skeleton assisted by a large multimodal model. Background Art

[0002] Human action recognition is a topic of great concern in the field of artificial intelligence research. It is widely used in human-computer interaction, human motion analysis, video surveillance, etc. Compared with the action recognition algorithm based on RGB video, the action recognition based on human skeleton joints has the advantages of light weight, high inference efficiency and low noise. From the perspective of training methods, the current human skeleton action recognition algorithms based on deep learning can be divided into three categories: supervised learning, semi-supervised learning and unsupervised learning. Among them, the human skeleton action recognition algorithm based on unsupervised representation learning is usually implemented based on the idea of ​​contrastive learning. It requires the model to learn similar representations between samples from unlabeled human skeleton sequence samples, which greatly reduces the manpower and material resources consumed by labeling and enhances the generalization ability of the model in production and life.

[0003] However, it is difficult to distinguish some local actions, such as "thumbs up" and "making a victory gesture", by relying solely on comparative learning between the skeleton and its corresponding sample features. Some methods use a large model to generate action descriptions for action label texts, and learn the similarity representation between skeleton features and corresponding text description encoding features in the encoding space through comparative learning, thereby using the large model's ability to understand actions to guide the human skeleton action recognition model to understand skeleton actions; some methods use pre-training to enable the human skeleton encoder to compress the human skeleton sequence into "codes", and use the "codes" as the input of the large model, and use the labels as output to fine-tune the large model, so that the large model can predict action categories.

[0004] However, these methods have the following shortcomings: 1. The text generated by the action label text cannot get rid of the dependence on the label. 2. The text descriptions generated for the same type of samples are consistent, and there is no detailed description of each human skeleton action itself. In reality, different people have different postures when doing the same action in different states, which requires the human skeleton action recognition model to be robust; 3. Although the multimodal large model has a strong understanding ability in various fields, the content it generates has two types of text noise: the wrong description of the action category appears, for example, for the action sample with the action label "put on shoe", the corresponding action text description generated by the multimodal large model is "She is standing on the floor, and it appears that she is trying to clean her toothbrush." ​​It does not contain any action distinguishing information, such as "He is standing". Summary of the invention

[0005] In view of the deficiencies of the above-mentioned prior art, the present invention provides a noise-robust unsupervised representation learning method and device for a human skeleton assisted by a large multimodal model. The method can provide action description text for skeleton samples for the unsupervised representation learning of a human skeleton action recognition model. At the same time, for the action text description of samples generated by the large multimodal model with errors or no distinguishing information, a sample noise evaluation method for jointly estimating abnormal samples and information entropy is proposed for the first time. The text noise in the generated action text description is denoised, the influence of text noise on model training is reduced, and a more credible supervisory signal is provided for the pre-training of a human skeleton sequence encoder, thereby further improving the robustness and generalization ability of the model.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] According to a first aspect of the present specification, a noise-robust human skeleton unsupervised representation learning method assisted by a multimodal large model is provided, comprising the following steps:

[0008] Step S1, data acquisition. The original human skeleton sequence is directly provided by the human skeleton sequence public dataset. The RGB video corresponding to the human skeleton sequence is extracted, and the corresponding original action text description is obtained using a multimodal large model.

[0009] Step S2, data preprocessing. The original human skeleton sequence and the corresponding original action text description are preprocessed to obtain a joint point modal human skeleton sequence and a corresponding action text description sequence, and the joint point modal human skeleton sequence is enhanced by homologous data to obtain two different enhanced human skeleton sequences.

[0010] Step S3, unsupervised representation learning. For the joint point modal human skeleton sequence, use the embedding layer and the human skeleton encoder to embed and encode to obtain the human skeleton features; for the corresponding action text description sequence, use the pre-trained CLIP (Contrastive Language-Image Pre-training, CLIP) model to encode and obtain the action text description features. Project the two into a unified feature space, and use contrastive learning to align the features between different enhancement sequences of the human skeleton, the human skeleton sequence features, and the action text description features.

[0011] Step S4: The action text description generated in step S1 contains text noise, but it is used to align with the corresponding human skeleton sample in step S3, which will cause the performance of the model to degrade on downstream tasks. For the type of text noise such as "wrong action text description", this method proposes to use a noise evaluation method based on abnormal sample identification to screen out "outlier" action text descriptions.

[0012] In step S5, in addition to the above-mentioned text noise, action text descriptions without distinguishing information are commonly present in any sample. The sample features of these action text descriptions have similar representations to all human skeleton features. This uniform distribution will make the information entropy larger. De-noising evaluation is performed based on information entropy estimation to reduce the target probability of non-distinguishing action text descriptions in the representation learning process.

[0013] In step S6, the learning objectives generated by the sample noise evaluation method in steps S4 and S5 are linearly weighted fused with the original learning objectives, and applied to the comparative learning in step S3 to learn the similarity representation between the human skeleton sequence features and the action text description features. Compared with the original learning objectives, the new learning objectives enable the model to focus on correct action text descriptions with obvious category characteristics.

[0014] Step S7, evaluate the model capability, completely freeze the pre-trained human skeleton sequence encoder and embedding layer parameters for a series of downstream tasks such as action classification, action retrieval, transfer learning, etc.

[0015] Furthermore, the step S1 specifically includes the following contents:

[0016] Step S11, sampling the key frames of the RGB video corresponding to the human skeleton sequence by using an interval sampling method to obtain G frame key frames.

[0017] Step S12, using fixed instructions, together with the key frames extracted in the previous step, are input into the multimodal large model LLaVA to generate original action text descriptions corresponding to the original human skeleton sequence, and for each original human skeleton sequence, G original action text descriptions are generated.

[0018] Furthermore, the step S3 specifically includes the following contents:

[0019] In step S31, different enhancements of the human skeleton sequence are projected to the feature space where the human skeleton sequence encoder is located through a multilayer perceptron (MLP) including layer normalization (LN).

[0020] Step S32, using a human skeleton sequence encoder and a pre-trained CLIP text encoder to encode the projected human skeleton sequence and the original action text description respectively, to obtain human skeleton sequence features and action text description features.

[0021] In step S33, the features obtained in step S32 are projected into a unified feature space to obtain the final representation of the human skeleton sequence features and the final representation of the corresponding action text description features. The similarity between the corresponding skeleton-skeleton enhancements and the similarity between the skeleton-text sample pairs are maximized by contrastive learning. The Normalized Temperature-Scaled Cross-Entropy (NT-Xent) is used as the optimization loss function to perform representation learning on the model. The definition is as follows:

[0022] (1)

[0023] in, and Represents the corresponding positive sample pair, and other samples in the batch are regarded as negative sample pairs. Indicates the calculation of cosine similarity, is a 0-1 indicator. The time value is 0, which is used to exclude sample self-similarity. is the temperature coefficient, maximizing the positive sample pair and When the similarity of is expressed, the loss can be minimized.

[0024] Furthermore, the step S4 uses a noise evaluation method for abnormal sample identification to screen abnormal action text description samples, which specifically includes the following contents:

[0025] Step S41, noise assessment is performed based on abnormal sample identification. In order to distinguish erroneous samples, it is first necessary to select a feature space measurement method. Cosine similarity is a common method used to measure vector similarity in deep learning. The cosine value of the angle between two vectors is used to represent their similarity in the feature space. The larger the cosine similarity, the smaller the distance between the two samples in the feature space. The erroneous noise texts are often small in number and have a small similarity with other samples, which makes these noises distinguishable by this evaluation method. For each G action text description feature corresponding to the human skeleton feature, the average cosine similarity between the G-1 other action text description features is calculated to obtain the average distance between the G action text descriptions and other action text descriptions.

[0026] Step S42, for the average distance obtained in step S41, it is necessary to define the "boundary" between the erroneous action sample and other samples. The mean indicates the medium-level value of a set of data, and the standard deviation can be used to measure the deviation of all samples from the mean. The deviation of abnormal samples is often large, so the mean and standard deviation of the average cosine similarity of G action text description features are calculated, and a threshold is set to filter out erroneous action text descriptions with a small similarity to other samples.

[0027] Step S43, eliminating the erroneous action text descriptions screened out in step S42, and generating an abnormal sample detection learning objective.

[0028] Furthermore, the step S5 generates a learning target using a noise sample estimation method based on information entropy estimation, which specifically includes the following contents:

[0029] Step S51, noise evaluation is performed based on information entropy estimation. As mentioned above, the example shows that the indistinguishable action text description is similar to any human skeleton sample, and information entropy can be well quantified. Its definition is as follows:

[0030] (2)

[0031] in, The information entropy of the prediction result is represented by It indicates the probability of predicting a certain category. When the probability of predicting a certain category is high, the information entropy is low. On the contrary, if the predicted probability of all categories is evenly distributed, the information entropy is high. This satisfies the above screening requirements. For each action text description feature, the similarity score is calculated with all human skeleton sample features in the batch in turn. The probability of each action text description feature and each human skeleton sample feature in the batch is obtained through the standard normalized exponential function (Softmax function). .

[0032] Step S52: Calculate the information entropy of each action text description according to the information entropy definition formula (certainty).

[0033] Step S53: Since the larger the information entropy, the less information the generated action text description contains, a learning goal based on information entropy estimation is generated by negating the information entropy.

[0034] Further, step S6 specifically includes the following contents:

[0035] Step S61, in order to apply the learning objectives generated in steps S4 and S5 to step S3 to improve the quality of skeleton-text comparison learning, the probability distribution of soft labels is used to make more detailed adjustments for model training. Specifically, the learning objectives generated by the noise evaluation method based on abnormal sample recognition and the learning objectives generated by the noise evaluation method based on information entropy estimation obtained in steps S4 and S5 are fused with the original learning objectives through a linear weighted fusion method to correct the original learning objectives.

[0036] Step S62, referring to the idea of ​​comparative learning, learn the similarity representation of human skeleton sample features and corresponding action text description features, and train the human skeleton sequence encoder end-to-end.

[0037] The second aspect of the present invention relates to a noise-robust unsupervised representation learning device for a human skeleton assisted by a multimodal large model, characterized in that it comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the noise-robust unsupervised representation learning method for a human skeleton assisted by a multimodal large model of the present invention.

[0038] The third aspect of the present invention relates to a computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, it implements the noise-robust human skeleton unsupervised representation learning method assisted by a multimodal large model of the present invention.

[0039] The working principle of the present invention is: directly inputting the video frames corresponding to the human skeleton sequence into the multimodal large model, generating a text description for each human skeleton action, and learning the similarity between the two in a unified feature space based on the idea of ​​contrastive learning in representation learning. At the same time, according to the characteristics that noise samples usually have a smaller cosine similarity and a larger information entropy with other samples in the text feature space and the unified feature space, noise samples are screened out and the similarity of noise samples is reduced, thereby improving the robustness of the model.

[0040] The advantages of the present invention are as follows: 1. The present invention generates an action text description for each human skeleton sample by corresponding to the RGB video modality, and performs an unsupervised representation learning method for the human skeleton, thereby getting rid of the reliance of existing methods on labels when constructing text supervision signals, and providing rich, diverse and powerful supervision signals for the training of the human skeleton encoder. 2. The present invention proposes a noise evaluation method based on the joint estimation of abnormal samples and information entropy, which screens out the noise text generated by the multimodal large model, reduces the target probability of its representation learning, and further improves the quality of the feature representation obtained by the model after learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required to be used in the embodiments are briefly introduced below.

[0042] Figures 1(a) and 1(b) are schematic diagrams of the overall structure of the method of the present invention. Figure 1(a) is a structural diagram of a noise-robust unsupervised representation learning method for human skeleton assisted by a multimodal large model, and Figure 1(b) is a structural diagram of a downstream task of action recognition.

[0043] Figure 2 (a) and Figure 2 (b) are comparison diagrams of the downstream task performance of the unsupervised representation learning method HiCo (Hierarchical Contrast for Unsupervised Skeleton-based Action Representation Learning) before and after the application of the method of the present invention, where Figure 2 (a) is the performance of the model on the downstream task of action recognition, and Figure 2 (b) is the performance of the model on the downstream task of action retrieval.

[0044] Figure 3 (a) and Figure 3 (b) are comparison diagrams of the downstream task performance of the unsupervised representation learning method UmURL (Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding) before and after the application of the method of the present invention, where Figure 3 (a) is the performance of the model on the downstream task of action recognition, and Figure 3 (b) is the performance of the model on the downstream task of action retrieval.

[0045] Figure 4 It is a structural schematic diagram of the implementation process of the method of the present invention.

[0046] Figure 5 It is a structural diagram of the device of the present invention. DETAILED DESCRIPTION

[0047] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0048] Example 1

[0049] As shown in FIG. 1( a ) and FIG. 1( b ), this embodiment provides a noise-robust human skeleton unsupervised representation learning method assisted by a multimodal large model, comprising the following steps:

[0050] Step S1, extracting through the RGB video corresponding to the human skeleton sequence to obtain the original action text description corresponding to the human skeleton sequence, specifically includes the following steps.

[0051] Step S11, during the data set collection process, the RGB video and the human key point information are usually collected simultaneously to obtain an RGB video sequence and a corresponding original human skeleton sequence. The G frame image obtained by interval sampling the RGB video sequence is used as input.

[0052] Step S12, using the multimodal large model LLaVA, taking "What is (are) he / she (they) trying to do?" as the text input instruction, and taking the G frame image obtained by sampling the RGB video corresponding to each of the above human skeleton sequences as the visual input of LLaVA, generating the original action text description for each frame sequence of the RGB video.

[0053] Step S2: First, prepare the original human skeleton sequence for training. After preprocessing and data enhancement, a human skeleton sequence with three-dimensional joint modalities (hereinafter referred to as human skeleton sequence) is obtained, which is recorded as (Another enhanced sequence is denoted as ), where the three dimensions represent the number of frames, the number of joints, and the number of sample channels of the action respectively. The original action text description generated in step S12 is subjected to string preprocessing and collection to obtain a vector of the action text description, which is recorded as , G represents the number of extracted key frames, that is, the action text description corresponding to G different frames.

[0054] Step S3, respectively encode the human skeleton sequence and the corresponding action text description obtained in step S2, obtain their respective features and use the idea of ​​contrastive learning to perform unsupervised representation learning, which specifically includes the following steps.

[0055] Step S31: In order to stably encode the human skeleton sequence and facilitate the subsequent unsupervised representation learning between skeletons and texts, the human skeleton sequence input is first mapped to Specifically, this module is implemented by a multilayer perceptron (MLP) consisting of two fully connected layers (FC Layer), and layer normalization (LN) is introduced to make the training more stable.

[0056] (3)

[0057] Among them, FC is the fully connected layer, LN is the layer normalization, is the Rectified Linear Unit (ReLU) activation function. The representation of the entire human skeleton sequence after embedding is recorded as (Another human skeleton enhancement sequence is recorded as ), N represents the batch size, and L represents the length of the skeleton sequence after embedding.

[0058] Step S32, after mapping the skeleton modality input to the same dimensional embedding space, the skeleton and text modalities need to be encoded using the skeleton and text encoders respectively to obtain the feature representations of the skeleton and text modalities. Since the encoding process and the encoder architecture are not the main content of the present invention, the encoding process is simply expressed as follows:

[0059] (4)

[0060] (5)

[0061] in, (Another human skeleton enhancement sequence is recorded as ) represents the final feature representation of the human skeleton sequence after being encoded by the encoder. , represents the final feature representation of the G action text descriptions obtained after encoding by the text encoder for each human skeleton sample. is a sequence-to-sequence encoder that can be learned during the pre-training phase, and It represents a pre-trained text encoder whose parameters are completely frozen during the whole pre-training process. In the method of the present invention, a CLIP text encoder with a specification of "ViT-L / 14@336px" is used.

[0062] Step S33, in order to utilize the multimodal large model knowledge to assist the unsupervised representation learning of the human skeleton, the representation of the projection layer skeleton and the text is first projected and normalized.

[0063] (6)

[0064] (7)

[0065] in, (The final feature representation of another human skeleton enhanced sequence is recorded as )and They represent the final feature representation of the human skeleton sequence after being projected into the common space and the final feature representation of the corresponding action text description, and for the skeleton feature projection, A multi-layer perceptron is used for projection, and for the text feature projection layer , only a single linear layer is used to implement it. To facilitate subsequent description, the batch of action text descriptions is ignored. is the G action text description corresponding to the first human skeleton sample feature in S.

[0066] The idea of ​​contrastive learning in unsupervised representation learning is to learn the similarity representation between the current sample and the corresponding sample (hereinafter referred to as the positive sample) and the dissimilar representation between the current sample and other samples in the batch (hereinafter referred to as the negative sample). The present invention uses Normalized Temperature-Scaled Cross-Entropy (NT-Xent) as the optimization loss function to perform representation learning on the model, and its definition is as follows:

[0067] (8)

[0068] in, and Represents the corresponding positive sample pair, and other samples in the batch are regarded as negative sample pairs. Indicates the calculation of cosine similarity, is a 0-1 indicator. The time value is 0, which is used to exclude sample self-similarity. is the temperature coefficient, and in the experiment of the present invention, the value is 0.2. From the above formula (8), it can be seen that the maximum positive sample pair and When the similarity of is expressed, the loss can be minimized.

[0069] For the convenience of subsequent description, the learning objectives of the model training of the present invention can be as follows:

[0070] (9)

[0071] (10)

[0072] (11)

[0073] In the above formula, represents the inter-skeleton representation learning loss function, represents the loss function of the unsupervised representation learning of the human skeleton assisted by the multimodal large model proposed in the present invention, is the overall loss function. represents the cross entropy function, represents the standard normalized exponential function (Softmax function), Represents the original learning goal of skeleton-text contrastive learning, Represents the skeleton-skeleton comparison learning objective, which is a matrix composed of 0 and 1, with positive samples being 1 and negative samples being 0. represents the transpose operation on a vector. is the temperature coefficient. The larger its value is, the more "confident" the model should be about the learning goal. Conversely, the more uncertain the learning strategy should be.

[0074] Step S4: The action text description generated by the multimodal large model is not completely correct. Among the G corresponding action text descriptions generated by a skeleton sample, there are erroneous action text descriptions, i.e., the first type of text noise mentioned above. The abnormal action text descriptions are screened out based on abnormal sample recognition and applied to the comparative learning in step S3, which specifically includes the following steps.

[0075] Step S41, since the text encoder has been pre-trained on a large amount of text data, it has a strong ability to distinguish text descriptions expressing different semantics. Specifically, the wrong action text description has a smaller cosine similarity with other text descriptions. To this end, the present invention proposes a noise evaluation method based on abnormal sample recognition, which first needs to calculate the above distance:

[0076] (12)

[0077] (13)

[0078] in, Contains G action text descriptions corresponding to the first human skeleton sample feature, represents the cosine similarity between the G action text descriptions corresponding to the first skeleton sample, represents the transpose operation on a vector. Used to mask self-similarity and negative sample similarity values, Represents the average cosine similarity between each action description text and the other G-1 action description texts.

[0079] Step S42: In the vector space, the smaller the cosine similarity between two vectors, the larger the vector distance between them. Therefore, a threshold is determined, and the text with an average cosine similarity less than the threshold is considered to be an incorrect action text description. It is necessary to reduce its similarity with the corresponding skeleton during the pre-training process. The mean and standard deviation are usually important indicators for measuring the distribution of a group of numbers, so they are used to generate the threshold. , and generate new learning objectives .

[0080] (14)

[0081] (15)

[0082] in, Represents the similarity threshold of the G action text descriptions corresponding to the first human skeleton sample, which is used to determine whether the current text description is a noise sample. and Used to calculate The mean and standard deviation of the second dimension, is the offset coefficient, which is set to 3.0 in the experiment.

[0083] Step S43, the obtained learning objective and the original learning objective are linearly weighted to generate a new optimization objective, where: is a hyperparameter with a value of 0.3. The corrected learning objective As shown below,

[0084] (16)

[0085] The new targets generated by this method can give lower learning target probabilities to some samples that are irrelevant to the current skeleton action semantics, thereby reducing the impact of noisy action text descriptions generated during unsupervised representation learning on model learning.

[0086] Step S5, as mentioned above, the text generated by the multimodal large model contains not only erroneous action text descriptions, but also indistinguishable action text descriptions, which have high similarity not only with the corresponding human skeleton samples, but also with other samples, and can be well detected by information entropy. This method proposes to screen out the above action text descriptions based on information entropy estimation and use them for comparative learning in step S3. and Except for the different data enhancement before encoding, the rest are the same, and the noise evaluation method based on information entropy estimation is also applicable. For the convenience of expression, The following steps are specifically included for explanation using an example.

[0087] Step S51, by calculating the cosine similarity between the current action description text and all skeletons in the batch, the probability of each action text description feature and other human skeleton sample features in the batch is generated.

[0088] (17)

[0089] in, Represents the final feature representation of the human skeleton modality with a batch size of N, for The final feature representation of the action text description corresponding to the first human skeleton sample feature. represents the transpose operation on a vector. represents the standard normalized exponential function (Softmax function), Represents the similarity probability between the final representation of each action text description and the final representation of the human skeleton in the batch.

[0090] Step S52, the information entropy of the action description text is obtained by the definition of the above information entropy. After normalization, its value range is [0,1], which is used to measure whether the distribution between the current text and all skeletons in the batch is deterministic. If the current action description text representation is similar to the skeleton sample representation in the batch, the information entropy is close to 1. If it has a large similarity with a specific skeleton sample representation and a small similarity with others, the value of the information entropy will be close to 0. Its calculation is generated by the following formula:

[0091] (18)

[0092] in, It represents the information entropy between the text description of each action corresponding to the first skeleton sample and the skeleton samples in the batch, and N represents the batch size of the human skeleton sequence.

[0093] Step S53: Since the larger the information entropy, the higher the similarity between the action text description and all skeleton samples in the batch, the less the information it contains, the more information is used. Generate skeleton-text learning objectives based on information entropy estimation noise assessment method .

[0094] (19)

[0095] Step S6, merge the obtained noise assessment method learning objective based on abnormal sample recognition with the noise recognition method learning objective based on information entropy estimation, and optimize the encoding space of the human skeleton encoder. The specific steps are as follows:

[0096] Step S61, through step S4 and step S5, the noise sample evaluation method based on abnormal sample detection and information entropy estimation proposed by the present invention can use the text encoder and the skeleton encoder to filter out noisy text action descriptions, and provide a more effective and meaningful learning goal for the learning of the human skeleton encoder, thereby enhancing the reliability and robustness of the human skeleton encoder. The above labels are collected, and the learning goal after denoising and optimization by the method of the present invention can be obtained as follows:

[0097] (20)

[0098] (twenty one)

[0099] in, , is a hyperparameter. In the present invention, is 0.3, It is 0 at the beginning of training and gradually increases to 0.3 during training. The growth process is shown in the above formula. Epoch is the current training round. The total training round in the present invention is 450. is the original learning target. The text descriptions corresponding to the human skeleton samples are all set to 1, and the other samples are all set to 0. is the denoising learning target generated by the abnormal sample recognition noise evaluation method proposed in the present invention, It is a denoising learning target generated by the noise evaluation method based on information entropy estimation proposed by the present invention. Represents the new learning objective after the sample noise evaluation method is jointly estimated by abnormal samples and information entropy.

[0100] Step S62, combining the above information noise estimation algorithm idea, obtains the unsupervised representation learning optimization goal of the noise-robust human skeleton sequence encoder assisted by the final multimodal large model as follows:

[0101] (twenty two)

[0102] (twenty three)

[0103] in, Represents the final loss function of the present invention, It is a hyperparameter and its value is 0.6. Represents the learning objectives generated by the noise evaluation method based on abnormal sample identification and the noise evaluation method based on information entropy estimation. Noise-robust learning loss function assisted by multimodal large models.

[0104] Step S7, after pre-training is completed, the parameters of the embedding layer and the human skeleton encoder are completely frozen for a series of downstream tasks, such as adding and fine-tuning a linear classifier to achieve human skeleton action classification, action retrieval for input queries, and other downstream tasks.

[0105] Tables 1 and 2 show the performance comparison of the proposed method on action classification and action retrieval tasks. The experiment used public datasets commonly used in the field of human skeleton unsupervised representation learning: NTU RGB+D 60, NTU RGB+D 120 and PKUMMDⅡ. The table shows the Top-1 accuracy of the model in downstream tasks of action recognition and action retrieval after noise-robust unsupervised representation learning using the proposed method.

[0106] The NTU RGB+D 60 dataset contains 56,880 action samples recorded by 40 subjects, with a total of 60 types of actions. The dataset is divided into two modes: cross subject (x-sub): half of the subjects’ data is used for training and the rest for testing; cross view (x-view): samples collected by cameras 2 and 3 are used as training data, and samples collected by camera 1 are used as test data.

[0107] NTU RGB+D 120 is an extended version of NTU RGB+D 60, with 120 categories and 113945 samples, recorded by 106 subjects in 32 settings (different background and camera positions). Cross-subject is similar to NTU RGB+D 60; cross setup (x-setup): samples collected under even-numbered setup numbers are used as training data, and samples collected under odd-numbered setup numbers are used as test data.

[0108] PKU-MMD is commonly used for skeleton-based action recognition. PKU-MMD II contains 7,000 action samples for 51 categories of actions, each recorded from one or two subjects.

[0109] Human skeleton action recognition is the main downstream task in human skeleton unsupervised representation learning. The pre-trained embedding layer and human skeleton encoder parameters of step S1-step S6 are frozen, and a linear classifier (single-layer fully connected layer) is added thereafter for fine-tuning to evaluate the effect of model representation learning. As shown in Table 1, the noise-robust unsupervised representation learning method assisted by a multimodal large model proposed in the present invention is significantly better than the existing unsupervised representation learning method, indicating that the introduction of text modality in skeleton representation learning and the noise-robust unsupervised representation learning between the human skeleton sequence and the action text description make good use of the multimodal large model's ability to understand actions to assist the human skeleton encoder in unsupervised representation learning, so that the model obtains more beneficial supervised learning signals in the representation learning stage.

[0110] Table 1 Comparison of skeleton action recognition performance with other methods on multiple datasets, where NTU 60, NTU 120 and PKU II are NTU RGB+D 60, NTU RGB+D 120 and PKU-MMD II respectively

[0111]

[0112] The action retrieval task is to find the most similar action sample by calculating the cosine similarity in the sample feature space given a skeleton as a query, and evaluate its retrieval accuracy. This downstream task does not require additional training and can more intuitively reflect the quality of the model's representation of the sample. As can be seen from Table 2, the performance of this method in retrieval exceeds that of existing methods, further demonstrating that after the multimodal large model assists in representation learning, the feature representation learned by the model is more discriminative.

[0113] In addition, we selected the current open source human skeleton unsupervised representation learning project, reproduced the results of the paper, and added the human skeleton unsupervised representation learning strategy assisted by the multimodal large model proposed in this invention without changing the original skeleton learning strategy, and evaluated it on the downstream tasks of action recognition and action retrieval. The results are shown in Figure 2 (a) and Figure 2 (b) and Figure 3 (a) and Figure 3 (b), where the items with "*" represent the results after adding the method of the present invention. The results in the figure show that this method is applicable to the existing human skeleton unsupervised representation learning method, and provides a universally effective text supervision signal for the human skeleton encoder through the noise-robust action text description provided by the multimodal large model, optimizes the feature space of the human skeleton encoder, and has universal adaptability and effectiveness.

[0114] Table 2 Comparison of skeleton action retrieval performance with other methods on NTU RGB+D 60 and NTU RGB+D 120 datasets

[0115]

[0116] Example 2

[0117] like Figure 5 This embodiment provides a noise-robust unsupervised representation learning device for a human skeleton assisted by a multimodal large model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the noise-robust unsupervised representation learning method for a human skeleton assisted by a multimodal large model of Example 1.

[0118] Example 3

[0119] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the noise-robust unsupervised representation learning method of a human skeleton assisted by a large multimodal model of embodiment 1 is implemented.

[0120] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A noise-robust unsupervised representation learning method for human skeleton assisted by a multimodal large model, characterized in that: The following steps are involved: Step S1, data acquisition; the original human skeleton sequence is directly provided by the human skeleton sequence public dataset; the RGB video corresponding to the human skeleton sequence is extracted, and the corresponding original action text description is obtained using a multimodal large model; Step S2, data preprocessing: preprocessing the original human skeleton sequence and the corresponding original action text description to obtain a joint point modal human skeleton sequence and a corresponding action text description sequence, and performing homologous data enhancement on the joint point modal human skeleton sequence to obtain two different enhanced human skeleton sequences; Step S3, unsupervised representation learning; for the joint point modal human skeleton sequence, use the embedding layer and the human skeleton encoder to embed and encode to obtain the human skeleton features; for the corresponding action text description sequence, use the pre-trained CLIP model to encode to obtain the action text description features; project the two into a unified feature space, and use contrastive learning to align the features between different enhanced sequences of the human skeleton, the human skeleton sequence features and the action text description features; Step S4, using a method based on abnormal sample identification to perform noise assessment and filter out "outlier" action text descriptions; Step S5, using a method based on information entropy estimation to perform noise assessment, reducing the target probability of non-distinguishing action text descriptions in the representation learning process; specifically including: Step S51, performing noise assessment based on information entropy estimation; wherein the definition of information entropy is as follows: According to the above formula, for each action text description feature, the similarity score is calculated with all the human skeleton sample features in the batch in turn, and the probability of each action text description feature and each human skeleton sample feature in the batch is obtained through the standard normalized exponential function. ; Step S52: Calculate the information entropy of each action text description according to formula (2): ; Step S53: The greater the information entropy, the less information the generated action text description contains, and thus a learning target based on information entropy estimation is generated by negating the information entropy; Step S6, linearly weighted fusion of the learning target generated by the sample noise evaluation method in step S4 and step S5 and the original learning target, and applying them to the comparative learning in step S3 to learn the similarity representation between the human skeleton sequence features and the action text description features; Step S7, evaluate the model capability and completely freeze the pre-trained human skeleton sequence encoder and embedding layer parameters for downstream tasks.

2. According to claim 1, a noise-robust human skeleton unsupervised representation learning method assisted by a multimodal large model is characterized in that: Step S1 specifically includes the following contents: Step S11, using interval sampling method, sampling the key frames of the RGB video corresponding to the human skeleton sequence to obtain Frame keyframe; Step S12, using fixed instructions, together with the key frames extracted in the previous step, input into the multimodal large model LLaVA to generate the original action text description corresponding to the original human skeleton sequence. For each original human skeleton sequence, generate A text description of the original action.

3. The method for learning a noise-robust human skeleton unsupervised representation assisted by a multimodal large model according to claim 1, characterized in that: Step S3 specifically includes: Step S31, projecting different enhancements of the human skeleton sequence to the feature space where the human skeleton sequence encoder is located through a multi-layer perception MLP including layer normalization LN; Step S32, using a human skeleton sequence encoder and a pre-trained CLIP text encoder to encode the projected human skeleton sequence and the original action text description respectively, to obtain human skeleton sequence features and action text description features; In step S33, the features obtained in step S32 are projected into a unified feature space respectively to obtain a final representation of the human skeleton sequence features and a final representation of the corresponding action text description features, and the similarity between the corresponding skeleton-skeleton enhancements and the similarity between the skeleton-text sample pair features are maximized by comparative learning.

4. The method for learning a noise-robust human skeleton unsupervised representation assisted by a multimodal large model according to claim 1, characterized in that: Step S4 specifically includes: Step S41, perform noise assessment based on abnormal sample identification; for each action text description feature corresponding to each human skeleton feature, calculate and compare it with other features. The average cosine similarity between the action text description features is obtained The average distance between the action text description and other action text descriptions; step S42, calculate The mean and standard deviation of the average cosine similarity of the action text description features are calculated, and a threshold is set to filter out incorrect action text descriptions that have a small similarity with other samples; Step S43, removing the erroneous action text descriptions screened out in step S42, and generating an abnormal sample denoising learning target.

5. The method for learning human skeleton representations robust to noise under the assistance of a multimodal large model according to claim 1, characterized in that: Step S6 specifically includes the following contents: Step S61, in order to apply the learning objectives generated in step S4 and step S5 to step S3 to improve the quality of skeleton-text comparison learning; specifically, the learning objectives generated by the noise evaluation method based on abnormal sample recognition and the learning objectives generated by the noise evaluation method based on information entropy estimation obtained in step S4 and step S5 are fused with the original learning objectives through a linear weighted fusion method to correct the original learning objectives; Step S62, referring to the idea of ​​comparative learning, learn the similarity representation of human skeleton sample features and corresponding action text description features, and train the human skeleton sequence encoder end-to-end.

6. A noise-robust human skeleton unsupervised representation learning device assisted by a multimodal large model, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the noise-robust human skeleton unsupervised representation learning method assisted by a multimodal large model as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the noise-robust unsupervised representation learning method for human skeleton assisted by a multimodal large model as described in any one of claims 1-5 is implemented.

Images