A three-dimensional human body behavior recognition method under a small sample condition

Abstract

The application discloses a three-dimensional human behavior recognition method under a small sample condition, and belongs to the cross technical field of digital image processing and machine learning. The application utilizes the statistical distribution information of base class feature vectors and the cosine similarity of new class sample data in a feature space to guide the feature distribution calibration of new classes and base classes; utilizes multi-dimensional Gaussian sampling to sample the new class samples after distribution calibration, and solves the overfitting problem caused by the small training samples of the small sample three-dimensional human behavior recognition; the application utilizes short fragment small voxel 3DV to extract effective and more behavior fine-grained information; meanwhile, visual Transformer is introduced, global receptive field and attention mechanism thereof are utilized for time sequence feature fusion, the network can pay attention to the human behavior fragments with the most discriminability, invalid behavior fragment interference is reduced, the feature extraction capability of the original three-dimensional dynamic voxel algorithm is further improved, and the distinguishing capability of the network for similar actions is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the cross technical field of digital image processing and machine learning, and more particularly relates to a three-dimensional human body behavior recognition method under a small sample condition. BACKGROUND

[0002] Three-dimensional human body behavior recognition based on depth video information is a frontier topic in the field of computer vision and pattern recognition, and has important scientific theoretical significance and application value in the fields of video monitoring, human-computer interaction and intelligent medical treatment. With the popularization of depth cameras, the difficulty of obtaining three-dimensional data is gradually reduced, and with the introduction of deep learning technology, the performance of three-dimensional human body behavior recognition has been greatly improved.

[0003] However, the existing mainstream "big data + deep learning" theoretical paradigm faces the limitation that when the number of training samples is insufficient, the recognition and generalization performance of the deep learning network is often difficult to effectively guarantee. However, in many real-world applications (such as fall / bed fall detection and dangerous behavior detection), effective training samples are often difficult to obtain, resulting in a serious lack of quantity and diversity. Humans are very good at recognizing a new class of behaviors from a small number of samples, but existing behavior recognition models do not have this ability.

[0004] On the one hand, when the number of data sets is too small, deep learning cannot effectively extract common features between classes, but tends to simulate some unique properties in the training set, which also limits the classification accuracy of the network for samples not seen in the test set, resulting in overfitting problems.

[0005] Current research on small sample three-dimensional human body behavior recognition is still in its infancy, and existing methods mainly extract features from human three-dimensional skeleton sequence information through graph convolution network GCN. Through theoretical means such as measure learning or feature selection, the discriminative ability of the extracted features is enhanced, and finally the class to which the test set sample belongs is distinguished in the measure space through Euclidean distance and other measurement methods. However, the existing measure learning method cannot well distinguish similar actions in new classes. SUMMARY

[0006] In view of the above defects or improvement needs of the prior art, the present application provides a three-dimensional human body behavior recognition method under a small sample condition, which aims to improve the overfitting and the problem of being unable to well distinguish similar actions in the three-dimensional human body behavior recognition under a small sample condition.

[0007] To achieve the above purpose, the present application provides a three-dimensional human body behavior recognition method under a small sample condition, comprising:

[0008] S1. Extracting a plurality of small-scale short segment three-dimensional dynamic voxels containing local timing features from the behavior video;

[0009] S2. Training a human behavior feature extraction network using the three-dimensional dynamic voxels as a training set; wherein the human behavior feature extraction network comprises: a PointNet++ feature extraction network, a Transformer encoder, and a multi-layer perception classifier; the PointNet++ feature extraction network is used to encode the plurality of short segment three-dimensional dynamic voxels into feature vectors respectively; the Transformer encoder is used to calculate the similarity between the learnable class features and each feature vector, and to perform weighted summation on each short segment according to the similarity to obtain further encoded behavior features; the multi-layer perception classifier is used to output behavior classification results according to the further encoded behavior features;

[0010] S3. Using the trained feature extraction network to extract feature vectors of all samples in the base class and the new class, and calculating the statistical distribution information of each class in the base class; and using the statistical distribution information of the base class feature vectors and the cosine similarity of the new class sample data in the feature space to guide the feature distribution calibration between the new class and the base class;

[0011] S4. Using multi-dimensional Gaussian sampling to sample each new class using the calibrated mean and covariance matrix as parameters, and using the sampled feature vectors to train a classification network to obtain a small sample classifier;

[0012] S5. Using the trained feature extraction network to extract the feature vector of the test sample, inputting the feature vector into the small sample classifier, and outputting the new class category to which the test sample belongs.

[0013] Further, step S1 specifically comprises:

[0014] Voxelizing the behavior video frame by frame by setting a voxel size of 20mm*20mm*20mm or less;

[0015] Densely sampling the voxelized behavior video to a fixed length, segmenting the densely sampled video using a fixed window size and step, and extracting three-dimensional dynamic voxels from each video segment to obtain a plurality of short segment three-dimensional dynamic voxels containing local timing features.

[0016] Further, the voxel size is set to 15mm*15mm*15mm.

[0017] Further, step S3 specifically comprises:

[0018] Calculating the mean and covariance matrix of the feature vectors of all training samples in each class in the base class:

[0019]

[0020] Calculate the cosine similarity between the features of the new class support set samples and all base class feature centers;

[0021] After arranging from large to small, discard the dissimilar part with similarity lower than the set value, select the class with the highest similarity in the remaining high similarity part to join the base class standard distribution set S N ;

[0022] For similarity from high to low, if the ratio of the similarity of adjacent classes is greater than the set peak side ratio threshold, the adjacent classes are also added to the base class standard distribution set;

[0023] Add the k-means subclass clustering center with similarity higher than the lowest similarity in S N to the subclass standard distribution set S M , and ensure that the class subclass center in S N does not appear in S M ;

[0024] Use the base class standard distribution set S N and the subclass standard distribution set S M to calibrate the mean of the new class distribution and estimate the covariance matrix of the new class distribution, and the distribution center and covariance matrix of the new class after calibration are:

[0025]

[0026] Further, before calculating the cosine similarity between the features of the new class support set samples and all base class feature centers, the method further comprises:

[0027] Using the k-means clustering method, each base class is clustered into k subclasses, and the feature center μ ik of all subclasses is calculated.

[0028] Further, the small sample classifier in S5 uses a multi-layer perception MLP or a linear classifier SVM.

[0029] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects.

[0030] The present application effectively retains more behavioral granularity information by using short fragment small voxel 3DV extraction, and introduces visual Transformer, uses its global receptive field and attention mechanism for temporal feature fusion, so that the network can pay attention to the most discriminative human behavior segment, reduces the interference of invalid behavior segments, further improves the feature extraction capability of the original three-dimensional dynamic voxel algorithm, and improves the network's ability to distinguish similar actions.

[0031] The application uses the design of a generative model on a small sample classification model, establishes the connection between the base class data distribution and the new class data distribution by introducing a distribution calibration algorithm, migrates the base class data distribution rule to the new class data, and enhances the robustness and performance of the small sample classification network through Gaussian sampling in the feature space.

[0032] The application uses cosine distance, k-means center and adaptive threshold mining to alleviate the parameter sensitivity problem of distribution calibration, and effectively improves the performance of the distribution calibration algorithm in the field of small sample three-dimensional human behavior recognition. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a three-dimensional human behavior recognition method flowchart under the condition of small samples provided by the application;

[0034] Figure 2 is the global and local dynamic voxel of the 'quiet' and 'yawn' behavior;

[0035] Figure 3 is a structure diagram of a time sequence feature extraction network based on a transformer;

[0036] Figure 4 is a schematic diagram of a new class and base class center distribution calibration method;

[0037] Figure 5 is a comparison diagram of the distribution sampled by the new class after distribution calibration and the real distribution. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the application clearer and more apparent, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0039] Compared with traditional three-dimensional human behavior recognition, the difficulty brought by the addition of small sample definition mainly lies in: when a model is trained with only a small amount of training data, the model will often overfit these samples by minimizing the training loss of these samples. This biased distribution based on a small number of examples can impair the generalization ability of the model, because it cannot reflect the basic fact distribution of the test case sampling during evaluation. Therefore, how to reduce the biased distribution of the training sample and improve the stability of the training is a problem to be solved by the application.

[0040] For the distinction between similar actions, such as the actions "rubbing hands" and "clapping", the difference between the two actions is limited to a small area of the hand, and the posture of most of the body is fixed and consistent. In addition, considering the overall action process, both actions involve lifting the hands from the side of the body until they converge in front of the chest. Similar action processes also make the classification of actions more difficult. In view of the above characteristics of the database, the inter-class difference of human behavior is small, and many behaviors are very similar in form, with similar movement patterns in some parts, which can cause obvious inter-class overlap in some classifications during classification, reduce the discrimination of the features, and thus affect the classification ability of small samples. However, the existing mainstream feature description methods generally have poor description ability for fine-grained behaviors, and therefore, how to enhance the feature coding ability for fine-grained behaviors is a problem that needs to be studied by the present application.

[0041] Based on the above problems, the present application adopts a two-stage training mode, first trains a reliable feature extractor based on the three-dimensional dynamic voxel algorithm using complete base class data, and then projects all new class samples into the feature space using the feature extractor. After generating a new feature vector for small sample classification by "distribution calibration" on the support set samples, a simple multilayer perceptron is trained for classification and recognition in the feature space.

[0042] Reference Figure 1 The present application provides a three-dimensional human behavior recognition method under small sample conditions, comprising the following steps:

[0043] S1. Extracting a plurality of small-scale short segment three-dimensional dynamic voxels containing local temporal features from the behavior video;

[0044] Specifically, step S1 includes voxelizing the behavior video frame by frame by setting the voxel size; wherein the voxel size set by the present application is 20mm*20mm*20mm or less; densely sampling the voxelized behavior video to a fixed length, segmenting the densely sampled video using a fixed window size and step, and extracting three-dimensional dynamic voxels from each video segment to obtain a plurality of small-scale short segment three-dimensional dynamic voxels containing local temporal features;

[0045] The three-dimensional dynamic voxel (3DV) algorithm mainly consists of two parts: three-dimensional human behavior voxelization expression and human behavior feature extraction network. In the human behavior voxelization expression stage, 3DV uses a time series pooling algorithm to compress the time and space information contained in the entire depth video into a single voxel set representation. Then in the feature extraction part, the PointNet++ point cloud-based feature extraction method is used to further encode the motion information of the human body.

[0046] The classification failure in the 3DV method is studied, and it is found that when there is only a small difference between the overall three-dimensional dynamic voxels of different actions, the 3DV algorithm is difficult to correctly classify such different actions. For example, between the actions of "hand rubbing" and "clapping", the overall action process is similar, and only the movement mode of the hand in the late action is different. For example, the "quiet" behavior, the action subject is only a finger, and the action amplitude is also limited in the small space of the mouth. However, since the 3DV extracts the global dynamic code of the entire action, it cannot highlight the difference of the action in the small time segment, and the difficulty of classifying similar actions is increased.

[0047] The reason is found that in the original point cloud space, the details of the behavior can be accurately observed, but in the process of generating three-dimensional dynamic voxels, the voxelization operation causes part of the details in the space to be erased. The voxel size used in the original 3DV is 35mmx35mmx35mm, which also causes the network to be unable to further model the details when the size of the detailed structure is less than 35mm, and to distinguish the differences in the details. In view of the above shortcomings, the present application uses a smaller voxel size to help the network capture more detailed information, and preferably uses a voxel size of 15mmx15mmx15mm to facilitate fine modeling of detailed parts such as hands.

[0048] In addition to the fine modeling in space, finer time segments are also important for the classification of fine-grained actions. The modeling process of the original 3DV pays more attention to the long-term process of the movement from the beginning to the end, but the short-term segment in the action process can provide more detailed information to help the network distinguish the differences between fine-grained actions. Figure 2 For example, as shown in the "quiet" and "yawn" actions, if the whole process of the action from the beginning to the end is focused on, both actions start from a standing posture, the action subject moves the hand from the body side to the mouth, and stays in the mouth. In the time scale of the entire action sequence, it is difficult to distinguish whether the action subject's hand gesture is to raise the index finger or to open the palm Figure 2 (the left half of the middle part); but if only the short action segment after the hand stays in the mouth is focused on, the hand action details can provide fine movement information with higher recognition Figure 2 (the right half of the middle part). Therefore, the present application uses the short-term segment in the action process to provide more detailed information to help the network distinguish the differences between fine-grained actions.

[0049] S2. Train the human behavior feature extraction network using the three-dimensional dynamic voxels as the training set;

[0050] The human behavior feature extraction network constructed by the application can learn which behavior fragments play a more important role in feature extraction by introducing a timing self-attention mechanism based on the original 3DV network structure, whether the whole action or the starting or ending fragment should be paid more attention to, and then extract the action fragment most conducive to behavior classification, while reducing the influence of low-differentiation dynamic voxels of similar action process fragments between different behavior categories on classification accuracy. The features of different time fragments are modeled in a deeper level by using the Transformer structure. The overall network structure is as shown in Figure 3 As shown in the network structure design, a shared parameter PointNet++ skeleton network is used to extract the features of multiple short-time dynamic voxels, and the feature vector of each short-time dynamic voxel fragment can be regarded as containing the main motion information of the corresponding time fragment. After obtaining the feature map composed of all feature vectors, a set of category classification features is first spliced on the feature map to fuse the feature encodings of all time fragments and express the most effective part for overall classification, then a set of learnable position encodings is added to emphasize the position of the current fragment in the overall video, and finally the feature map with completed position encoding and classification feature splicing is input into the multi-layer Transformer self-attention encoder.

[0051] During the training of the feature extraction network, only base class sample data can be used for training, a cross-entropy loss function is used, an Adam optimizer is used for optimization, the learning rate is set to 0.001, the learning rate is reduced to one-fourth of the original every 5 epochs, and the network is trained for 20 epochs to converge.

[0052] S3. Use the trained feature extraction network to extract the feature vectors of all samples in the base class and the new class, and calculate the statistical distribution information of each category in the base class; and use the statistical distribution information of the base class feature vectors and the cosine similarity of the new class sample data in the feature space to guide the feature distribution calibration of the new class and the base class.

[0053] The small sample three-dimensional human behavior problem researched by the application is proposed at the same time as the NTU RGB+D 120 (NTU120) dataset, which includes 120 different action categories, a total of 114480 video samples from 106 different actors captured from 155 different depth camera perspectives, and is the largest dataset for three-dimensional human behavior recognition problems

[0054] The small sample three-dimensional human behavior recognition task researched by the application divides the whole 120 classes of actions into two non-intersecting parts, one part is called "base class" and is composed of 100 classes of actions, all action samples in the base class can be used to train a stable feature extraction network; the other part is called "new class" and is composed of the remaining 20 classes of actions, only one sample is provided for each class to train a small sample behavior recognition classification network, the known sample is called "support set", and all the remaining samples in the new class form a "test set" to test the classification accuracy of the small sample classifier.

[0055] As described above, the difficulty brought by the addition of small sample limitation mainly lies in that when a model is trained only with a small amount of training data, the model will often overfit the few samples by minimizing the training loss of the samples.

[0056] To solve the problem, the application adopts the method of "distribution calibration" to perform data enhancement on the one sample provided by the new class, to generate a large number of samples similar to the one sample so that the training can be stably performed. Figure 4 It is assumed that the feature vector distribution of each class of samples conforms to a multi-dimensional Gaussian distribution, and that similar classes usually have similar distribution means and variances, so although the true mean and variance of the new class distribution cannot be obtained only with one sample, the relevant information of the known base class can be transferred to the new class to correct the biased distribution that may be caused by the single sample.

[0057] Specifically, the base class feature acquisition method is as follows: after each sample in the training set passes through the feature extraction network in step S2, the output of the network is a c-dimensional feature vector, which is obtained at the output of the BN layer of the classification head "linear layer-BN-RELU-linear layer", which is usually selected as 256-dimensional, and the mean and covariance matrix of all training sample feature vectors of each class in the training set are calculated:

[0058]

[0059] In the formula, μ i represents the feature mean of the i-th base class, Σ i represents the covariance matrix of the i-th base class, n i represents the number of samples in the i-th base class, x j represents the feature vector of the j-th sample in the i-th base class.

[0060] To avoid the case that when the similarity between the base class center and the new class is not enough, the low-similarity center will reduce the distribution accuracy of the new class after distribution calibration, the application expands clustering operation in each base class category, uses the k-means clustering method to cluster each base class into k sub-classes, and calculates the feature center μ ik of all sub-classes.

[0061] The specific method of the new class distribution calibration is as follows:

[0062] Calculate the cosine similarity between the features of the new class support set samples and all base class feature centers;

[0063] After sorting from large to small, discard the dissimilar part with a similarity lower than a set value (such as 0.5), and select the class with the highest similarity in the remaining high-similarity part to join the base class standard distribution set S N ;

[0064] If the similarity ratio of adjacent classes from high to low is greater than the set peak side ratio threshold, the adjacent classes are also added to the base class standard distribution set;

[0065] Add the k-means subclass clustering centers with a similarity higher than the lowest similarity in S N to the subclass standard distribution set S M , and ensure that the class subclass centers in S N do not appear in S M .

[0066] Next, use the base class standard distribution set S N and the subclass standard distribution set S M to calibrate the new class distribution. After calibration, the distribution center and covariance matrix of the new class are:

[0067]

[0068] In the formula, μ' represents the feature distribution mean of the calibrated new class, Σ' represents the covariance matrix of the calibrated new class, μ i represents the base class feature mean used for new class mean calibration, μ j represents the k-means subclass feature mean used for new class mean calibration, x represents the feature vector of the new class support set sample, |S N | and |S M | represent the number of elements in the two sets respectively, Σ i represents the base class covariance matrix used for new class covariance matrix calibration.

[0069] α, ω are hyperparameters, used for further fine-tuning of the calibrated new class distribution;

[0070] S4. Small sample classifier training: after the distribution of the small sample support set samples is calibrated, use multi-dimensional Gaussian sampling to sample each new class with the calibrated mean and covariance matrix as parameters, and use the sampled feature vectors to train the classification network;

[0071] The classification network uses a multi-layer perceptron (MLP) network containing two linear layers, the loss function is set to cross-entropy loss, the Adam optimizer, the learning rate is set to 0.001, and 5 epochs are trained to converge; other linear classifiers such as SVM can also be used for the small sample classifier.

[0072] S5. When testing the test set samples in the new class, first use the feature extraction network trained in the first stage to extract the feature vector of the test sample, input the feature vector into the small sample classifier trained in the second stage, and output the new class category to which the test sample belongs.

[0073] In this specific embodiment, the TSNE visualization of the new class distribution obtained by using the calibrated multi-dimensional Gaussian sampling and the true distribution of the new class test set is as shown in Figure 5 The distributions are basically the same, proving the effectiveness of the method.

[0074] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for three-dimensional human behavior recognition under small sample conditions, characterized in that, include: S1. Extract multiple small-scale short-segment 3D dynamic voxels containing local temporal features from behavioral videos; S2. Using three-dimensional dynamic voxels as the training set, train the human behavior feature extraction network; wherein, the human behavior feature extraction network includes: a PointNet++ feature extraction network, a Transformer encoder, and a multilayer perceptron classifier; the PointNet++ feature extraction network is used to encode multiple short-segment three-dimensional dynamic voxels into feature vectors respectively; the Transformer encoder is used to calculate the similarity between the learnable class features and each feature vector, and to perform a weighted summation of each short segment based on the similarity to obtain further encoded behavior features; the multilayer perceptron classifier is used to output behavior classification results based on the further encoded behavior features; S3. Use the trained feature extraction network to extract feature vectors of all samples in the base class and the new class, and calculate the statistical distribution information of each category in the base class; and use the statistical distribution information of the base class feature vectors and the cosine similarity of the new class sample data in the feature space to guide the feature distribution calibration of the new class and the base class. S4. Use multidimensional Gaussian sampling to sample each new class with the calibrated mean and covariance matrix as parameters, and use the sampled feature vectors to train the classification network to obtain a small sample classifier; S5. Use the trained feature extraction network to extract the feature vector of the test sample, input the feature vector into the few sample classifier, and output the new class to which the test sample belongs; Step S3 specifically includes: Calculate the mean and covariance matrix of the feature vectors of all training samples for each class in the base class: in, This represents the feature mean of the i-th base class. Let i represent the covariance matrix of the i-th base class. This represents the number of samples in the i-th base class. This represents the feature vector of sample j in the i-th base class; Calculate the central cosine similarity between the features of the new class support set samples and the features of all base classes; After sorting from largest to smallest, discard the dissimilar portions with similarity below a set value. From the remaining highly similar portions, select the class with the highest similarity and add it to the base class standard distribution set. middle; For cases where the similarity ratio between adjacent categories is greater than the set peak-side ratio threshold, the adjacent categories are also added to the base class standard distribution set. High similarity The lowest similarity k-means sub-cluster centers are added to the sub-class standard distribution set. In, and guarantee The category subclass center is not in Appeared in; Using the base class standard distribution set and the standard distribution set of subclasses The mean of the new class distribution is calibrated, and the covariance matrix of the new class distribution is estimated simultaneously. The distribution center of the new class after calibration is determined. Covariance Matrix for: in, Represents a set The Middle The feature mean of each base class; Represents a set The Middle k-means sub-cluster centers; Represents the feature vector of the new class support set samples; and Representing sets respectively and The number of elements in; Represents a set The Middle The covariance matrix of each base class; This is a hyperparameter.

2. The method for three-dimensional human behavior recognition under small sample conditions according to claim 1, characterized in that, Step S1 specifically includes: The behavioral video is voxelized frame by frame using a set voxel size; wherein the voxel size is less than 20mm*20mm*20mm. The voxelized behavioral video is densely sampled to a fixed length. The densely sampled video is then segmented using a fixed window size and stride. Three-dimensional dynamic voxels are extracted from each video segment to obtain multiple short-segment three-dimensional dynamic voxels containing local temporal features.

3. The method for three-dimensional human behavior recognition under small sample conditions according to claim 2, characterized in that, The voxel size is set to 15mm*15mm*15mm.

4. The method for three-dimensional human behavior recognition under small sample conditions according to claim 1, characterized in that, Before calculating the cosine similarity between the features of the new class support set samples and the feature centers of all base classes, the method further includes: The k-means clustering method is used to cluster each base class into k subclasses, and the feature centers of all subclasses are calculated. .

5. A method for three-dimensional human behavior recognition under small sample conditions according to any one of claims 1-4, characterized in that, In S5, the small sample classifier uses a multilayer perceptron (MLP) or a linear classifier (SVM).

6. A three-dimensional human behavior recognition system under small sample conditions, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the three-dimensional human behavior recognition method under small sample conditions as described in any one of claims 1 to 5.

Images