A few-shot action recognition method and system in a smart health environment

Abstract

The application discloses a few-sample action recognition method in a smart health environment, comprising: inputting a support set and a query set containing video samples, extracting representative frames from each video through a frame extraction algorithm, and performing data enhancement on the frames; using a feature extraction network to extract the frame set features of the samples; by calculating in the transformer, the support sample features matched with the query sample are constructed; according to the similarity between the sample features in the same class, a weight is assigned to each sample; combining the support sample features and their weights, the prototype of each class is constructed; for each pair of class prototype and query sample feature, the distance between the two is calculated, the class with the smallest distance is taken as the predicted result, and the model is optimized according to the distance loss. In addition, the application also discloses a few-sample action recognition system in a smart health environment. The application makes full use of various information of the samples, so that the generated action recognition result is more accurate and more in line with user expectations.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, action recognition, few-shot learning, and particularly relates to a few-shot action recognition method and system in a smart health environment. BACKGROUND

[0002] Whether to have the ability to learn and generalize from a small number of samples is a clear dividing line between artificial intelligence and human intelligence. Because humans can easily establish cognition of new things through one or a few examples, while machine learning algorithms usually need thousands of supervised samples to ensure their generalization ability. Few-shot learning (FSL) aims to identify new categories from a small number of labeled samples, which is of great significance and challenge in the field of machine learning. FSL methods based on meta-learning are representative in this field, and their main methods include matching networks, prototype networks and relationship networks.

[0003] In recent years, with the rapid development of artificial intelligence and computer vision technology, action recognition as an important research direction has received widespread attention. Action recognition technology aims to automatically recognize and understand human actions and behaviors by analyzing video or image sequences. This technology has a wide range of applications in intelligent monitoring, human-computer interaction, sports analysis, medical health and other fields. However, traditional action recognition methods often rely on a large amount of labeled data. For application scenarios with high data acquisition and labeling costs, this poses a huge challenge. Few-shot action recognition (Few-Shot Action Recognition) is an important computer vision task that aims to recognize and understand different action categories from action data with very limited sample quantities. Therefore, in order to cope with this challenge, few-shot action recognition has gradually become a research hotspot. Many experts and scholars have proposed solutions from the aspects of algorithm, network structure, multi-modal fusion, etc. to improve the performance of few-shot action recognition models.

[0004] One of the current prior arts is a few-shot action recognition method combined with attention mechanism in Perrett T et al. Temporal-Relational CrossTransformers for Few-Shot Action Recognition, called Temporal-Relational CrossTransformers (TRX), which extracts temporal features by using attention mechanism to match each query subsequence with all subsequences in the support set and aggregates these evidences to build query-specific class prototypes; By performing attention operations on time-ordered subsequences rather than individual frames, the model can better match actions performed at different speeds and in different parts of the video, thus distinguishing fine-grained categories. The disadvantage of this method is that when building class prototypes, it considers the distance between support samples and query samples, but does not consider the distance between support samples, and is easily disturbed by samples with low class relevance.

[0005] The second current prior art is a dual-prototype network (DPN) for FSL in Jiang L et al. A dual-prototype network combining query-specific and class-specific attentive learning for few-shot action recognition, which can effectively reduce intra-class differences and increase inter-class discrimination; The support samples are reweighted to highlight representative samples and reduce the influence of outliers and noise; By combining the proposed class-specific attention and query-specific attention, two prototype sets for FSAR are established; Query-specific attention reweights support samples adaptively according to the similarity between support samples and each query sample when calculating prototypes, thereby expanding the difference between support samples and improving inter-class discrimination.

[0006] The third prior art is a hierarchical compositional representation learning method (HCR) for few-shot action recognition in Li C et al. "Hierarchical compositional representations for few-shot action recognition", which first extracts representative frames as multiple sub-actions from the frame set through a clustering algorithm, and then further decomposes the sub-actions into more fine-grained SAS actions through a partial attention module (PAM), thereby forming a hierarchical compositional representation of the sub-actions; the earth mover distance (EMD) is used as a distance function to match the sub-action representation, so as to better compare the fine-grained patterns; finally, after obtaining the matching similarity of EMD, the probability of various actions is calculated by using a softmax function. The disadvantage of this method is that the traditional hierarchical clustering method is modified to divide the video into sub-actions with different time lengths, instead of equally dividing the video sequence. Therefore, similar video frames are clustered together, and continuous semantics are preserved in the sub-actions. However, the algorithm efficiency is relatively low, and there is room for improvement. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the prior art and provide a few-shot action recognition method and system in a smart health environment. The main problems solved by the present application are: 1) how to solve the problem that the existing few-shot action recognition method based on transformer can achieve high accuracy on some data sets, but the class prototypes generated by the algorithm do not perfectly consider the distance between samples, resulting in that the generated class prototypes do not fully represent the characteristics of the class; 2) how to solve the problem that the frames used to generate sample features in the existing few-shot action recognition method based on transformer are obtained by uniform sampling, which cannot fully extract representative frames in the video, and further improvement is made on the method of extracting representative frames.

[0008] To solve the above problems, the present application provides a few-shot action recognition method in a smart health environment, which comprises:

[0009] Input the support set and the query set containing video samples, the support set is a sample set used for the model to learn features, the query set is a sample set used for the model to predict, and representative frames are extracted from each video by a frame extraction algorithm to obtain the frame set of the support set and the query set samples;

[0010] Data augmentation is performed on each frame image in the frame set of the support set and the query set samples, and the frame set of the support set and the query set samples after data preprocessing is obtained by processing the frame set into a uniform size;

[0011] extracting frame set features of the pre-processed support set and query set samples by using a feature extraction network, adding position encoding to the extracted features, and constructing sample features from frame features of each sample, and outputting a support sample feature set and a query sample feature;

[0012] inputting the support sample feature set and the query sample feature, dividing data into sample pairs of support samples and query samples, inputting each sample pair into a transformer for calculation, generating support sample features adaptively re-weighted according to similarity with query samples, and outputting a processed support sample feature set and a query sample feature;

[0013] calculating similarity between sample features of the same type for each support sample according to the support sample feature set, summing, averaging, and normalizing the results to obtain sample weights of each support sample;

[0014] combining the processed support sample feature set and the sample weights of each support sample, weighting and summing all sample features according to corresponding weights in each class to construct prototypes of support samples of each class;

[0015] inputting the prototypes of support samples of each class and the query sample feature, calculating distance between each pair of class prototypes and query sample features, and taking the class with the smallest distance as the predicted result.

[0016] Preferably, the input includes a support set and a query set of video samples, the support set is a sample set used for the model to learn features, and the query set is a sample set used for the model to predict, representative frames are extracted from each video by a frame extraction algorithm to obtain frame sets of support set and query set samples, and specifically:

[0017] The support set includes CxK video samples, and the query set includes a single video sample, where C and K are hyperparameters, C represents the number of classes, and K represents the number of samples per class.

[0018] The frame extraction algorithm uses the principle that "the more similar the frames are, the higher the degree of repetition of the feature information they express, and extracting frames with lower similarity from the frame set extracted from a video can express more feature information", divides the frame set into N clusters by using the similarity between frames, extracts the frame located in the middle position from each cluster as a representative frame, and thus outputs frames with relatively more information, where N is a hyperparameter representing the number of frames per sample.

[0019] Preferably, the feature extraction network is used to extract frame set features of the pre-processed support set and query set samples, add position encoding to the extracted features, construct sample features from frame features of each sample, and output a support sample feature set and a query sample feature, and specifically:

[0020] The backbone network for feature extraction is ResNet50 pre-trained on ImageNet dataset, only replacing the last fully connected layer of the backbone network, so that the dimension of the output feature is a hyperparameter d. During the training process, the parameters of the backbone network are frozen except for the replaced fully connected layer;

[0021] Each support sample feature is represented as:

[0022]

[0023] The query sample feature is represented as:

[0024] Q={q1+PE(1),…,q n +PE(n),…,q N +PE(N)}

[0025] Wherein, is the feature information of the nth frame of the kth sample of the cth class in the support set, q n is the feature information of the nth frame of the query sample, and PE is the position encoding, which adds position information to each frame feature;

[0026] The feature set of the support sample is represented as:

[0027] S={S 1 ,…,S c ,…,S C}

[0028] Wherein, S c is the feature set of the cth class support sample, represented as:

[0029]

[0030] is the feature of each support sample, which is further simplified as:

[0031]

[0032] Similarly, the query sample feature is further simplified as:

[0033] Q={Q1,…,Q n ,…,Q N}。

[0034] Preferably, the input support sample feature set and query sample feature set divide the data into support sample and query sample pairs. Each sample pair is input into a transformer for calculation to generate support sample features that are adaptively reweighted based on their similarity to the query sample. The output is the processed support sample feature set and query sample feature set, specifically:

[0035] In the transformer, the supporting sample features are used for the transformer's key and value, and the query sample features are used for the query. Here, query, key, and value are all feature vectors, derived from the sample features through a specific weight matrix W. q W k W v The similarity weight between the supporting sample features and the query sample features is calculated using the query and key. The value is then weighted and summed according to the weight to obtain the adaptively reweighted supporting sample features.

[0036] The formula for calculating transformer is shown below:

[0037]

[0038] in, It is an intermediate variable, representing the initial attention weight Q obtained by calculating the m-th feature in the query sample feature representation and the n-th feature of the k-th sample of class c in the support set. m To query the m-th feature q in the sample feature representation m +PE(m), where L is the standard layer normalization;

[0039] It is the attention weight corresponding to the feature of the m-th query sample and the l-th feature of the k-th sample of the c-th class in the support set, where l is an integer variable in the range [1, N], used to... Summation, that is, summing each of the given c, m, k go through All the calculations are then added together; Indicates the normalized result Attention weights; d k The scaling factor is d. k It can alleviate, to some extent, the situation where the result of the dot product is too large in dimensionality, causing the result to be in a region where the gradient of the softmax function is very small;

[0040] Similarly, W q W k W v The weight matrix is... is the mth frame feature of the kth sample of the cth category after processing;

[0041] In order to ensure the consistency of the dimensions and facilitate subsequent calculations, the query sample features are processed by the weight matrix W generated by the transformer to generate the value v , and the specific process is as follows:

[0042] u m =W v *Q m

[0043] Wherein, u m represents the query sample feature after processing with consistent dimensions;

[0044] The feature set of the processed support sample is composed of the value generated by the transformer , denoted as T={T 1 ,…,T c ,…,T c}, wherein T c is the feature set of the cth category support sample, denoted as Wherein is the feature of the kth sample of the cth category, denoted as Similarly, the query sample feature U is simplified as U={u1,…,u n ,…,u N}.

[0045] Preferably, according to the feature set of the support sample, the similarity between the features of the same sample is calculated for each support sample, and the results are summed, averaged and normalized to obtain the sample weight of each support sample, which is specifically:

[0046] Since the sample feature is composed of multiple frame feature vectors, the dynamic time warping algorithm DTW is used to calculate the similarity between the features of the same sample. The frames of two samples are time-aligned, then the distance between the aligned frame feature vectors is calculated, and finally the vector distance between the videos is obtained to get the similarity between the videos.

[0047] Specifically, the cumulative distance γ between the mth feature vector of the sample feature and the nth feature vector of the sample feature is:

[0048]

[0049] Wherein, is the distance between the two feature vectors, which is calculated by using the Euclidean distance, and the specific process is as follows:

[0050]

[0051] where x 1i and x 2i are eigenvectors of the i-th element of the eigenvector and d is the dimension of the output feature, representing the length of the eigenvector, i.e., the number of elements contained;

[0052] According to the DTW algorithm, the distance between two sample features is the cumulative distance between the last eigenvector of the two sample features i.e.

[0053] Based on the distance D, the average distance of each video sample feature to all other video sample features of the same class is calculated, and the calculation formula is as follows:

[0054]

[0055] wherein, represents the distance between the sample feature of the k-th video in the c-th class and the sample feature of the i-th video, is the average distance of the sample feature of the k-th video in the c-th class to all other video sample features of the same class, and thus the average distance of each video to all other videos of the same class is obtained;

[0056] Finally, the sample weight of each support sample is calculated as Based on the feature that "the greater the average distance of a video to all other videos of the same class, the lower the average similarity of the video to all other videos of the same class", therefore, the similarity weight is calculated by the following method:

[0057]

[0058] wherein, plays a role of ensuring that the similarity is a positive number, and the greater the average distance, the lower the similarity, in addition, in order to facilitate subsequent calculation, a normalized processing method is adopted, represents the sum of the average similarity of all samples of the c-th class.

[0059] Preferably, the combination of the processed feature set of the support sample and the sample weight of each support sample is to weight and sum all sample features according to the corresponding weight in each class, and to construct the prototype of each class of support sample, specifically:

[0060] The calculation formula of the prototype is:

[0061]

[0062] wherein, is the weight of the kth support sample of the cth class, is the n-th feature vector of the kth support sample of the cth class;

[0063] Therefore, the class prototype of the cth class is The prototype set of the support sample is P = {P 1 ,…,P c ,…,P C}.

[0064] Preferably, the input prototype of each class of the support sample and the query sample feature, for each pair of class prototype and query sample feature, calculate the distance between them, take the class with the smallest distance as the predicted result, specifically:

[0065] The calculation formula of the distance between the class prototype and the query sample feature is:

[0066]

[0067] wherein, is the n-th feature vector of the cth prototype, u n is the n-th feature vector of the query sample, and the predicted class c' is represented as c' = argmin c D(Q, S c ), argmin c represents the value when D(Q, S c ) takes the minimum value c;

[0068] The loss function of the few-shot action recognition model is as follows:

[0069]

[0070] wherein, c' is the predicted class generated by the model from the training task, and D(Q, S c ) is the distance between the query sample and the cth prototype;

[0071] The model is trained through the loss, thereby optimizing the model parameters, and finally forming the few-shot action recognition model, and the user can monitor the monitoring video for action recognition by using the model.

[0072] Correspondingly, the application also provides a few-shot action recognition system in a smart health environment, comprising:

[0073] A frame extraction unit inputs a support set and a query set containing video samples, the support set is a sample set for learning features of the model, the query set is a sample set for prediction of the model, representative frames are extracted from each video through a frame extraction algorithm to obtain a frame set of the support set and the query set samples;

[0074] a data enhancement unit configured to perform data enhancement on each frame image in the frame set of the support set and the query set sample and process the frame image into a uniform size to obtain the frame set of the support set and the query set sample after data preprocessing;

[0075] a feature extraction unit configured to extract frame features of the frame set of the support set and the query set sample after data preprocessing by using a feature extraction network, add position encoding to the extracted features, and output the frame features of each sample as sample features to obtain a support sample feature set and a query sample feature;

[0076] a sample matching unit configured to input the support sample feature set and the query sample feature, divide data into sample pairs of support samples and query samples, input each sample pair into a transformer to calculate and generate support sample features adaptively reweighted according to similarity with the query sample, and output a processed support sample feature set and the query sample feature;

[0077] a weight distribution unit configured to calculate similarity between sample features of the same type for each support sample according to the support sample feature set, sum, average and normalize the results to obtain sample weights of the support samples;

[0078] a prototype construction unit configured to combine the processed support sample feature set and the sample weights of the support samples, sum all sample features according to corresponding weights in each class to construct prototypes of the support samples in each class;

[0079] a result generation unit configured to input the prototypes of the support samples in each class and the query sample feature, calculate distance between each pair of class prototypes and the query sample feature, and take a class with the smallest distance as a predicted result.

[0080] The present application has the following beneficial effects:

[0081] The present application fully utilizes differences between classes, between videos of the same type, and between frames of the same video, and features of each part are connected by extraction and integration, various information of samples is fully utilized, and the generated action recognition result is more consistent with human expectations.

[0082] In the present application, the transformer method is used, each query frame feature is matched with all frame features in the support set by using an attention mechanism, and weighted summation is performed to construct a support sample specific to the query. This method is beneficial to highlight frames with greater importance in the support sample for the query sample, the reconstructed support sample is more matched with the query sample and is easier to compare, the accuracy of the result is greatly improved, and the method can be used in scenes with relatively high requirements for action recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 is a few-shot action recognition method flow chart in a smart health environment according to an embodiment of the present application;

[0084] Figure 2 is a few-shot action recognition system structure diagram in a smart health environment according to an embodiment of the present application. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0086] Figure 1 is a few-shot action recognition method flow chart in a smart health environment according to an embodiment of the present application, as shown in the figure, the method comprises: Figure 1

[0087] S1, input a support set and a query set containing video samples, the support set is a sample set used for learning features of a model, the query set is a sample set used for predicting the model, representative frames are extracted from each video by a frame extraction algorithm to obtain frame sets of the support set and the query set samples;

[0088] S2, data augmentation is performed on each frame image in the frame set of the support set and the query set samples, and the frame set is processed into a uniform size to obtain the frame set of the support set and the query set samples after data preprocessing;

[0089] S3, a feature extraction network is used to extract features of the frame set of the support set and the query set samples after data preprocessing, and position coding is added to the extracted features, the frame features of each sample constitute the features of the sample, and the output is a feature set of the support sample and a query sample feature;

[0090] S4, input the feature set of the support sample and the query sample feature, divide the data into sample pairs of the support sample and the query sample, input each sample pair into a transformer to calculate, generate support sample features adaptively reweighted according to similarity with the query sample, and output the processed feature set of the support sample and the query sample feature;

[0091] S5, according to the feature set of the support sample, the similarity between the features of the same sample is calculated for each support sample, the results are summed, averaged and normalized to obtain the sample weight of each support sample;

[0092] ​S6, combining the feature set of the processed support samples and the sample weight of each support sample, in each class, all sample features are weighted and summed according to the corresponding weight to construct the prototype of each class of support samples;

[0093] S7, input the prototype of each class of support samples and the query sample feature, for each pair of class prototype and query sample feature, calculate the distance between them, and take the class with the smallest distance as the predicted result.

[0094] Step S1 is as follows:

[0095] S1-1: the support set contains CxK video samples, and the query set contains a single video sample, wherein C and K are hyperparameters, C represents the number of classes, and K represents the number of samples per class;

[0096] S1-2: the frame extraction algorithm uses the principle that "the more similar the frames are, the higher the degree of repetition of the feature information expressed, and the more frames with low similarity can express more feature information", and divides the frame set into N clusters by using the similarity between frames, extracts the frame located in the middle position from each cluster as the representative frame, thereby outputting the frame with more information, wherein N is a hyperparameter, representing the number of frames per sample, generally set as N = 8.

[0097] Step S2 is as follows:

[0098] S2-1: the input is the frame set of the support set and the query set samples, data augmentation is performed on each frame image, generally random flipping and random cropping are adopted during training, and center cropping is adopted during inference, at the same time, the frame set is processed into a uniform size, generally the picture size is adjusted to a size with a short side of 256 pixels, and then cropped to a size of 224*224 pixels during data augmentation, and the output is the frame set of the support set and the query set samples after data preprocessing.

[0099] Step S3 is as follows:

[0100] S3-1: the backbone network used for feature extraction is ResNet50 pre-trained on ImageNet dataset, only the last fully connected layer of the backbone network is replaced, so that the dimension of the output feature is a hyperparameter d = 2048, and during the training process, the parameters of the backbone network are frozen except the replaced fully connected layer;

[0101] S3-2: each support sample feature is represented as:

[0102]

[0103] The query sample feature is represented as:

[0104] Q = {q1+PE(1),…,q n +PE(n),…,q N +PE(N)}

[0105] wherein, is the feature information of the nth frame of the kth sample of the cth class of support set, q n is the feature information of the nth frame of the query sample, and PE is the position encoding, which serves to add position information to each frame feature;

[0106] S3-3: The feature set of the support sample is represented as:

[0107] S = {S 1 ,…,S c ,…,S C}

[0108] wherein, S c is the feature set of the cth class of support sample, which is represented as:

[0109]

[0110] is the feature of the kth sample of the cth class, which is represented as shown in S3-1, and is further simplified as:

[0111]

[0112] Similarly, the feature of the query sample is further simplified as:

[0113] Q = {Q1,…,Q n ,…,Q N}.

[0114] Step S4 is specifically as follows:

[0115] S4-1: In the transformer, the support sample feature is used for the key and value of the transformer, and the query sample feature is used for the query, wherein the query, the key and the value are all feature vectors generated by the sample feature through specific weight matrices W q , W k , W v , the similarity weight between the support sample feature and the query sample feature is calculated through the query and the key, the value is weighted and summed according to the weight, and finally the adaptively reweighted support sample feature is obtained;

[0116] S4-2: The calculation formula of the transformer is as follows:

[0117]

[0118] wherein, is an intermediate variable, representing the initial attention weight calculated by the m-th feature in the query sample feature representation and the n-th feature of the k-th sample of the c-th class in the support set, Q m is the m-th feature q m + PE(m), and L is the standard layer normalization.

[0119] is the attention weight corresponding to the m-th query sample feature and the l-th feature of the k-th sample of the c-th class in the support set, and l is an integer variable in the range [1, N], which is used to calculate sum, that is, all the after the calculation of are added up. denotes the normalized attention weight; d k is a scaling factor, divided by d k can alleviate the problem of excessive dimension of the result obtained by the dot product to some extent, so that the result is in the region where the gradient of the softmax function is very small.

[0120] Similarly, W q , W k , and W v are the weight matrices, is the m-th frame feature of the k-th sample of the c-th class after processing;

[0121] S4-3: In order to ensure the consistency of the dimension and facilitate subsequent calculation, the query sample feature is processed by the weight matrix W v generated by the transformer to generate the value, and the specific process is as follows:

[0122] u m = W v * Q m

[0123] wherein, u m represents the query sample feature after processing with consistent dimensions;

[0124] S4-4: The feature set T of the processed support sample is composed of the generated by the transformer, denoted as T = {T 1 ,…, T c ,…, T C}, wherein T c is the feature set of the c-th support sample, denoted as wherein Let the feature of the k-th sample in class c be denoted as: Similarly, the query sample feature U can be simplified to U = {u1, ..., u} n ,…,u N}

[0125] Step S5 is as follows:

[0126] S5-1: Since the sample features are composed of multiple frame feature vectors, the Dynamic Time Warping (DTW) algorithm is used to calculate the similarity between the features of samples of the same type. The frames of the two samples are time-aligned, and then the distance between the aligned frame feature vectors is calculated. Finally, the similarity between the videos is obtained through the vector distance between each video.

[0127] S5-2: For any two sample features and The distance between the two can be calculated using the DTW algorithm, specifically, for sample features. The m-th feature vector and sample features The cumulative distance γ between the nth eigenvectors is:

[0128]

[0129] in, The distance between two feature vectors is calculated using Euclidean distance, as follows:

[0130]

[0131] Where, x 1i With x 2i respectively, feature vectors and The i-th element, d is the dimension of the output feature, representing the length of the feature vector, that is, the number of elements it contains;

[0132] According to the DTW algorithm, the distance between two sample features is the cumulative distance between the last feature vectors of the two sample features. Right now

[0133] Based on distance D, the average distance from each video sample feature to the features of all other video samples of the same type is calculated using the following formula:

[0134]

[0135] in, This represents the distance between the sample features of the k-th video in class c and the sample features of the i-th video. The average of the distance of the sample feature of the kth video of the cth category to all other sample features of the same category, thereby obtaining the average distance of each video to all other videos of the same category;

[0136] Finally, the sample weight of each support sample is calculated as Based on the feature that "the greater the average distance of a video to all other videos of the same category, the lower the average similarity of the video to all other videos of the same category", therefore, the similarity weight It is calculated by the following method:

[0137]

[0138] wherein, It plays a role of ensuring that the similarity is a positive number, and the greater the average distance, the lower the similarity. In addition, in order to facilitate subsequent calculation, a normalized processing method is adopted, which represents the sum of the average similarity of all samples of the cth category.

[0139] Step S6 is as follows:

[0140] S6-1: The calculation formula of the prototype is:

[0141]

[0142] wherein, is the weight of the kth support sample of the cth category, is the nth feature vector of the kth support sample of the cth category;

[0143] Therefore, the class prototype of the cth category is The prototype set of the support sample is P={P 1 ,…,P c ,…,P C}.

[0144] Step S7 is as follows:

[0145] S7-1: The calculation formula of the distance between the class prototype and the query sample feature is:

[0146]

[0147] wherein, is the nth feature vector of the prototype of the cth category, u n is the nth feature vector of the query sample, and the predicted category c' is represented as c'=argmin c D(Q,S c ), argmin c represents the value when D(Q,S c ) takes the minimum value c.

[0148] S7-2: the loss function of the few-shot action recognition model is as follows:

[0149]

[0150] wherein c' is the predicted category generated by the model from the training task, D(Q, S c ) is the distance between the query sample and the c prototype;

[0151] The model is trained through the loss, thereby optimizing the model parameters, and finally forming the few-shot action recognition model, so that the user can monitor the monitoring video for action recognition.

[0152] Correspondingly, the application also provides a few-shot action recognition system in a smart health environment, as shown in the figure, comprising: Figure 2

[0153] a frame extraction unit 1 for inputting a support set and a query set containing video samples, the support set is a sample set for learning features of the model, the query set is a sample set for predicting the model, representative frames are extracted from each video through a frame extraction algorithm to obtain the frame set of the support set and the query set samples;

[0154] a data enhancement unit 2 for data enhancement on each frame image in the frame set of the support set and the query set samples, and processing it into a uniform size to obtain the frame set of the support set and the query set samples after data preprocessing;

[0155] a feature extraction unit 3 for extracting the frame set features of the support set and the query set samples after data preprocessing by using a feature extraction network, and adding position coding to the extracted features, the frame features of each sample constitute the features of the sample, and the output is the feature set of the support sample and the query sample feature;

[0156] a sample matching unit 4 for inputting the feature set of the support sample and the query sample feature, dividing the data into sample pairs of support samples and query samples, inputting each sample pair into the transformer to calculate, generating support sample features adaptively reweighted according to the similarity with the query sample, and outputting the processed feature set of the support sample and the query sample feature;

[0157] a weight allocation unit 5 for calculating the similarity between the features of the same sample according to the feature set of the support sample, summing, averaging and normalizing the results to obtain the sample weight of each support sample;

[0158] ​A prototype construction unit 6 is configured to combine the feature set of the processed support samples and the sample weight of each support sample, and to construct a prototype of each class by weighted summing all sample features according to the corresponding weight in each class.

[0159] A result generation unit 7 is configured to input the prototype of each class of support samples and the query sample feature, and to calculate the distance between each pair of class prototype and query sample feature, and to take the class with the smallest distance as the predicted result.

[0160] Therefore, the present application makes full use of the differences between classes, videos of the same class, and frames of the same video, and the features of each part are connected through extraction and integration, making full use of various information of the sample, so that the generated action recognition result is more in line with human expectations; in addition, the method using the transformer in the present application matches each query frame feature with all frame features in the support set by using the attention mechanism, and performs weighted summing to construct a support sample specific to the query. This method is beneficial to highlight the frames in the support sample that are more important to the query sample, and the reconstructed support sample is more matched with the query sample and easier to compare, greatly improving the accuracy of the result, and can be used in scenes with relatively high requirements for action recognition accuracy.

[0161] The above describes in detail the method and system for few-shot action recognition in a smart health environment provided by the embodiment of the present application, and the principle and implementation mode of the present application are described by applying specific examples; the above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for few-shot action recognition in a smart health environment, the method comprising: The method comprises: Inputting a support set and a query set containing video samples, the support set being a sample set for the model to learn features, and the query set being a sample set for the model to predict, representative frames being extracted from each video by a frame extraction algorithm to obtain frame sets of the support set and the query set samples; Data augmentation is performed on each frame image in the frame set of the support set and the query set samples, and the frame set is processed to a uniform size to obtain the frame set of the support set and the query set samples after data preprocessing; Features of the frame set of the support set and the query set samples after data preprocessing are extracted by a feature extraction network, and position encoding is added to the extracted features, the frame features of each sample constituting the features of the sample, and the output being a feature set of the support samples and a query sample feature; The feature set of the support samples and the query sample feature are inputted, the data is divided into sample pairs of support samples and query samples, each sample pair is inputted into a transformer for calculation, support sample features adaptively reweighted according to similarity with the query sample are generated, and the output is a processed feature set of the support samples and the query sample feature; According to the feature set of the support samples, the similarity between the features of the same sample is calculated for each support sample, and the results are summed, averaged and normalized to obtain the sample weight of each support sample; In each class, all sample features are weighted and summed according to the corresponding weight to construct the prototype of each class of support samples, in combination with the processed feature set of the support samples and the sample weight of each support sample; The prototype of each class of support samples and the query sample feature are inputted, the distance between each pair of class prototype and query sample feature is calculated, and the class with the smallest distance is taken as the prediction result; Specifically, the feature set of the support samples and the query sample feature are inputted, the data is divided into sample pairs of support samples and query samples, each sample pair is inputted into a transformer for calculation, support sample features adaptively reweighted according to similarity with the query sample are generated, and the output is a processed feature set of the support samples and the query sample feature, which is specifically as follows: In the transformer, the support sample features are used for keys and values of the transformer, and the query sample features are used for queries, wherein the query, the key, and the value are all feature vectors generated by the sample features through specific weight matrices W q , W k , W v , and the similarity weight between the support sample features and the query sample features is calculated through the query and the key, the value is weighted and summed according to the weight, and finally the adaptive reweighted support sample features are obtained. The transformer calculation formula is as follows: wherein, is an intermediate variable representing the initial attention weight computed from the feature of the m-th frame in the query sample feature representation and the feature of the n-th frame of the k-th sample of the c-th class in the support set, Q m is the feature of the m-th frame in the query sample feature representation, m + PE(m), PE is the position encoding, and L is the standard layer normalization. is the attention weight of the query sample feature corresponding to the m-th frame and the feature of the l-th frame of the k-th sample of the c-th class of the support set, l is an integer variable in the range [1, N], which is used to combine sum, i.e. all of the after the calculation of are added up; denotes the normalized attention weight; d k is a scaling factor; to support the feature information of the nth frame of the kth sample of the cth category, is the processed feature of the mth frame of the kth sample of the cth category. The query sample features are processed through the weight matrix W of the transformer to generate values v , specifically as follows: u m = W v * Q m wherein u m represents the processed query sample features with consistent dimensions; The set of features of the processed support samples is generated by a transformer comprising, denoted as T = {T 1 ,…,T c ,…,T C}, where T c is the set of features of the c-th class of support samples, denoted as where is the feature of the k-th sample of the c-th class, denoted as The query sample features U are simplified to U = {u1,…,u n ,…,u N}. 2.The few-shot action recognition method in a smart health environment of claim 1, wherein, The support set and the query set containing video samples are inputted, the support set being a sample set for the model to learn features, and the query set being a sample set for the model to predict, representative frames being extracted from each video by a frame extraction algorithm to obtain frame sets of the support set and the query set samples, which is specifically as follows: The support set contains C×K video samples, and the query set contains a single video sample, wherein C and K are hyperparameters, C represents the number of classes, and K represents the number of samples per class; The frame extraction algorithm divides the frame set into N clusters by using the similarity between frames, extracts the frame located in the middle position from each cluster as the representative frame, and thus outputs the frame with more information, wherein N is a hyperparameter representing the number of frames per sample. 3.The few-shot action recognition method in a smart health environment of claim 1, wherein, The frame set features of the preprocessed support set and query set samples are extracted by using a feature extraction network, and position encoding is added to the extracted features, the frame features of each sample constitute the features of the sample, and the output is the feature set of the support sample and the query sample feature, specifically: The backbone network for feature extraction is ResNet50 pre-trained on the ImageNet dataset, and only the last fully connected layer of the backbone network is replaced to make the dimension of the output feature d, a hyperparameter, during the training process, the parameters of the backbone network are frozen except the replaced fully connected layer; Each support sample feature is represented as: The query sample feature is represented as: Q = {q1+ PE(1),..., q n + PE(n),..., q N + PE(N)} wherein, q is the feature information of the nth frame of the kth sample of the cth category for supporting the centralized training, n PE is the position encoding for querying the feature information of the nth frame of the sample, and serves to add position information to each frame of features; The feature set of the support sample is represented as: S = {S 1 ,…,S c ,…,S C} where S c is the feature set of the cth class of support samples, denoted as: For each support sample feature, further simplify to: The query sample feature representation is further simplified as: Q = {Q1,..., Q n ..., Q N}. 4.The few-shot action recognition method in a smart health environment of claim 1, wherein, The similarity between the features of the same sample is calculated for each support sample based on the feature set of the support sample, and the results are summed, averaged, and normalized to obtain the sample weight of each support sample, specifically: Since the sample feature is composed of multiple frame feature vectors, the dynamic time warping algorithm DTW is used to calculate the similarity between the features of the same sample, the frames of two samples are time-aligned, then the distance between the aligned frame feature vectors is calculated, and finally the similarity between the videos is obtained through the vector distance between the videos; In particular, the accumulated distance γ between the feature vector of the mth frame of the sample feature and the feature vector of the nth frame of the sample feature is: wherein, is the distance between two feature vectors, calculated using the Euclidean distance, as follows: wherein x 1i and x 2i are eigenvectors and are the i-th elements of the eigenvectors x and d is the dimension of the output feature, representing the length of the eigenvector, i.e. the number of elements contained. According to the DTW algorithm, the distance between two sample features is the accumulated distance between the last feature vectors of the two sample features That is Based on the distance D, the average distance of each video sample feature to all other video sample features of the same class is calculated, and the calculation formula is as follows: wherein, represents the distance of the sample feature of the kth video in the cth category to the sample feature of the jth video, is the average distance of the sample feature of the kth video in the cth category to the sample features of all other videos in the same category, thus obtaining the average distance of each video to all other videos in the same category; Finally, the sample weight of each support sample is calculated as similarity weight This is calculated by the following method: wherein, plays a role of ensuring that the similarity is a positive number while the greater the average distance, the lower the similarity, denotes the sum of the average similarities of all samples of the cth class. 5.The few-shot action recognition method in a smart health environment of claim 4, wherein, The prototype of each class of support sample is constructed by weighting and summing all sample features according to the corresponding weight in each class based on the processed feature set of the support sample and the sample weight of each support sample, specifically: The calculation formula of the prototype is: wherein, is the weight of the kth support sample of the cth class, is the n-th frame feature vector of the kth support sample of the cth class. Thus, the class prototype of the c-th class is The set of prototypes supporting the samples is P = {P 1 ,..., P c ,..., P C}.

6. The few-shot action recognition method in a smart health environment of claim 5, wherein, The distance between each pair of class prototype and query sample feature is calculated, and the class with the smallest distance is taken as the predicted result, specifically: The calculation formula of the distance between the class prototype and the query sample feature is: wherein, is the feature vector of the nth frame of the prototype of the cth class, u n is the feature vector of the nth frame of the query sample, and the predicted class c' is represented as c' = argmin c D(Q, S c ), argmin c represents the value when D(Q, S c ) takes the minimum value c; The loss function of the few-shot action recognition model is as follows: where c' is the predicted class generated by the model from the training task, D(Q, S c ) is the distance between the query sample and the c-th prototype. The model is trained by the loss to optimize the model parameters, and finally a few-shot action recognition model is formed, which can be used by users to monitor and recognize actions in monitoring videos. 7.A few-shot action recognition system in a smart health environment, characterized in that, The system comprises: A frame extraction unit inputs a support set and a query set containing video samples, the support set is a sample set for learning features by the model, the query set is a sample set for prediction by the model, representative frames are extracted from each video by a frame extraction algorithm to obtain the frame set of the support set and the query set samples; A data enhancement unit performs data enhancement on each frame image in the frame set of the support set and the query set samples, and processes it into a uniform size to obtain the frame set of the preprocessed support set and query set samples; A feature extraction unit extracts the frame set features of the preprocessed support set and query set samples by using a feature extraction network, and adds position encoding to the extracted features, the frame features of each sample constitute the features of the sample, and the output is the feature set of the support sample and the query sample feature; The sample matching unit inputs the feature set of the support sample and the feature of the query sample, divides the data into a sample pair of the support sample and the query sample, inputs each sample pair into the transformer for calculation, generates the support sample feature adaptively reweighted according to the similarity with the query sample, and outputs the processed feature set of the support sample and the feature of the query sample. The weight distribution unit calculates the similarity between the features of the same sample according to the feature set of the support sample, sums, averages and normalizes the results to obtain the sample weight of each support sample. The prototype construction unit combines the processed feature set of the support sample and the sample weight of each support sample, weights and sums all sample features according to the corresponding weight in each class to construct the prototype of each class of the support sample. The result generation unit inputs the prototype of each class of the support sample and the feature of the query sample, calculates the distance between each pair of class prototype and query sample feature, and takes the class with the smallest distance as the predicted result. The sample matching unit specifically comprises: In the transformer, the support sample features are used for keys and values of the transformer, and the query sample features are used for queries, wherein the query, the key, and the value are all feature vectors generated by the sample features through a specific weight matrix W q , W k , W v , and the similarity weight between the support sample features and the query sample features is calculated through the query and the key, the value is weighted and summed according to the weight, and finally the adaptive reweighted support sample features are obtained. The transformer calculation formula is as follows: wherein, is an intermediate variable representing the initial attention weight computed from the feature of the m-th frame in the query sample feature representation and the feature of the n-th frame of the k-th sample of the c-th class in the support set, Q m is the feature of the m-th frame in the query sample feature representation, m + PE(m), PE is the position encoding, and L is the standard layer normalization. is the attention weight of the query sample feature corresponding to the m-th frame and the feature of the l-th frame of the k-th sample of the c-th class of the support set, where l is an integer variable in the range [1, N], used to weight summed, i.e. all of the after the calculation of are added up; denotes the normalized attention weight; d k is a scaling factor; to support the feature information of the nth frame of the kth sample of the cth category, is the processed feature of the mth frame of the kth sample of the cth category. The query sample features are processed through the weight matrix W of the transformer to generate values v , specifically as follows: u m = W v * Q m wherein u m represents the processed query sample features with consistent dimensions; The set of features of the processed support samples is generated by a transformer comprising, denoted as T = {T 1 ,…,T c ,…,T C}, where T c is the set of features of the c-th class of support samples, denoted as where is the feature of the k-th sample of the c-th class, denoted as The query sample features U are simplified to U = {u1,…,u n ,…,u N}.

8. The few-shot action recognition system in a smart health environment of claim 7, wherein, The input includes the support set and the query set of the video sample, the support set is a sample set used for the model to learn features, the query set is a sample set used for the model to predict, representative frames are extracted from each video by a frame extraction algorithm to obtain the frame set of the support set and the query set sample, and specifically comprises: The support set includes C×K video samples, and the query set includes a single video sample, wherein C and K are hyperparameters, C represents the number of classes, and K represents the number of samples per class. The frame extraction algorithm divides the frame set into N clusters by using the similarity between frames, extracts the frame located in the middle position in each cluster as a representative frame, and thus outputs a frame with more information, wherein N is a hyperparameter, and represents the number of frames per sample.

9. The few-shot action recognition system in a smart health environment of claim 7, wherein, The feature extraction network is used to extract the frame set features of the support set and the query set sample after the data preprocessing, and position coding is added to the extracted features, the frame features of each sample constitute the features of the sample, and the output is the feature set of the support sample and the feature of the query sample, and specifically comprises: The backbone network for feature extraction is a pre-trained ResNet50 on the ImageNet dataset, and only the last fully connected layer of the backbone network is replaced to make the dimension of the output feature a hyperparameter d, and in the training process, the parameters of the backbone network are frozen except the replaced fully connected layer; Each support sample feature is represented as: The query sample feature is represented as: Q = {q1+ PE(1),..., q n + PE(n),..., q N + PE(N)} wherein, q is the feature information of the nth frame of the kth sample of the cth category for supporting the centralized training, n PE is the position encoding for querying the feature information of the nth frame of the sample, which serves to add position information to each frame of features; The feature set of the support sample is represented as: S = {S 1 ,…,S c ,…,S C} where S c is the set of features of the cth class of support samples, denoted as: For each support sample feature, further simplify to: The query sample feature representation is further simplified as: Q = {Q1,..., Q n ..., Q N}.

10. The few-shot action recognition system in a smart health environment of claim 7, wherein, The weight distribution unit calculates the similarity between the features of the same sample according to the feature set of the support sample, sums, averages and normalizes the results to obtain the sample weight of each support sample. Since the sample features are composed of multiple frame feature vectors, the similarity between features of the same sample is calculated by using a dynamic time warping algorithm (DTW), time alignment is performed on the frames of two samples, the distance between the aligned frame feature vectors is calculated, and finally the similarity between the videos is obtained by the vector distance between the videos. In particular, the accumulated distance γ between the feature vector of the mth frame of the sample feature and the feature vector of the nth frame of the sample feature is: wherein, is the distance between two feature vectors, calculated using the Euclidean distance, as follows: wherein x 1i and x 2i are eigenvectors and the i-th element of x d is the dimension of the output feature, and represents the length of the eigenvector, i.e., the number of elements contained. According to the DTW algorithm, the distance between two sample features is the accumulated distance between the last feature vectors of the two sample features That is Based on the distance D, the average distance of each video sample feature to all other video sample features of the same class is calculated, and the calculation formula is as follows: wherein, represents the distance of the sample feature of the kth video in the cth category to the sample feature of the jth video, is the average distance of the sample feature of the kth video in the cth category to the sample features of all other videos in the same category, thus obtaining the average distance of each video to all other videos in the same category; Finally, the sample weight of each support sample is calculated as similarity weight is calculated by the following method: wherein, plays a role of ensuring that the similarity is a positive number while the greater the average distance, the lower the similarity, denotes the sum of the average similarities of all samples of the cth class.

11. The few-shot action recognition system in a smart health environment of claim 10, wherein, The features of the processed support samples and the sample weights of each support sample are combined, all sample features in each class are weighted and summed according to the corresponding weights, and the prototypes of each class of support samples are constructed, specifically as follows: The calculation formula of the prototype is as follows: wherein, is the weight of the kth support sample of the cth class, is the n-th frame feature vector of the kth support sample of the cth class. Thus, the class prototype of the c-th class is The set of prototypes supporting the samples is P = {P 1 ,…,P c ,…,P C}.

12. The few-shot action recognition system in a smart health environment of claim 11, wherein, The prototypes of each class of support samples and the query sample features are input, the distance between each pair of class prototype and query sample feature is calculated, and the class with the minimum distance is taken as the prediction result, specifically as follows: The calculation formula of the distance between the class prototype and the query sample feature is as follows: wherein, is the feature vector of the nth frame of the prototype of the cth class, u n is the feature vector of the nth frame of the query sample, and the predicted class c' is represented as c' = argmin c D(Q, S c ), argmin c represents the value when D(Q, S c ) takes the minimum value c; The loss function of the few-shot action recognition model is as follows: where c' is the predicted class generated by the model from the training task, D(Q, S c ) is the distance between the query sample and the c-th prototype. The model is trained by using the loss, so as to optimize the model parameters, and finally form the few-shot action recognition model. Users can use the model to monitor and recognize actions in the monitoring video.

Images