Method and system for identifying few-sample actions in smart health environment

Abstract

The invention discloses a few-sample action recognition method in an intelligent healthy environment, which comprises the following steps: inputting a support set and a query set containing video samples, extracting representative frames from each video through a frame extraction algorithm, and performing data enhancement on the representative frames; extracting frame set features of the sample by using a feature extraction network; the method comprises the following steps of: calculating in a transformer, and constructing a support sample feature matched with a query sample; according to the similarity between the intra-class sample features, distributing a weight for each sample; constructing a prototype of each class in combination with support sample features and weights thereof; and calculating the distance between each pair of class prototypes and query sample features, taking the class with the minimum distance as a prediction result, and optimizing the model according to the distance loss. In addition, the invention further discloses a few-sample action recognition system in the intelligent healthy environment. According to the method, various information of the sample is fully utilized, so that the generated action recognition result is higher in accuracy and better conforms to the expectation of the user.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, action recognition, and few-sample learning, and in particular to a few-sample action recognition method and system in a smart health environment. Background Art

[0002] The ability to learn and generalize from a small number of samples is a clear dividing line between artificial intelligence and human intelligence. Because humans can easily establish cognition of new things with only one or a few examples, while machine learning algorithms usually require thousands of supervised samples to ensure their generalization ability. Few-Shot Learning aims to identify new categories from a small number of labeled samples, which is of great significance and challenge in the field of machine learning. The FSL method based on meta-learning is representative in this field, and its main methods include matching networks, prototype networks, and relational networks.

[0003] In recent years, with the rapid development of artificial intelligence and computer vision technology, action recognition, as an important research direction, has received widespread attention. Action recognition technology aims to automatically identify and understand human actions and behaviors by analyzing video or image sequences. This technology has a wide range of applications in many fields such as intelligent monitoring, human-computer interaction, sports analysis, and medical health. However, traditional action recognition methods often rely on a large amount of labeled data. This poses a huge challenge for application scenarios with high data acquisition and labeling costs. Few-Shot Action Recognition is an important computer vision task that aims to identify and understand different action categories from action data with a very limited number of samples. Therefore, in order to meet this challenge, few-shot action recognition has gradually become a research hotspot. Many experts and scholars have proposed solutions from the aspects of algorithms, network structures, and multimodal fusion to improve the performance of few-shot action recognition models.

[0004] One of the current existing technologies is a few-shot action recognition method combined with an attention mechanism in "Temporal-Relational CrossTransformers for Few-Shot Action Recognition" by Perrett T et al., called Temporal-Relational CrossTransformers (TRX). This method uses an attention mechanism to match each query subsequence with all subsequences in the support set and aggregates these evidences to extract temporal features, thereby building a query-specific category prototype; by paying attention to subsequences arranged in chronological order rather than single frames, the model is able to better match actions performed at different speeds and in different parts of the video, thereby distinguishing fine-grained categories. The disadvantage of this method is that the distance between the support sample and the query sample is taken into account when constructing the category prototype, but the distance between the support samples is not taken into account, and it is easily disturbed by samples with low class relevance.

[0005] The second existing technology is a dual prototype network (DPN) for FSL in "Adual-prototype network combining query-specific and class-specific attentive learning for few-shot action recognition" by Jiang L et al. This method can effectively reduce the intra-class differences and increase the inter-class distinction. The support samples are reweighted to highlight representative samples and reduce the influence of outlier samples and noise. By combining the proposed class-specific attention with the query-specific attention, two prototype sets for FSAR are established. When calculating the prototype, the query-specific attention adaptively reweights the support samples according to the similarity between the support samples and each query sample, thereby expanding the differences between the support samples and improving the inter-class distinction.

[0006] The third existing technology is a hierarchical compositional representation learning method (HCR) for few-shot action recognition by Li C et al. in Hierarchical compositional representations for few-shot action recognition. This method first extracts representative frames from a frame set as multiple sub-actions through a clustering algorithm, and then further decomposes the sub-actions into more fine-grained SAS actions through a partial attention module (PAM), thereby forming a hierarchical compositional representation of the sub-actions; the earth movement distance (EMD) is used as a distance function to match the sub-action representations to better compare fine-grained patterns; finally, after obtaining the matching similarity of the EMD, the softmax function is used to calculate the probabilities of various actions. The disadvantage of this method is that the traditional hierarchical clustering method is modified to divide the video into sub-actions with different time lengths instead of evenly dividing the video sequence. Therefore, similar video frames are clustered together, and continuous semantics are retained in the sub-actions. However, the algorithm efficiency is relatively low, and there is room for improvement. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of existing methods and propose a method and system for few-sample action recognition in a smart health environment. The main problems solved by the present invention are: 1) How to solve the problem that although the existing transformer-based few-sample action recognition method can achieve high accuracy on some data sets, the category prototypes generated by the algorithm used do not fully consider the distance between samples, resulting in the generated class prototypes not fully representing the characteristics of the class; 2) How to solve the problem that the frames used to generate sample features in the existing transformer-based few-sample action recognition method are all obtained by uniform sampling, and the representative frames in the video cannot be fully extracted, and further improvements are made in the method of extracting representative frames.

[0008] In order to solve the above problems, the present invention proposes a method for identifying actions with a small number of samples in a smart health environment, the method comprising:

[0009] Input the support set and query set containing video samples. The support set is the sample set used to learn features for the model, and the query set is the sample set used to predict for the model. The representative frames are extracted from each video through the frame extraction algorithm to obtain the frame sets of the support set and query set samples.

[0010] Performing data enhancement on each frame image in the frame set of the support set and the query set samples, and processing them into a uniform size, so as to obtain the frame set of the support set and the query set samples after data preprocessing;

[0011] Using a feature extraction network, extract the frame set features of the support set and query set samples after the data preprocessing, and add position encoding to the extracted features. The frame features of each sample constitute the features of the sample, and the output is a feature set of the support sample and a feature set of the query sample;

[0012] Input the feature set of the support sample and the feature of the query sample, divide the data into sample pairs of the support sample and the query sample, input each sample pair into the transformer for calculation, generate the support sample feature that is adaptively re-weighted according to the similarity with the query sample, and output the processed feature set of the support sample and the query sample feature;

[0013] According to the feature set of the supporting samples, for each supporting sample, the similarity between the features of the same sample is calculated, and the results are summed, averaged, and normalized to obtain the sample weight of each supporting sample;

[0014] Combining the processed feature set of the support samples and the sample weights of each support sample, in each class, all sample features are weighted summed according to the corresponding weights to construct prototypes of each class of support samples;

[0015] Input the prototypes of each class of support samples and the query sample features, calculate the distance between each pair of class prototypes and query sample features, and take the class with the smallest distance as the prediction result.

[0016] Preferably, the input includes a support set and a query set of video samples, the support set is a sample set used to learn features for the model, and the query set is a sample set used to predict the model. Representative frames are extracted from each video through a frame extraction algorithm to obtain frame sets of support set and query set samples, specifically:

[0017] The support set contains C×K video samples, and the query set contains a single video sample, where C and K are hyperparameters, C represents the number of classes, and K represents the number of samples in each class;

[0018] The frame extraction algorithm utilizes the principle that "the more similar the frames are, the more repetitive the feature information they express. Extracting frames with lower similarity from the frame set decomposed from the video can express more feature information". The frame set is divided into N clusters using the similarity between frames. The frame in the middle of each cluster is extracted as the representative frame, thereby outputting frames with relatively more information, where N is a hyperparameter representing the number of frames for each sample.

[0019] Preferably, the feature extraction network is used to extract frame set features of the support set and query set samples after the data preprocessing, and position encoding is added to the extracted features. The frame features of each sample constitute the features of the sample, and the output is a feature set of the support sample and a feature set of the query sample, specifically:

[0020] The backbone network used for feature extraction is ResNet50 pre-trained on the ImageNet dataset. Only the last fully connected layer of the backbone network is replaced so that the dimension of the output feature is the hyperparameter d. During the training process, except for the replaced fully connected layer, the parameters of the backbone network are frozen.

[0021] Each support sample feature is expressed as:

[0022]

[0023] The query sample features are expressed as:

[0024] Q={q1+PE(1),…,q n +PE(n),…,q N +PE(N)}

[0025] in, To support the feature information of the nth frame of the kth sample of the cth class, q n To query the feature information of the nth frame of the sample, PE is the position code, which is used to add position information to each frame feature;

[0026] The feature set of the support sample is expressed as:

[0027] S={S 1 ,…,S c ,…,S C}

[0028] Among them, S c is the feature set of the c-th class support sample, expressed as:

[0029]

[0030] For each supporting sample feature, it is further simplified as:

[0031]

[0032] Similarly, the query sample feature representation is further simplified as follows:

[0033] Q={Q1,…,Q n ,…,Q N}.

[0034] Preferably, the input support sample feature set and query sample feature are divided into sample pairs of support samples and query samples, each sample pair is input into the transformer for calculation, support sample features adaptively re-weighted according to the similarity with the query sample are generated, and the processed support sample feature set and query sample feature are output, specifically:

[0035] In the transformer, the supporting sample features are used as the key and value of the transformer, and the query sample features are used as the query, where query, key, and value are all feature vectors, which are obtained by passing the sample features through a specific weight matrix W. q , W k , W v Generate, calculate the similarity weight between the supporting sample features and the query sample features through query and key, perform weighted summation on the value according to the weight, and finally obtain the supporting sample features that are adaptively re-weighted;

[0036] The transformer calculation formula is as follows:

[0037]

[0038] in, is an intermediate variable, which represents the initial attention weight obtained by calculating the mth feature in the query sample feature representation and the nth feature of the kth sample in the cth class of the support set. m is the mth feature q in the query sample feature representation m +PE(m), L is the standard layer normalization;

[0039] is the attention weight corresponding to the feature of the m-th query sample and the l-th feature of the k-th sample of the c-th class in the support set, where l is an integer variable in the range [1, N], used to Sum, that is, for each of the given c, m, k go through Add all the calculations after ; It means that after normalization Attention weight; d k is the scaling factor, divided by d k It can alleviate the situation that the dimension of the result obtained by the dot product is too large to a certain extent, making the result in the area where the gradient of the softmax function is very small;

[0040] Similarly, W q , W k , W v is the weight matrix, is the feature of the mth frame of the kth sample of the cth class after processing;

[0041] In order to ensure the consistency of dimensions and facilitate subsequent calculations, the query sample features are passed through the transformer to generate the weight matrix W of the value. v The processing is as follows:

[0042] u m =W v *Q m

[0043] Among them, u m Indicates the query sample features with consistent dimensions after processing;

[0044] The feature set of the processed support samples is generated by the transformer It is composed of T = {T 1 ,…,T c ,…,T c}, where T c is the feature set of the c-th class support sample, expressed as in is the feature of the kth sample of the cth class, expressed as Similarly, the query sample feature U is simplified to U = {u1,…,u n ,…,u N}.

[0045] Preferably, according to the feature set of the supporting samples, for each supporting sample, the similarity between the features of the same sample is calculated, and the results are summed, averaged, and normalized to obtain the sample weight of each supporting sample, specifically:

[0046] Since the sample features are composed of multiple frame feature vectors, the dynamic time warping algorithm DTW is used to calculate the similarity between similar sample features. The frames of the two samples are time-aligned, and then the distance between the aligned frame feature vectors is calculated. Finally, the similarity between the videos is obtained through the vector distance between each video.

[0047] Specifically, the sample characteristics The mth eigenvector and sample features The cumulative distance γ between the nth eigenvectors of is:

[0048]

[0049] in, is the distance between two eigenvectors, calculated using Euclidean distance, as follows:

[0050]

[0051] Among them, x 1i With x 2i The eigenvectors are and The i-th element of , d is the dimension of the output feature, representing the length of the feature vector, that is, the number of elements contained;

[0052] According to the DTW algorithm, the distance between two sample features is the cumulative distance between the last feature vectors of the two sample features. Right now

[0053] Based on the distance D, the average distance between each video sample feature and all other video sample features of the same type is calculated, and the calculation formula is as follows:

[0054]

[0055] in, represents the distance between the sample features of the kth video in the cth category and the sample features of the ith video, is the average distance between the sample features of the kth video in the cth category and the sample features of all other videos of the same category, thereby obtaining the average distance of each video to all other videos of the same category;

[0056] Finally, the sample weight of each support sample is calculated as Based on the characteristic that "the greater the average distance between a video and all other videos of the same type, the lower the average similarity between the video and all other videos of the same type", the similarity weight Calculated by the following method:

[0057]

[0058] in, It ensures that the similarity is positive. The larger the average distance, the lower the similarity. In addition, in order to facilitate subsequent calculations, a normalized processing method is adopted. Represents the sum of the average similarities of all samples in the cth class.

[0059] Preferably, the feature set of the processed support samples and the sample weights of each support sample are combined, and in each class, all sample features are weighted summed according to the corresponding weights to construct prototypes of each class of support samples, specifically:

[0060] The calculation formula of the prototype is:

[0061]

[0062] in, is the weight of the k-th support sample of the c-th class, is the nth eigenvector of the kth support sample of the cth class;

[0063] Therefore, the class prototype of class c is The prototype set of supporting samples is P = {P 1 ,…,P c ,…,P C}.

[0064] Preferably, the input of the prototypes of each class of the support samples and the query sample features, for each pair of class prototypes and query sample features, the distance between the two is calculated, and the class with the smallest distance is taken as the prediction result, specifically:

[0065] The calculation formula of the distance between the class prototype and the query sample feature is:

[0066]

[0067] in, is the nth eigenvector of the cth prototype, u n To query the nth feature vector of the sample, the predicted category c′ is expressed as c′=argmin c D(Q,S c ), argmin c It means that when D(Q,S c ) takes the value when the minimum value c is taken;

[0068] The loss function of the few-shot action recognition model is as follows:

[0069]

[0070] Where c′ is the predicted category generated by the model from the training task, D(Q,S c ) is the distance between the query sample and the prototype of type c;

[0071] The model is trained through this loss to optimize the model parameters, and finally a few-sample action recognition model is formed. Users can use this model to monitor surveillance videos for action recognition.

[0072] Accordingly, the present invention also provides a small sample action recognition system in a smart health environment, comprising:

[0073] The frame extraction unit inputs the support set and query set containing video samples. The support set is a sample set used to learn features for the model, and the query set is a sample set used to predict the model. The frame extraction algorithm is used to extract representative frames from each video to obtain the frame sets of the support set and query set samples.

[0074] A data enhancement unit performs data enhancement on each frame image in the frame set of the support set and the query set samples, and processes them into a uniform size to obtain the frame set of the support set and the query set samples after data preprocessing;

[0075] A feature extraction unit, using a feature extraction network, extracts frame set features of the support set and query set samples after the data preprocessing, and adds position coding to the extracted features. The frame features of each sample constitute the features of the sample, and the output is a feature set of the support sample and a feature of the query sample;

[0076] The sample matching unit inputs the feature set of the support sample and the query sample features, divides the data into sample pairs of the support sample and the query sample, inputs each sample pair into the transformer for calculation, generates the support sample features that are adaptively re-weighted according to the similarity with the query sample, and outputs the processed feature set of the support sample and the query sample features;

[0077] A weight allocation unit, which calculates the similarity between the features of the same type of samples for each supporting sample according to the feature set of the supporting samples, and sums, averages and normalizes the results to obtain the sample weight of each supporting sample;

[0078] The prototype construction unit combines the processed feature set of the support samples and the sample weights of each support sample, and in each class, performs weighted summation of all sample features according to the corresponding weights to construct a prototype of each class of the support samples;

[0079] The result generating unit inputs the prototypes of each class of the supporting samples and the query sample features, calculates the distance between each pair of class prototypes and query sample features, and takes the class with the smallest distance as the prediction result.

[0080] The implementation of the present invention has the following beneficial effects:

[0081] The present invention makes full use of the differences between classes, between videos of the same type, and between frames of the same video. The features of each part are interconnected through extraction and integration, and the various information of the samples is fully utilized to make the generated action recognition results more in line with people's expectations.

[0082] The transformer method of the present invention uses an attention mechanism to match each query frame feature with all frame features in the support set, and performs weighted summation to construct a support sample specific to the query. This method is conducive to highlighting frames in the support sample that are more important to the query sample. The reconstructed support sample is more matched with the query sample and easier to compare, which greatly improves the accuracy of the result and can be used in scenes with relatively high requirements for action recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 is a flow chart of a method for identifying a few samples of actions in a smart health environment according to an embodiment of the present invention;

[0084] Figure 2 This is a structural diagram of a small sample motion recognition system in a smart health environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] Figure 1 is a flow chart of a method for identifying a few samples of actions in a smart health environment according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0087] S1, input the support set and query set containing video samples. The support set is the sample set used to learn features for the model, and the query set is the sample set used to predict the model. The representative frames are extracted from each video through the frame extraction algorithm to obtain the frame sets of the support set and query set samples;

[0088] S2, performing data enhancement on each frame image in the frame set of the support set and the query set samples, and processing them into a uniform size, so as to obtain the frame set of the support set and the query set samples after data preprocessing;

[0089] S3, using a feature extraction network to extract frame set features of the support set and query set samples after the data preprocessing, and adding position encoding to the extracted features, the frame features of each sample constitute the features of the sample, and output as a feature set of the support sample and a feature set of the query sample;

[0090] S4, input the feature set of the support sample and the query sample feature, divide the data into sample pairs of the support sample and the query sample, input each sample pair into the transformer for calculation, generate the support sample feature that is adaptively re-weighted according to the similarity with the query sample, and output the processed support sample feature set and query sample feature;

[0091] S5, according to the feature set of the supporting samples, for each supporting sample, calculate the similarity between the features of the same sample, sum, average and normalize the results, so as to obtain the sample weight of each supporting sample;

[0092] S6, combining the processed feature set of the support samples and the sample weights of each support sample, in each class, performing weighted summation of all sample features according to corresponding weights, and constructing prototypes of each class of support samples;

[0093] S7, input the prototypes of each class of support samples and query sample features, calculate the distance between each pair of class prototypes and query sample features, and take the class with the smallest distance as the prediction result.

[0094] Step S1 is as follows:

[0095] S1-1: The support set contains C×K video samples, and the query set contains a single video sample, where C and K are hyperparameters, C represents the number of classes, and K represents the number of samples in each class;

[0096] S1-2: The frame extraction algorithm uses the principle that "the more similar the frames are, the more repetitive the feature information they express. Extracting frames with lower similarity from the frame set decomposed from the video can express more feature information". The frame set is divided into N clusters using the similarity between frames. The frame in the middle of each cluster is extracted as the representative frame, thereby outputting frames with relatively more information. N is a hyperparameter, which represents the number of frames in each sample and is generally set to N=8.

[0097] Step S2 is as follows:

[0098] S2-1: The input is the frame set of support set and query set samples, and data enhancement is performed on each frame image. Generally, random flipping and random cropping are adopted during training, and center cropping is adopted during inference. At the same time, they are processed into a uniform size. Generally, the image size is proportionally adjusted to a size of 256 pixels on the short side, and then cropped to a size of 224*224 pixels during the data enhancement process. The output is the frame set of support set and query set samples after data preprocessing.

[0099] Step S3 is as follows:

[0100] S3-1: The backbone network used for feature extraction is ResNet50 pre-trained on the ImageNet dataset. Only the last fully connected layer of the backbone network is replaced so that the dimension of the output feature is the hyperparameter d=2048. During the training process, the parameters of the backbone network are frozen except for the replaced fully connected layer.

[0101] S3-2: Each support sample feature is represented as:

[0102]

[0103] The query sample features are expressed as:

[0104] Q={q1+PE(1),…,q n +PE(n),…,q N +PE(N)}

[0105] in, To support the feature information of the nth frame of the kth sample of the cth class, q n To query the feature information of the nth frame of the sample, PE is the position code, which is used to add position information to each frame feature;

[0106] S3-3: The feature set of the support sample is expressed as:

[0107] S={S 1 ,…,S c ,…,S C}

[0108] Among them, S c is the feature set of the c-th class support sample, expressed as:

[0109]

[0110] is the feature of the kth sample of the cth class, and its representation is shown in S3-1, which is further simplified as:

[0111]

[0112] Similarly, the query sample feature representation is further simplified as follows:

[0113] Q={Q1,…,Q n ,…,Q N}.

[0114] Step S4 is as follows:

[0115] S4-1: In the transformer, the supporting sample features are used as the key and value of the transformer, and the query sample features are used as the query, where query, key, and value are all feature vectors, which are obtained by passing the sample features through a specific weight matrix W q , W k , W v Generate, calculate the similarity weight between the supporting sample features and the query sample features through query and key, perform weighted summation on the value according to the weight, and finally obtain the supporting sample features that are adaptively re-weighted;

[0116] S4-2: The transformer calculation formula is as follows:

[0117]

[0118] in, is an intermediate variable, which represents the initial attention weight obtained by calculating the mth feature in the query sample feature representation and the nth feature of the kth sample in the cth class of the support set. m is the mth feature q in the query sample feature representation m +PE(m), L is the standard layer normalization;

[0119] is the attention weight corresponding to the feature of the m-th query sample and the l-th feature of the k-th sample of the c-th class in the support set, where l is an integer variable in the range [1, N], used to Sum, that is, for each of the given c, m, k go through Add all the calculations after ; It means that after normalization Attention weight; d k is the scaling factor, divided by d k It can alleviate the situation that the dimension of the result obtained by the dot product is too large to a certain extent, making the result in the area where the gradient of the softmax function is very small;

[0120] Similarly, W q , W k , W v is the weight matrix, is the feature of the mth frame of the kth sample of the cth class after processing;

[0121] S4-3: In order to ensure the consistency of dimensions and facilitate subsequent calculations, the query sample features are passed through the transformer to generate the weight matrix W of the value v The details are as follows:

[0122] u m =W v *Q m

[0123] Among them, u m Indicates the query sample features with consistent dimensions after processing;

[0124] S4-4: The feature set T of the processed support sample is generated by the transformer It is composed of T = {T 1 ,…,T c ,…,T C}, where T c is the feature set of the c-th class support sample, expressed as in is the feature of the kth sample of the cth class, expressed as Similarly, the query sample feature U is simplified to U = {u1,…,u n ,…,u N}.

[0125] Step S5 is as follows:

[0126] S5-1: Since the sample features are composed of multiple frame feature vectors, the dynamic time warping algorithm DTW is used to calculate the similarity between similar sample features. The frames of the two samples are time-aligned, and then the distance between the aligned frame feature vectors is calculated. Finally, the similarity between the videos is obtained by the vector distance between each video.

[0127] S5-2: For any two sample features and The distance between the two can be calculated by the DTW algorithm. Specifically, for the sample features The mth eigenvector and sample features The cumulative distance γ between the nth eigenvectors of is:

[0128]

[0129] in, is the distance between two eigenvectors, calculated using Euclidean distance, as follows:

[0130]

[0131] Among them, x 1i With x 2i The eigenvectors are and The i-th element of , d is the dimension of the output feature, representing the length of the feature vector, that is, the number of elements contained;

[0132] According to the DTW algorithm, the distance between two sample features is the cumulative distance between the last feature vectors of the two sample features. Right now

[0133] Based on the distance D, the average distance between each video sample feature and all other video sample features of the same type is calculated, and the calculation formula is as follows:

[0134]

[0135] in, represents the distance between the sample features of the kth video in the cth category and the sample features of the ith video, is the average distance between the sample features of the kth video in the cth category and the sample features of all other videos of the same category, thereby obtaining the average distance of each video to all other videos of the same category;

[0136] Finally, the sample weight of each support sample is calculated as Based on the characteristic that "the greater the average distance between a video and all other videos of the same type, the lower the average similarity between the video and all other videos of the same type", the similarity weight Calculated by the following method:

[0137]

[0138] in, It ensures that the similarity is positive. The larger the average distance, the lower the similarity. In addition, in order to facilitate subsequent calculations, a normalized processing method is adopted. Represents the sum of the average similarities of all samples in the cth class.

[0139] Step S6 is as follows:

[0140] S6-1: The calculation formula for the prototype is:

[0141]

[0142] in, is the weight of the k-th support sample of the c-th class, is the nth eigenvector of the kth support sample of the cth class;

[0143] Therefore, the class prototype of class c is The prototype set of supporting samples is P = {P 1 ,…,P c ,…,P C}.

[0144] Step S7 is as follows:

[0145] S7-1: The calculation formula for the distance between the class prototype and the query sample feature is:

[0146]

[0147] in, is the nth eigenvector of the cth prototype, u n To query the nth feature vector of the sample, the predicted category c′ is expressed as c′=argmin c D(Q,S c ), argmin c It means that when D(Q,S c ) takes the value when the minimum value c is taken;

[0148] S7-2: The loss function of the few-shot action recognition model is as follows:

[0149]

[0150] Where c′ is the predicted category generated by the model from the training task, D(Q,S c ) is the distance between the query sample and the prototype of type c;

[0151] The model is trained through this loss to optimize the model parameters, and finally a few-sample action recognition model is formed. Users can use this model to monitor surveillance videos for action recognition.

[0152] Accordingly, the present invention also provides a small sample action recognition system in a smart health environment, such as Figure 2 As shown, including:

[0153] The frame extraction unit 1 is used to input a support set and a query set containing video samples. The support set is a sample set used to learn features for the model, and the query set is a sample set used to predict the model. A representative frame is extracted from each video through a frame extraction algorithm to obtain a frame set of support set and query set samples.

[0154] A data enhancement unit 2 is used to perform data enhancement on each frame image in the frame set of the support set and the query set sample, and process it into a uniform size to obtain the frame set of the support set and the query set sample after data preprocessing;

[0155] The feature extraction unit 3 is used to extract the frame set features of the support set and query set samples after the data preprocessing by using the feature extraction network, and add position coding to the extracted features. The frame features of each sample constitute the features of the sample, and the output is the feature set of the support sample and the query sample feature;

[0156] The sample matching unit 4 is used to input the feature set of the support sample and the query sample feature, divide the data into sample pairs of the support sample and the query sample, input each sample pair into the transformer for calculation, generate the support sample feature adaptively re-weighted according to the similarity with the query sample, and output the processed support sample feature set and query sample feature;

[0157] The weight distribution unit 5 is used to calculate the similarity between the features of the same sample for each supporting sample according to the feature set of the supporting sample, and sum, average and normalize the results to obtain the sample weight of each supporting sample;

[0158] A prototype construction unit 6 is used to combine the processed feature set of the supporting samples and the sample weights of each supporting sample, and in each class, perform weighted summation of all sample features according to the corresponding weights to construct a prototype of each class of supporting samples;

[0159] The result generating unit 7 is used to input the prototypes of each class of the supporting samples and the query sample features, calculate the distance between each pair of class prototypes and query sample features, and take the class with the smallest distance as the prediction result.

[0160] Therefore, the present invention makes full use of the differences between classes, between videos of the same type, and between frames of the same video. The features between the various parts are interconnected through extraction and integration, making full use of various information of the sample, so that the generated action recognition results are more in line with people's expectations; in addition, the method using transformer in the present invention uses an attention mechanism to match each query frame feature with all frame features in the support set, and perform weighted summation to construct a support sample specific to the query. This method is conducive to highlighting the frames in the support sample that are more important to the query sample. The reconstructed support sample is more matched with the query sample and easier to compare, which greatly improves the accuracy of the result and can be used in scenes with relatively high requirements for action recognition accuracy.

[0161] The above is a detailed introduction to a method and system for identifying a small number of motion samples in a smart health environment provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and scope of application. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for few-sample action recognition in a smart health environment, characterized in that: The method comprises: Input the support set and query set containing video samples. The support set is the sample set used to learn features for the model, and the query set is the sample set used to predict for the model. The representative frames are extracted from each video through the frame extraction algorithm to obtain the frame sets of the support set and query set samples. Performing data enhancement on each frame image in the frame set of the support set and the query set samples, and processing them into a uniform size, so as to obtain the frame set of the support set and the query set samples after data preprocessing; Using a feature extraction network, extract the frame set features of the support set and query set samples after the data preprocessing, and add position encoding to the extracted features. The frame features of each sample constitute the features of the sample, and the output is a feature set of the support sample and a feature set of the query sample; Input the feature set of the support sample and the feature of the query sample, divide the data into sample pairs of the support sample and the query sample, input each sample pair into the transformer for calculation, generate the support sample feature that is adaptively re-weighted according to the similarity with the query sample, and output the processed feature set of the support sample and the query sample feature; According to the feature set of the supporting samples, for each supporting sample, the similarity between the features of the same sample is calculated, and the results are summed, averaged, and normalized to obtain the sample weight of each supporting sample; Combining the processed feature set of the support samples and the sample weights of each support sample, in each class, all sample features are weighted summed according to the corresponding weights to construct prototypes of each class of support samples; Input the prototypes of each class of support samples and the query sample features, calculate the distance between each pair of class prototypes and query sample features, and take the class with the smallest distance as the prediction result.

2. The method for identifying a small number of actions in a smart health environment according to claim 1, characterized in that: The input includes a support set and a query set of video samples. The support set is a sample set used to learn features for the model, and the query set is a sample set used to predict the model. Representative frames are extracted from each video through a frame extraction algorithm to obtain frame sets of support set and query set samples, specifically: The support set contains C×K video samples, and the query set contains a single video sample, where C and K are hyperparameters, C represents the number of classes, and K represents the number of samples in each class; The frame extraction algorithm uses the similarity between frames to divide the frame set into N clusters, and extracts the frame in the middle position from each cluster as the representative frame, thereby outputting frames with relatively more information, where N is a hyperparameter representing the number of frames for each sample.

3. The method for identifying a small number of actions in a smart health environment according to claim 1, characterized in that: The feature extraction network is used to extract the frame set features of the support set and query set samples after the data preprocessing, and the extracted features are encoded with positions. The frame features of each sample constitute the features of the sample, and the output is the feature set of the support sample and the query sample feature, which is specifically: The backbone network used for feature extraction is ResNet50 pre-trained on the ImageNet dataset. Only the last fully connected layer of the backbone network is replaced so that the dimension of the output feature is the hyperparameter d. During the training process, except for the replaced fully connected layer, the parameters of the backbone network are frozen. Each support sample feature is expressed as: The query sample features are expressed as: Q={q1+PE(1),…,q n +PE(n),…,q N +PE(N)} in, To support the feature information of the nth frame of the kth sample of the cth class, q n To query the feature information of the nth frame of the sample, PE is the position code, which is used to add position information to each frame feature; The feature set of the support sample is expressed as: S={S 1 ,…,S c ,…,S c } Among them, S c is the feature set of the c-th class support sample, expressed as: For each supporting sample feature, it is further simplified as: Similarly, the query sample feature representation is further simplified as follows: Q={Q1,…,Q n ,…,Q N }。 4. The method for identifying a small number of actions in a smart health environment according to claim 3, characterized in that: The input support sample feature set and query sample feature are divided into sample pairs of support samples and query samples, each sample pair is input into the transformer for calculation, and support sample features adaptively re-weighted according to the similarity with the query sample are generated, and the processed support sample feature set and query sample feature are output, which are specifically: In the transformer, the supporting sample features are used as the key and value of the transformer, and the query sample features are used as the query, where query, key, and value are all feature vectors, which are obtained by passing the sample features through a specific weight matrix W. q , W k , W v Generate, calculate the similarity weight between the supporting sample features and the query sample features through query and key, perform weighted summation on the value according to the weight, and finally obtain the supporting sample features that are adaptively re-weighted; The transformer calculation formula is as follows: in, is an intermediate variable, which represents the initial attention weight obtained by calculating the mth feature in the query sample feature representation and the nth feature of the kth sample in the cth class of the support set. m is the mth feature q in the query sample feature representation m +PE(m), L is the standard layer normalization; is the attention weight corresponding to the feature of the m-th query sample and the l-th feature of the k-th sample of the c-th class in the support set, where l is an integer variable in the range [1, N], used to Sum, that is, for each of the given c, m, k go through Add all the calculations after ; It means that after normalization Attention weight; d k is the scaling factor, divided by d k It can alleviate the situation that the dimension of the result obtained by the dot product is too large to a certain extent, making the result in the area where the gradient of the softmax function is very small; Similarly, W q , W k , W v is the weight matrix, is the feature of the mth frame of the kth sample of the cth class after processing; In order to ensure the consistency of dimensions and facilitate subsequent calculations, the query sample features are passed through the transformer to generate the weight matrix W of the value. v The details are as follows: u m =W v *Q m Among them, u m Indicates the query sample features with consistent dimensions after processing; The feature set of the processed support samples is generated by the transformer It is composed of T = {T 1 ,…,T c ,…,T c }, where T c is the feature set of the c-th class support sample, expressed as in is the feature of the kth sample of the cth class, expressed as Similarly, the query sample feature U is simplified to U = {u1,…,u n ,…,u N }.

5. The method for identifying a small number of actions in a smart health environment according to claim 4, characterized in that: According to the feature set of the supporting samples, for each supporting sample, the similarity between the features of the same sample is calculated, and the results are summed, averaged, and normalized to obtain the sample weight of each supporting sample, which is specifically: Since the sample features are composed of multiple frame feature vectors, the dynamic time warping algorithm DTW is used to calculate the similarity between similar sample features. The frames of the two samples are time-aligned, and then the distance between the aligned frame feature vectors is calculated. Finally, the similarity between the videos is obtained through the vector distance between each video. Specifically, the sample characteristics The mth eigenvector and sample features The cumulative distance γ between the nth eigenvectors of is: in, is the distance between two eigenvectors, calculated using Euclidean distance, as follows: Among them, x 1i With x 2i The eigenvectors are and The i-th element of , d is the dimension of the output feature, representing the length of the feature vector, that is, the number of elements contained; According to the DTW algorithm, the distance between two sample features is the cumulative distance between the last feature vectors of the two sample features. Right now Based on the distance D, the average distance between each video sample feature and all other video sample features of the same type is calculated, and the calculation formula is as follows: in, represents the distance between the sample features of the kth video in the cth category and the sample features of the ith video, is the average distance between the sample features of the kth video in the cth category and the sample features of all other videos of the same category, thereby obtaining the average distance of each video to all other videos of the same category; Finally, the sample weight of each support sample is calculated as Similarity Weight Calculated by the following method: in, It ensures that the similarity is positive. The larger the average distance, the lower the similarity. In addition, in order to facilitate subsequent calculations, a normalized processing method is adopted. Represents the sum of the average similarities of all samples in the cth class.

6. A method for identifying a small number of actions in a smart health environment as claimed in claim 5, characterized in that: The processing of the feature set of the supporting samples and the sample weights of the supporting samples is combined, and in each class, all sample features are weighted and summed according to the corresponding weights to construct prototypes of each class of supporting samples, specifically: The calculation formula of the prototype is: in, is the weight of the k-th support sample of the c-th class, is the nth eigenvector of the kth support sample of the cth class; Therefore, the class prototype of class c is The prototype set of supporting samples is P = {P 1 ,…,P c ,…,P C }.

7. The method for identifying a small number of actions in a smart health environment according to claim 6, characterized in that: The input of the prototypes of the supporting samples and the query sample features, for each pair of class prototypes and query sample features, the distance between the two is calculated, and the class with the smallest distance is taken as the prediction result, specifically: The calculation formula of the distance between the class prototype and the query sample feature is: in, is the nth eigenvector of the cth prototype, u n To query the nth feature vector of the sample, the predicted category c′ is expressed as c′=argmin c D(Q,S c ), argmin c It means that when D(Q,S c ) takes the value when the minimum value c is taken; The loss function of the few-shot action recognition model is as follows: Where c′ is the predicted category generated by the model from the training task, D(Q,S c ) is the distance between the query sample and the prototype of type c; The model is trained through this loss to optimize the model parameters, and finally a few-sample action recognition model is formed. Users can use this model to monitor surveillance videos for action recognition.

8. A small sample action recognition system in a smart health environment, characterized in that: The system comprises: The frame extraction unit inputs the support set and query set containing video samples. The support set is a sample set used to learn features for the model, and the query set is a sample set used to predict the model. The frame extraction algorithm is used to extract representative frames from each video to obtain the frame sets of the support set and query set samples. A data enhancement unit performs data enhancement on each frame image in the frame set of the support set and the query set samples, and processes them into a uniform size to obtain the frame set of the support set and the query set samples after data preprocessing; A feature extraction unit, using a feature extraction network, extracts frame set features of the support set and query set samples after the data preprocessing, and adds position coding to the extracted features. The frame features of each sample constitute the features of the sample, and the output is a feature set of the support sample and a feature of the query sample; The sample matching unit inputs the feature set of the support sample and the query sample features, divides the data into sample pairs of the support sample and the query sample, inputs each sample pair into the transformer for calculation, generates the support sample features that are adaptively re-weighted according to the similarity with the query sample, and outputs the processed feature set of the support sample and the query sample features; A weight allocation unit, which calculates the similarity between the features of the same type of samples for each supporting sample according to the feature set of the supporting samples, and sums, averages and normalizes the results to obtain the sample weight of each supporting sample; The prototype construction unit combines the processed feature set of the support samples and the sample weights of each support sample, and in each class, performs weighted summation of all sample features according to the corresponding weights to construct a prototype of each class of the support samples; The result generating unit inputs the prototypes of each class of the supporting samples and the query sample features, calculates the distance between each pair of class prototypes and query sample features, and takes the class with the smallest distance as the prediction result.

9. A system for identifying action with a small number of samples in a smart health environment as claimed in claim 8, characterized in that: The input includes a support set and a query set of video samples. The support set is a sample set used to learn features for the model, and the query set is a sample set used to predict the model. Representative frames are extracted from each video through a frame extraction algorithm to obtain frame sets of support set and query set samples, specifically: The support set contains C×K video samples, and the query set contains a single video sample, where C and K are hyperparameters, C represents the number of classes, and K represents the number of samples in each class; The frame extraction algorithm uses the similarity between frames to divide the frame set into N clusters, and extracts the frame in the middle position from each cluster as the representative frame, thereby outputting frames with relatively more information, where N is a hyperparameter representing the number of frames for each sample.

10. A system for identifying action with a small number of samples in a smart health environment as claimed in claim 8, characterized in that: The feature extraction network is used to extract the frame set features of the support set and query set samples after the data preprocessing, and the extracted features are encoded with positions. The frame features of each sample constitute the features of the sample, and the output is the feature set of the support sample and the query sample feature, which is specifically: The backbone network used for feature extraction is ResNet50 pre-trained on the ImageNet dataset. Only the last fully connected layer of the backbone network is replaced so that the dimension of the output feature is the hyperparameter d. During the training process, except for the replaced fully connected layer, the parameters of the backbone network are frozen. Each support sample feature is expressed as: The query sample features are expressed as: Q={q1+PE(1),…,q n +PE(n),…,q N +PE(N)} in, To support the feature information of the nth frame of the kth sample of the cth class, q n To query the feature information of the nth frame of the sample, PE is the position code, which is used to add position information to each frame feature; The feature set of the support sample is expressed as: S={S 1 ,…,S c ,…,S c } Among them, S c is the feature set of the c-th class support sample, expressed as: For each supporting sample feature, it is further simplified as: Similarly, the query sample feature representation is further simplified as follows: Q={Q1,…,Q n ,…,Q N }。 11. A system for identifying action in a smart health environment using a small number of samples as claimed in claim 10, characterized in that: The input support sample feature set and query sample feature are divided into sample pairs of support samples and query samples, each sample pair is input into the transformer for calculation, and support sample features adaptively re-weighted according to the similarity with the query sample are generated, and the processed support sample feature set and query sample feature are output, which are specifically: In the transformer, the supporting sample features are used as the key and value of the transformer, and the query sample features are used as the query, where query, key, and value are all feature vectors, which are obtained by passing the sample features through a specific weight matrix W. q , W k , W v Generate, calculate the similarity weight between the supporting sample features and the query sample features through query and key, perform weighted summation on the value according to the weight, and finally obtain the supporting sample features that are adaptively re-weighted; The transformer calculation formula is as follows: in, is an intermediate variable, which represents the initial attention weight obtained by calculating the mth feature in the query sample feature representation and the nth feature of the kth sample in the cth class of the support set. m is the mth feature q in the query sample feature representation m +PE(m), L is the standard layer normalization; is the attention weight corresponding to the feature of the m-th query sample and the l-th feature of the k-th sample of the c-th class in the support set, where l is an integer variable in the range [1, N], used to Sum, that is, for each of the given c, m, k go through Add all the calculations after ; It means that after normalization Attention weight; d k is the scaling factor, divided by d k It can alleviate the situation that the dimension of the result obtained by the dot product is too large to a certain extent, making the result in the area where the gradient of the softmax function is very small; Similarly, W q , W k , W v is the weight matrix, is the feature of the mth frame of the kth sample of the cth class after processing; In order to ensure the consistency of dimensions and facilitate subsequent calculations, the query sample features are passed through the transformer to generate the weight matrix W of the value. v The details are as follows: u m =W v *Q m Among them, u m Indicates the query sample features with consistent dimensions after processing; The feature set of the processed support samples is generated by the transformer It is composed of T = {T 1 ,…,T c ,…,T c }, where T c is the feature set of the c-th class support sample, expressed as in is the feature of the kth sample of the cth class, expressed as Similarly, the query sample feature U is simplified to U = {u1,…,u n ,…,u N }.

12. A system for identifying action in a smart health environment using a small number of samples as claimed in claim 11, characterized in that: According to the feature set of the supporting samples, for each supporting sample, the similarity between the features of the same sample is calculated, and the results are summed, averaged, and normalized to obtain the sample weight of each supporting sample, which is specifically: Since the sample features are composed of multiple frame feature vectors, the dynamic time warping algorithm DTW is used to calculate the similarity between similar sample features. The frames of the two samples are time-aligned, and then the distance between the aligned frame feature vectors is calculated. Finally, the similarity between the videos is obtained through the vector distance between each video. Specifically, the sample characteristics The mth eigenvector and sample features The cumulative distance γ between the nth eigenvectors of is: in, is the distance between two eigenvectors, calculated using Euclidean distance, as follows: Among them, x 1i With x 2i The eigenvectors are and The i-th element of , d is the dimension of the output feature, representing the length of the feature vector, that is, the number of elements contained; According to the DTW algorithm, the distance between two sample features is the cumulative distance between the last feature vectors of the two sample features. Right now Based on the distance D, the average distance between each video sample feature and all other video sample features of the same type is calculated, and the calculation formula is as follows: in, represents the distance between the sample features of the kth video in the cth category and the sample features of the ith video, is the average distance between the sample features of the kth video in the cth category and the sample features of all other videos of the same category, thereby obtaining the average distance of each video to all other videos of the same category; Finally, the sample weight of each support sample is calculated as Similarity Weight Calculated by the following method: in, It ensures that the similarity is positive. The larger the average distance, the lower the similarity. In addition, in order to facilitate subsequent calculations, a normalized processing method is adopted. Represents the sum of the average similarities of all samples in the cth class.

13. A system for identifying action with a small number of samples in a smart health environment as claimed in claim 12, characterized in that: The processing of the feature set of the supporting samples and the sample weights of the supporting samples is combined, and in each class, all sample features are weighted and summed according to the corresponding weights to construct prototypes of each class of supporting samples, specifically: The calculation formula of the prototype is: in, is the weight of the k-th support sample of the c-th class, is the nth eigenvector of the kth support sample of the cth class; Therefore, the class prototype of class c is The prototype set of supporting samples is P = {P 1 ,…,P c ,…,P C }.

14. A system for identifying action with a small number of samples in a smart health environment as claimed in claim 13, characterized in that: The input of the prototypes of the supporting samples and the query sample features, for each pair of class prototypes and query sample features, the distance between the two is calculated, and the class with the smallest distance is taken as the prediction result, specifically: The calculation formula of the distance between the class prototype and the query sample feature is: in, is the nth eigenvector of the cth prototype, u n To query the nth feature vector of the sample, the predicted category c′ is expressed as c′=argmin c D(Q,S c ), argmin c It means that when D(Q,S c ) takes the value when the minimum value c is taken; The loss function of the few-shot action recognition model is as follows: Where c′ is the predicted category generated by the model from the training task, D(Q,S c ) is the distance between the query sample and the prototype of type c; The model is trained through this loss to optimize the model parameters, and finally a few-sample action recognition model is formed. Users can use this model to monitor surveillance videos for action recognition.

Images