Small sample class increment gait recognition method in multi-activity scene

Abstract

The invention discloses a small sample class increment gait recognition method in a multi-activity scene, and provides a gait recognition framework based on hybrid prototype enhancement in order to solve the problems existing in small sample class increment gait recognition in the multi-activity scene. First, the framework generates a hybrid prototype by introducing an ancillary activity tag, which prototype is more generalizable than a common prototype. And secondly, the prototype is adjusted through a selective prototype enhancement module based on an attention mechanism, so that the characterization capability and the discrimination capability of the prototype are improved. And finally, experimental verification is carried out on the public gait data set USC-HAD and the gait data set CDUT-AG collected by the user, and the result shows that the proposed method achieves the best effect on the small sample class increment gait recognition problem under the multi-activity scene, the problem of disastrous forgetting is effectively solved, and the effectiveness of the proposed method is verified.

Description

Technical Field

[0001] The present invention belongs to the technical field of gait recognition, and specifically relates to a small sample incremental gait recognition method in multiple activity scenarios. The proposed method is implemented based on a hybrid prototype enhanced model framework. Background Art

[0002] Gait recognition is a technology that identifies individuals by analyzing the motion patterns presented by human walking. Many studies have shown that each person's gait is unique, so the purpose of identity authentication can be achieved through gait data. In recent years, with the rapid development of technologies related to personal devices and wearable devices, gait recognition technology based on smart devices has received widespread attention. It usually embeds inertial sensors such as accelerometers and gyroscopes into portable smart devices. Due to its portability and low resource consumption, it can easily obtain motion data in daily life and use it for continuous user identification or verification. In recent years, deep learning technology has promoted the rapid development of the field of gait recognition. Deep learning models can effectively extract and learn gait features to achieve individual identity recognition. However, compared with humans, conventional deep neural networks cannot accumulate experience and retain knowledge, so they are not applicable when processing new users and new walking activity data.

[0003] Natural learning systems are progressive, and they can continuously absorb new knowledge while maintaining existing knowledge and continuously improving and expanding their cognitive capabilities. In real-world scenarios, gait recognition requires continuous learning capabilities, that is, the gait recognition system can continuously expand and adapt to new users while maintaining the recognition performance of old users that have been learned. However, traditional deep learning models are usually trained through offline batch learning, which means that the entire model needs to be retrained using all existing samples. This leads to high requirements for computing resources and time when learning new gait samples. When faced with new user gait data encountered in a continuous gait data stream, conventional methods cannot achieve "continuous" learning by retraining the model on all new and old user data. In this case, simply retraining the model in the face of new user gait data in a continuous gait data stream may lead to catastrophic forgetting of existing knowledge. Some research works have explored incremental learning of gait to meet the needs of new user registration. In addition, since collecting and annotating gait data is an expensive and time-consuming task, gait recognition in scenarios with a small number of samples has become a research focus.

[0004] In addition, in real scenarios, people not only walk on flat roads, but also walk on slopes, go up and down stairs, etc. Although the same person may have the same gait characteristics in different walking activities, different walking activities tend to show obvious differences in gait characteristics, which means that the model trained using only the user gait data of a single activity scenario cannot be generalized to other activity scenarios.

[0005] In order to process the continuous information flow in the real world, incremental learning-based methods have been proposed. These methods have the ability to maintain the old knowledge that has been learned while learning new knowledge. At the same time, incremental learning methods have also been applied to fields such as gait recognition. Although gait recognition methods using incremental learning can deal with the problem of continuous data flow well, most studies still need to label a large number of samples to retrain the model when identifying new data or new categories, which is obviously not feasible in practical applications, especially for realizing personnel recognition in terminal devices with limited computing and storage resources.

[0006] In the process of gait recognition, small sample incremental learning is a kind of learning based on incremental learning, which takes into account the situation that the number of samples of new tasks is small, that is, small sample learning. It is more in line with real-world scenarios, but also more difficult. Due to the introduction of new categories, the model can only use samples of new categories to update parameters, which makes it easy for the network to forget the category information that has been learned during the update process, causing catastrophic forgetting. In addition, the incremental stage uses a small sample training strategy with a small number of samples in each category. Although the network can classify these small number of new category samples well during the training process, the classification performance of these categories will be significantly reduced on the test set, indicating that the model has overfitting problems. Catastrophic forgetting and overfitting are two important issues that limit the development of small sample incremental gait recognition in multi-activity scenarios. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the small sample incremental gait recognition method in multiple activity scenarios provided by the present invention solves the problem that in the existing gait recognition process, the gait recognition model in a single scenario is difficult to accurately identify users with a small sample amount of gait data in multiple activity scenarios.

[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a small sample incremental gait recognition method in multiple activity scenarios, comprising the following steps:

[0009] S1, collect gait samples in multiple activity scenarios and build a data set;

[0010] The labels of gait samples in the dataset include user labels and activity labels;

[0011] S2, constructing a model framework based on hybrid prototype enhancement, including a feature extractor and a selected prototype enhancement module connected in sequence;

[0012] S3. Pre-train the feature extractor of the model framework using the data set to obtain a pre-trained model;

[0013] S4. Based on the pre-trained model, incremental training is performed on the model framework, including:

[0014] Using the pre-trained model to extract features of each support sample in the data set, and forming an initial prototype of each class according to the category; the initial prototype is a mixed prototype including user labels and activity labels;

[0015] Inputting the initial prototype into the selected prototype enhancement module for incremental training to obtain an enhanced prototype, and then obtaining a gait recognition model;

[0016] S5. For the query sample of the gait to be recognized input into the gait recognition model, the corresponding initial prototype is obtained through the feature extractor with frozen parameters, which is merged with all the prototypes in the previous session and the prototype is enhanced. Then, the user category of the query sample is identified by calculating the distance between the query sample and the enhanced prototype to complete the gait recognition.

[0017] Furthermore, in step S4, the method of forming the initial prototype of each class according to the category is specifically as follows:

[0018] S401, combine m different categories of activity tags and n different categories of user tags in the data set to form a combination class Y={y i}and i∈[0,m*n);

[0019] S402: Obtain the initial prototype of each class by calculating the embedding mean of each support sample in all support sample sets in the data set.

[0020] Furthermore, in step S4, the initial prototype is represented as:

[0021]

[0022] Where P b represents the initial prototype of the combination class b, S b represents the set of support samples belonging to category b, |S b | represents the number of support samples, f b (·) is the pre-trained feature extractor, x i Indicates the support sample, y i Represents the category label.

[0023] Furthermore, in step S4, in each incremental session, the method for performing incremental training based on the hybrid prototype is specifically as follows:

[0024] S411, generating a mixed prototype of a new category support sample through a feature extractor, merging it into a prototype container of the previous session, and forming all mixed prototypes in the current prototype container into a support prototype of the current incremental session;

[0025] S412, extracting a hybrid prototype of the query sample through a feature extractor Input to the selected prototype enhancement module for evaluation;

[0026] S413, in the prototype enhancement module, the mixed prototype is calibrated by a 1×1 convolution kernel with a learning parameter q Processing to obtain a new feature map

[0027] Mixed prototypes with the same user or activity label are transformed into Merge to get a new feature map and

[0028] S414, according to the characteristic graph and The features in , construct the maximum attention weight matrix A;

[0029] S415, the maximum attention weight matrix A and the feature map Multiply and add mixed prototypes As the residual term, the enhanced prototype is obtained.

[0030] Furthermore, in step S413, the feature map and It is expressed as:

[0031]

[0032] In the formula, s q (·),s k (·),s v (·) represents a linear transformation function with q, k, and v as parameters, Represents a hybrid prototype The new prototype obtained by averaging the prototypes labeled with y y represents a user tag or activity tag, and the final prototype Represents one of the user prototype or activity prototype, and T represents the transposition operation.

[0033] Furthermore, in step S414, the weight element α of the maximum attention weight matrix A is ij It is expressed as:

[0034]

[0035] In the formula, and Respectively and The features at the i-th and j-th positions in Represents the original attention matrix The index of the maximum value of the attention score on the first dimension, is the scaling factor and T represents the transpose operation.

[0036] Further, the selection prototype enhancement module includes an activity enhancement module SPEM-SUDA and a user enhancement module SPEM-SADU;

[0037] The enhanced prototype It is a prototype obtained by adjusting the activity enhancement module SPEM-SUDA and the user module SPEM-SADU and performing feature fusion, which is expressed as:

[0038]

[0039] In the formula, and They represent the prototype features after being adjusted by the activity enhancement module SPEM-SUDA and the user enhancement module SPEM-SADU respectively.

[0040] Furthermore, in step S4, the loss function of the selected prototype enhancement module is express:

[0041]

[0042] In the formula, and They represent the loss functions of the activity enhancement module SPEM-SUDA and the user enhancement module SPEM-SADU, and Respectively Decomposed support prototype and query prototype, cos(·,·) represents the cosine classifier, represents the cross entropy function, and R represents batch data.

[0043] The beneficial effects of the present invention are:

[0044] In this paper, a gait recognition framework based on hybrid prototype enhancement is proposed to solve the problems existing in small-sample incremental gait recognition in multi-activity scenarios. First, the framework generates hybrid prototypes by introducing auxiliary activity labels, and adjusts the prototypes by enhancing user information and activity information through the selective prototype enhancement module. The prototype is more generalized than the ordinary prototype, thereby improving the representation and discrimination capabilities of the prototype. Secondly, the prototype is adjusted through the selective prototype enhancement module based on the attention mechanism in the framework, thereby improving the representation and discrimination capabilities of the prototype. Finally, experimental verification is carried out on the public gait dataset USC-HAD and the self-collected gait dataset CDUT-AG. The results show that the proposed method achieves the best results in the problem of small-sample incremental gait recognition in multi-activity scenarios, effectively solves the problem of catastrophic forgetting, and verifies the effectiveness of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of the incremental difference recognition method for small sample classes in multi-activity scenarios provided by the present invention.

[0046] Figure 2 This is a structural block diagram of the model framework based on hybrid prototype enhancement (FC-MAGR) provided by the present invention.

[0047] Figure 3 Schematic representation of different prototypes provided for the present invention.

[0048] Figure 4 This is a schematic diagram of the selected prototype enhancement module provided by the present invention.

[0049] Figure 5 The comparison results provided by the present invention on the USC-HAD dataset and the CDUT-AG dataset.

[0050] Figure 6 The comparison results provided by the present invention on the USC-HAD dataset and the CDUT-AG dataset.

[0051] Figure 7 The classification results of the method provided by the present invention and the comparative method CEC after limiting different numbers of activities in the USC-HAD dataset.

[0052] Figure 8 The classification results of the method provided by the present invention and the comparative method CEC after limiting different numbers of activities in the CDUT-AG dataset. DETAILED DESCRIPTION

[0053] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0054] The embodiment of the present invention provides a small sample incremental gait recognition method in multiple activity scenarios. Figure 1 As shown, it includes the following steps:

[0055] S1, collect gait samples in multiple activity scenarios and build a data set;

[0056] The labels of gait samples in the dataset include user labels and activity labels;

[0057] S2, constructing a model framework based on hybrid prototype enhancement, including a feature extractor and a selected prototype enhancement module connected in sequence;

[0058] S3. Pre-train the feature extractor of the model framework using the data set to obtain a pre-trained model;

[0059] S4. Based on the pre-trained model, incremental training is performed on the model framework, including:

[0060] Using the pre-trained model to extract features of each support sample in the data set, and forming an initial prototype of each class according to the category; the initial prototype is a mixed prototype including user labels and activity labels;

[0061] Inputting the initial prototype into the selected prototype enhancement module for incremental training to obtain an enhanced prototype, and then obtaining a gait recognition model;

[0062] S5. For the query sample of the gait to be recognized input into the gait recognition model, the corresponding initial prototype is obtained through the feature extractor with frozen parameters, which is merged with all the prototypes in the previous session and the prototype is enhanced. Then, the user category of the query sample is identified by calculating the distance between the query sample and the enhanced prototype to complete the gait recognition.

[0063] The gait recognition method provided by the embodiment of the present invention mainly realizes identity recognition in different activity scenarios, and its purpose is to identify the user with a small number of gait samples in multiple activity scenarios, and will not forget the old users that have been recognized. The model framework based on hybrid prototype enhancement (FC-MAGR) provided in this embodiment consists of multiple sessions, and the sessions are carried out in sequence. In each session, only the training data of the current session can be used to adjust the training model, that is, all samples of previous sessions are unavailable. At the same time, in the evaluation stage after the training of each session is completed, the test samples of all categories of the session and all previous sessions will be used for evaluation.

[0064] Based on this, in step S1 of the embodiment of the present invention, the constructed data set is divided into a training set and a test set and then allocated to multiple sessions, wherein Represents the training set data used in each session, Represents the test set data used in multiple sessions. and Samples in and Belong to the label space Y l , but the samples do not overlap. Assume that the label space of session l is represented by Y l , and i≠j, Specifically, in the first session A large-scale dataset with enough categories and enough data samples for each category is used. In addition, in subsequent incremental sessions In this embodiment, data is organized in the form of N-way-K-shot, that is, the support set of the current incremental session has only N categories, each category has only K data samples, and the query set contains several data samples of the N categories, and the samples of the support set and the query set do not intersect.

[0065] In the embodiment of the present invention, the model framework based on hybrid prototype enhancement is constructed as follows Figure 2 As shown, in Figure 2 In (a), the model architecture is pre-trained to learn feature representations with discriminative and generalization capabilities. By pre-training on large-scale unlabeled data, the network can learn useful general abstract features of the input data. Figure 2In (b), the features of each support sample are first extracted using the pre-trained model, and the basic prototypes of each class are formed according to the category. However, these basic prototypes are obtained through basic learning, so they lack sufficient feature representation to describe the category. Therefore, this embodiment proposes a selected prototype enhancement module (SPEM) to solve the problem of redundancy in the learned features. In the subsequent incremental session training process, the feature extractor is frozen, Figure 2 (c) shows the entire process of the incremental stage. In a new session, the model learns the initial prototypes of the supported samples and updates them to the prototype container, only adding prototypes of new categories.

[0066] Based on Figure 2 In the framework structure shown in FIG. 1 , in step S4 of the embodiment of the present invention, the method for forming the initial prototype of each class according to the category is specifically as follows:

[0067] S401, combine m different categories of activity tags and n different categories of user tags in the data set to form a combination class Y={y i}and i∈[0,m*n);

[0068] S402: Obtain the initial prototype of each class by calculating the embedding mean of each support sample in all support sample sets in the data set.

[0069] In step S4 of this embodiment, the initial prototype is represented as:

[0070]

[0071] Where P b represents the initial prototype of the combination class b, S b represents the set of support samples belonging to category b, |S b | represents the number of support samples, f b (·) is the pre-trained feature extractor, x i Indicates the support sample, y i Represents the category label.

[0072] Specifically, this embodiment designs a pseudo-class incremental learning scheme based on a hybrid prototype of users and activities. In this process, gait samples with dual labels of users and activities are used at the same time. Assume that the dataset has m different categories of activity labels, whose set is A = {a1, a2, ..., a m}, and has n user tags of different categories, whose set is U = {u1,u2,...,u nThe ultimate goal of the above model framework is to achieve user classification. However, in the training process, activity labels are used as prior knowledge to add to the training. It is hoped that the model's recognition of different activities can increase the model's recognition of users. Therefore, the above method is used in this embodiment to generate the initial prototype.

[0073] Compared with the prototype generated by the traditional prototype network, the initial prototype generated by the method of this embodiment The additional activity information in the gait recognition makes up for the shortcomings of the basic user prototype. For gait recognition, the walking place or walking activity category is the main factor affecting the gait pattern. The gait patterns of the same user in different activity scenes are quite different, while the gaits of different users in the same activity scene often show a certain degree of similarity. Considering the above problems, using only user category information often cannot obtain a satisfactory identification prototype. Therefore, we recommend using a hybrid prototype. It uses gait activity information as a supplement to better serve our final decision.

[0074] In step S4 of the embodiment of the present invention, the gait data is classified into possible prototypes according to different classification targets, such as Figure 3 As shown, the prototypes of different activities in (a) are relatively clearly distinguished, but cannot provide a basis for user classification. In (b), the user prototypes seem to be distinct, but the supporting activities or the supporting activities of the same category of different user prototypes may overlap partially, and this overlapping sample may lead to decision errors. (c) is the hybrid prototype proposed by the present invention, which takes into account both the distinction between different user prototypes and the overlap of different prototypes in the same activity, thus presenting a more obvious decision boundary in the recognition task.

[0075] In an embodiment of the present invention, the generated hybrid prototype is used as the input of the Selective Prototype Enhancement Module (SPEM), and is processed by the self-attention mechanism to obtain attention maps that highlight important local features. These attention maps calculate the importance scores between and within classes, selectively amplify the feature information of the target category, and finally obtain discriminative prototype features.

[0076] In step S4 of the embodiment of the present invention, based on Figure 4 The structure of the selected prototype enhancement module is shown. In each incremental session, the method of incremental training based on the mixed prototype is as follows:

[0077] S411, generating a mixed prototype of a new category support sample through a feature extractor, merging it into a prototype container of the previous session, and forming all mixed prototypes in the current prototype container into a support prototype of the current incremental session;

[0078] S412, extracting a hybrid prototype of the query sample through a feature extractor Input to the selected prototype enhancement module for evaluation;

[0079] S413, in the prototype enhancement module, the mixed prototype is calibrated by a 1×1 convolution kernel with a learning parameter q Processing to obtain a new feature map

[0080] Mixed prototypes with the same user or activity label are transformed into Merge to get a new feature map and

[0081] S414, according to the characteristic graph and The features in , construct the maximum attention weight matrix A;

[0082] S415, the maximum attention weight matrix A and the feature map Multiply and add mixed prototypes As the residual term, the enhanced prototype is obtained.

[0083] In step S413 of this embodiment, the feature map and It is expressed as:

[0084]

[0085] In the formula, s q (·),s k (·),s v (·) represents a linear transformation function with q, k, and v as parameters, Represents a hybrid prototype The new prototype obtained by averaging the prototypes labeled with y y represents a user tag or activity tag, and the final prototype Represents one of the user prototype or activity prototype, and T represents the transposition operation.

[0086] In step S414 of this embodiment, the new prototype is a prototype with more compact information, which is more biased towards our target category - user category, or auxiliary category - activity category. Therefore, in order to make full use of the hybrid prototype and new prototypes , calculate a maximum attention weight matrix A to obtain this bias information, and convert the weight element α of the maximum attention weight matrix A ij It is expressed as:

[0087]

[0088] In the formula, and Respectively and The features at the i-th and j-th positions in Represents the original attention matrix The index of the maximum value of the attention score on the first dimension, is the scaling factor and T represents the transpose operation.

[0089] In this embodiment, by adding the mixed prototype As a residual term, it prevents the problem of gradient disappearance and gradient explosion.

[0090] In this embodiment, the selection prototype enhancement module includes an activity enhancement module SPEM-SUDA and a user enhancement module SPEM-SADU;

[0091] The enhanced prototype in this embodiment It is a prototype obtained by adjusting the activity enhancement module SPEM-SUDA and the user enhancement module SPEM-SADU and performing feature fusion, which is expressed as:

[0092]

[0093] In the formula, and They represent the prototype features after being adjusted by the activity enhancement module SPEM-SUDA and the user enhancement module SPEM-SADU respectively.

[0094] Specifically, in the above process, there are two types of labels: user and activity. Therefore, after adding the prior knowledge of specific labels, the original hybrid prototype is converted into a corresponding user prototype or activity prototype according to the corresponding label. Under the action of the maximum attention weight matrix A, the SPEM module aggregates the prototype features of the same user under different activities into the attention map, which can highlight and enhance the prototypes with important features and obtain the first adjusted prototype. On the other hand, we can also use activity prototypes to adjust This is equivalent to the SPEM module highlighting the prototype features of different users in the same activity to obtain the second adjusted prototype. In order to improve the resolution of the prototype, the prototype and prototype The added fusion features are used as the final prototype.

[0095] In this embodiment, in order to achieve the best effect of SPEM on prototype enhancement, the SPEM-SADU and SPEM-SUDA branches are trained respectively using cross entropy loss and cosine classifier, and finally the respective losses are added together as our final loss. That is, in step S4 of this embodiment, the loss function of the prototype enhancement module is selected. express:

[0096]

[0097] In the formula, and They represent the loss functions of the activity module SPEM-SUDA and the user enhancement module SPEM-SADU, and Respectively Decomposed support prototype and query prototype, cos(·,·) represents the cosine classifier, represents the cross entropy function, and R represents batch data.

[0098] In step S5 of the embodiment of the present invention, the calculated score on the final prototype is used as the basis for user identification, and the expression is:

[0099]

[0100] In the formula, and Represents the support prototype and query prototype respectively, and belongs to the final prototype part of.

[0101] The embodiment of the present invention provides an example of verifying the effect of the above gait recognition method.

[0102] In this embodiment, the proposed method is compared and analyzed with other methods on the datasets USC-HAD and CDUT-AG, including Finetune, iCarl, NCM, SPPR and CEC.

[0103] Among them, the USC-HAD activity recognition dataset contains 14 people (7 men and 7 women) and 12 types of activity behavioral datasets. Since non-periodic gait activities cannot provide enough information for identifying users. Therefore, in order to conduct incremental gait recognition experiments, we only used samples of the first 6 types of activities, including: Walking forward, Walking left, Walking right, Walking upstairs, Walking downstairs, Running forward. USC-HAD uses a wired inertial sensor acquisition device MotionNode and connects it to a micro laptop to collect data. During the data collection process, the MotionNode is worn on the subject's right front hip (the x-axis of the MotionNode points to the bottom surface and is perpendicular to the plane formed by the y-axis and z-axis). The USC-HAD experiment requires subjects to conduct 5 experiments on each activity at different indoor and outdoor locations on different days, and each subject has enough activity gait samples.

[0104] CDUT-AG contains data from 60 healthy people aged between 20 and 30 years old. In the CDUT-AG data collection experiment, we used self-developed sensor insoles for data collection. Each sensor insole integrates an inertial sensor (BMI160, including a three-axis accelerometer and a three-axis gyroscope) and 10 pressure sensors, and all sensors are connected to the control chip nrf52832 with a Bluetooth module. Finally, the collected inertial and pressure data are transmitted to the host computer via Bluetooth for data analysis. In the experiment, we asked the subjects to perform three common walking activities: horizontal walking, going upstairs, and going downstairs. Among them, horizontal walking is walking back and forth in a horizontal corridor about 8 meters long, which lasts about 2.5 minutes. Going up and downstairs is done between floors, and it takes about 15 seconds to complete each time going up or downstairs. Each activity experiment is performed 10 times. Some statistical information about the datasets USC-HAD and CDUT-AG is shown in Table 1.

[0105] Table 1: Statistics of datasets USC-HAD and CDUT-AG

[0106]

[0107] The detailed experimental results are shown in Tables 2 and 3 and Figure 5 middle.

[0108] Table 2: Comparison of results of different methods on the USC-HAD dataset

[0109]

[0110]

[0111] Table 3: Comparison of results of different methods on the CDUT-AG dataset

[0112]

[0113] All baseline comparison methods can achieve the highest classification accuracy in their respective 0th session, and the generalization ability of the model decreases in subsequent sessions due to the increase of new classes. Although studies have shown that NCM classifiers perform better than softmax classifiers in incremental learning tasks, in this work, iCarl and NCM methods failed to achieve relatively high classification accuracy in 0th session. This may be due to the particularity of our multi-activity gait dataset. Simply stacking data sample sets of multiple activity scenes makes it difficult for us to obtain a sample mean that can represent each user prototype, that is, each user's characteristics, resulting in low classification performance in the initial session. Therefore, we need some special means to construct a reasonable sample set, such as using meta-learning tasks, which is reflected in SPPR, CEC and our method.

[0114] In addition, as can be seen from Tables 2 and 3, our method achieves excellent results on both datasets. Specifically, on the USC-HAD dataset, our method achieves the best results in all sessions and has the best AA score (average accuracy, measuring the average accuracy of the model in different sessions), which is 18.17% higher than the second-ranked CEC. At the same time, we also achieve the best PD score (performance decline rate, measuring the absolute difference between the accuracy of the model in the last session and the accuracy of the first session), which is 30.13% lower than CEC. On CDUT-AG, except for the 0th session, our method achieves the best results in the remaining sessions. Our method achieves an AA score of 91.85% and a PD score of 8.21%, which are at least 1.97% and 4.19% higher than other methods, respectively. The experimental results show the great advantages of our method on multi-activity gait datasets, especially in the USC-HAD data, where there are more activity categories and our method has a more obvious improvement over other methods.

[0115] In the embodiment of the present invention, an ablation experiment is also conducted on the above-mentioned method to verify the contribution of each component of the proposed model framework to the final classification effect; among them, HPTS (hybrid prototype training scheme) and SPEM (selected prototype enhancement module) are the main components of our proposed method. HTPS generates hybrid prototypes by introducing auxiliary labels, and the adjusted prototypes are obtained by SPEM selected prototype module. Here we discuss the impact of different combinations of HTPS, SPEM-SUDA and SPEM-SADU on model performance on the datasets USC-HAD and CDUT-AG, respectively, and the results are shown in Tables 4 and 5.

[0116] Table 4: Ablation test results of the key modules of the model HTPS, SPEM-SUDA and SPEM-SADU on the dataset USC-HAD

[0117]

[0118] Table 5: Ablation test results of the key modules of the model HTPS, SPEM-SUDA and SPEM-SADU on the CDUT-AG dataset

[0119]

[0120] For the baseline model without HPTS and SPEM, only user labels are used in the entire training process, and the traditional user category prototypes are generated using a frozen feature extractor and extended through incremental sessions. The prototypes are not adjusted during the entire process. When HPTS is used, the AA scores of the models on the USC-HAD and CDUT-AG datasets are improved by 24.84% and 5.02%, respectively, and the PD scores are reduced by 36.1% and 9.91%, respectively. Using HPTS achieves better performance on both datasets. Since the user prototypes generated by the traditional prototype network only consider the user ID and are not further divided on the activities, most of the key information is discarded. In contrast, HTPS uses a more fine-grained prototype to retain these key information, thus achieving satisfactory results. At the same time, since the performance difference between the two datasets (USC-HAD contains 6 categories of activities and CDUT-AG contains 3 categories of activities) shows that the advantages of HPTS are more obvious when there are more activity scenarios, which requires further experimental proof.

[0121] In addition, this embodiment verifies the improvement of SPEM on HPTS. Compared with HTPS alone, after using SPEM, the AA scores of the model on the USC-HAD and CDUT-AG datasets increased by about 1.51% and 2.24%, respectively, and the PD scores decreased by about 1.66% and 2.51%, respectively, verifying that the prototype adjusted by SPEM has improved its generalization representation ability to a certain extent. In addition, since SPEM uses an adjustment mechanism for selecting prototypes, SPEM-SUDA and SPEM-SADU can complement each other's information to a certain extent, thereby achieving better results after feature fusion, which can be concluded by comparing cases 3, 4, and 5 in Tables 4 and 5. Ablation experiments show that HTPS and SPEM have a positive impact on the performance improvement of the model.

[0122] In order to demonstrate the advantages of our proposed method in the field of multi-activity gait recognition, we use t-SNE to compress the optimized embedding prototype into a 2D space for visualization. Figure 6 As shown (in Figure 6 In the figure, dots of different colors represent data points of different categories (user categories), and the black "x" (i, j) represents the prototype of user i under activity j). We visualize the data of the first session and the last session on the USC-HAD dataset. Figure 6 (a) It can be seen that there are 6 user categories in the first session. Figure 6 As shown in (b), there are 14 user categories in the last session, and each user category contains 6 activity categories. In the figure, we use (i, j) to represent the prototype of user i under activity j. In the gait data containing multiple activities, the same user forms multiple clusters in the metric space, and each cluster usually represents a type of activity of a certain user. The same user is scattered into multiple clusters in the metric space, so the use of traditional prototype networks to cope with user classification tasks is full of uncertainty. In our work, we propose to use HTPS, which regards each cluster as a separate prototype, that is, the original prototype is refined into multiple prototypes representing the cluster center. In the incremental task, a small adjustment of the position of a cluster prototype in the metric space is much less costly than adjusting the original prototype.

[0123] In order to evaluate the superiority of our method on multi-activity datasets, we increase the number of activities from 1 to the upper limit of the number of activities in the dataset (the maximum number of activities in the dataset USC-HAD is 6, and the maximum number of activities in the dataset CDUT-AG is 3) and conduct experimental verification in turn. Given that the CEC method is the most competitive except for our method, we compare the performance of the two methods when the number of activities increases. Figure 7 and Figure 8The classification results of our method and the comparison method CEC after limiting different numbers of activities in the USC-HAD dataset and CDUT-AG dataset are shown respectively. It can be found from the figure that as the number of activities increases, the AA and PD scores of the comparison method are constantly decreasing, while our method always remains relatively stable, which is consistent with our expected results.

[0124] At the same time, we compare the accuracy difference between our method and the comparative method CEC in different experiments. Tables 6 and 7 show the improvement of our method relative to CEC. On the one hand, our method basically maintains a positive improvement under different numbers of activities, which shows that our method can be applied to multi-activity scenarios and has a certain generalization ability for different activity scenarios. On the other hand, as the number of activities continues to increase, the improvement of our method relative to CEC is also increasing.

[0125] Table 6: The incremental results of our method compared to CEC in the classification accuracy of different number of activities in the USC-HAD dataset

[0126]

[0127] Table 7: The incremental results of our method compared to CEC classification accuracy for different number of activities in the CDUT-AG dataset

[0128]

[0129] As can be seen from Table 6, in the USC-HAD dataset, as the number of activities increases from 1 to 6, the ΔAA score (the difference between our method and CEC's AA) increases from +2.79% to +18.17%, and the ΔPD score (the difference between our method and CEC's PD) decreases from -3.67% to -30.12%. As can be seen from Table 7, in the CDUT-AG dataset, as the number of activities increases from 1 to 3, ΔAA increases from -0.49% to +1.97%, and ΔPD decreases from +0.28% to -4.19%. This shows that our method performs better in the face of data with more activities, that is, the more activities there are, the more obvious the advantage of our method.

[0130] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0131] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A small sample incremental gait recognition method in multiple activity scenarios, characterized in that: The following steps are involved: S1, collect gait samples in multiple activity scenarios and build a data set; The labels of gait samples in the dataset include user labels and activity labels; S2, constructing a model framework based on hybrid prototype enhancement, including a feature extractor and a selected prototype enhancement module connected in sequence; S3. Pre-train the feature extractor of the model framework using the data set to obtain a pre-trained model; S4. Based on the pre-trained model, incremental training is performed on the model framework, including: Use the pre-trained model to extract features of each support sample in the dataset and form an initial prototype for each class based on the category; The initial prototype is a hybrid prototype including user tags and activity tags; Inputting the initial prototype into the selected prototype enhancement module for incremental training to obtain an enhanced prototype, and then obtaining a gait recognition model; S5. For the query sample of the gait to be recognized input into the gait recognition model, the corresponding initial prototype is obtained through the feature extractor with frozen parameters, which is merged with all the prototypes in the previous session and the prototype is enhanced. Then, the user category of the query sample is identified by calculating the distance between the query sample and the enhanced prototype to complete the gait recognition.

2. The small sample incremental gait recognition method in multiple activity scenarios according to claim 1 is characterized in that: In step S4, the method of forming the initial prototype of each class according to the category is specifically as follows: S401, combine m different categories of activity tags and n different categories of user tags in the data set to form a combination class Y={y i }and i∈[0,m*n); S402: Obtain the initial prototype of each class by calculating the embedding mean of each support sample in all support sample sets in the data set.

3. The small sample incremental gait recognition method in multiple activity scenarios according to claim 1, characterized in that: In step S4, the initial prototype is represented as: Where P b represents the initial prototype of the combination class b, S b represents the set of support samples belonging to category b, |S b | represents the number of support samples, f b (·) is the pre-trained feature extractor, x i Indicates the support sample, y i Represents the category label.

4. The small sample incremental gait recognition method in multiple activity scenarios according to claim 1, characterized in that: In step S4, in each incremental session, the method for performing incremental training based on the hybrid prototype is specifically as follows: S411, generating a mixed prototype of a new category support sample through a feature extractor, merging it into a prototype container of the previous session, and forming all mixed prototypes in the current prototype container into a support prototype of the current incremental session; S412, extracting a hybrid prototype of the query sample through a feature extractor Input to the selected prototype enhancement module for evaluation; S413, in the prototype enhancement module, the mixed prototype is calibrated by a 1×1 convolution kernel with a learning parameter q Processing to obtain a new feature map Mixed prototypes with the same user or activity label are transformed into Merge to get a new feature map and S414, according to the characteristic graph and The features in , construct the maximum attention weight matrix A; S415, the maximum attention weight matrix A and the feature map Multiply and add mixed prototypes As the residual term, the enhanced prototype is obtained.

5. The small sample incremental gait recognition method in multiple activity scenarios according to claim 4 is characterized in that: In step S413, the feature map and It is expressed as: In the formula, s q (·),s k (·),s v (·) represents a linear transformation function with q, k, and v as parameters, Represents a hybrid prototype The new prototype obtained by averaging the prototypes labeled with y y represents a user tag or activity tag, and the final prototype Represents one of the user prototype or activity prototype, and T represents the transposition operation.

6. The small sample increment gait recognition method in multiple activity scenarios according to claim 4, characterized in that: In step S414, the weight element α of the maximum attention weight matrix A ij It is expressed as: In the formula, and Respectively and The features at the i-th and j-th positions in Represents the original attention matrix The index of the maximum value of the attention score on the first dimension, is the scaling factor and T represents the transpose operation.

7. The small sample increment gait recognition method in multiple activity scenarios according to claim 4, characterized in that: The selection prototype enhancement module includes an activity enhancement module SPEM-SUDA and a user enhancement module SPEM-SADU; The enhanced prototype It is a prototype obtained by adjusting the activity enhancement module SPEM-SUDA and the user module SPEM-SADU and performing feature fusion, which is expressed as: In the formula, and They represent the prototype features after being adjusted by the activity enhancement module SPEM-SUDA and the user enhancement module SPEM-SADU respectively.

8. The small sample increment gait recognition method in multiple activity scenarios according to claim 7, characterized in that: In step S4, the loss function of the selected prototype enhancement module express: In the formula, and They represent the loss functions of the activity enhancement module SPEM-SUDA and the user enhancement module SPEM-SADU, and Respectively Decomposed support prototype and query prototype, cos(·,·) represents the cosine classifier, represents the cross entropy function, and R represents batch data.

Images