A category incremental behavior recognition model construction method and a behavior recognition method

Abstract

The application discloses a class incremental behavior recognition model construction method based on knowledge importance and a behavior recognition method. In view of the problems of catastrophic forgetting in the incremental learning process and class imbalance in the long-tail data set, the application first proposes a judgment method for obtaining a mixed weight by using a weight subset so that a teacher model can autonomously identify important knowledge, retains important knowledge in old data, and realizes that a student network can automatically update and adapt according to the judgment effect of the teacher network in the incremental learning process, thereby relieving the problem of catastrophic forgetting to a certain extent. Meanwhile, by limiting the range of the weight subset, the problem of class imbalance in the long-tail data set is also solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, further relates to the fields of machine learning and behavior recognition, and in particular to a class-incremental behavior recognition model construction and a behavior recognition method based on knowledge importance. BACKGROUND

[0002] Traditional wearable behavior recognition mainly adopts a fixed recognition model, that is, a classification model is trained offline using existing labeled data, and the classification model is used to classify and recognize individual behaviors. Such a model is fixed in many aspects, such as the number of categories defined. However, in real environments, the scenarios people face are dynamic, and as time goes on, the categories encountered are constantly increasing, such as an individual learning to run for a period of time and then learning to swim after a period of time. How to save the memory of learned behaviors in the process of constantly learning and recognizing new behaviors in an open environment is an important challenge in the field of behavior recognition.

[0003] Incremental learning is an effective method to adapt to dynamic changes in an open environment, and its model construction method is similar to the learning and adaptation ability of humans to new things. Incremental learning refers to the process of learning a model and then, when new, previously unseen training data appears, the model only needs to moderately update the features of the new data, without retraining the entire model, and the effective information learned previously is not forgotten. Ideal incremental learning should meet the following requirements: no need to store old data; sufficient stability for old knowledge; and sufficient plasticity for new knowledge. However, in the current research process, catastrophic forgetting is still a major problem in incremental learning that has not been completely solved.

[0004] Existing technologies for class-incremental behavior recognition do not fully utilize the complementary relationship between new and old knowledge in the incremental process. For example, Chinese patent CN201810354381.1 proposes a method for realizing class-incremental behavior recognition based on random forests, but this method does not well solve the problem of catastrophic forgetting in incremental learning; Chinese patent CN201911325203.7 designs a self-adaptive learning method applied to robots, which can realize behavior recognition through the robot's self-adaptive process; Chinese patent CN201810102422.8 realizes behavior recognition using multi-source data fusion and indoor positioning; Chinese patent CN112990280-A adopts a class-incremental method for image classification problems. American patent US2020302230-A1 applies class-incremental learning to the field of object detection. However, these methods do not solve the problem of catastrophic forgetting in incremental learning.

[0005] Traditional methods based on knowledge distillation have a bth In the incremental step, the model is initialized with the parameters learned in the previous step, and a new output node is added, where the weights of the fully connected layer are randomly initialized. Then, the model learns the new class with the input data being new data D b and a part of old data , where the new data D b comes from the new class C b , and the old data comes from the old class .

[0006] The traditional knowledge distillation-based method uses cross-entropy loss (Cross-Entropy Loss) L CE and knowledge distillation loss (Knowledge Distillation Loss) L KD .

[0007] However, even if the knowledge distillation and cross-entropy loss-based techniques are used, the class-incremental behavior recognition technology still has challenges such as catastrophic forgetting in the incremental learning process in practical applications. The method of using distillation loss and cross-entropy loss as loss functions together has the following shortcomings: 1. The processing ability of the teacher model for old knowledge is not fully judged. 2. Due to the class imbalance problem between new and old classes in the incremental process, the model may learn new data particularly well, while increasing the forgetting degree of old knowledge. 3. Due to the differences between new and old classes in the incremental process, there is also a certain connection between them, and how to more effectively learn new knowledge is also a problem that needs to be solved.

[0008] Therefore, it is of great significance and application value to study how to balance new and old classes, reduce the forgetting degree of old knowledge by the model, and learn new knowledge from the connection between new and old classes while learning new knowledge in the incremental process. A model with higher robustness is needed, which can learn new knowledge while alleviating catastrophic forgetting and has good learning effect on new knowledge. SUMMARY

[0009] To solve the above problems, the application provides a class incremental behavior recognition method and system based on knowledge importance, which adopts a judgment method for enabling a teacher model to autonomously identify important knowledge, so as to retain important knowledge in old data and dynamically adjust the learning degree of new and old knowledge, solve the class imbalance problem in long-tail data set, and finally enable the model to obtain important knowledge in old knowledge in the process of incremental learning, so as to more effectively alleviate the catastrophic forgetting problem than the current most advanced method.

[0010] In one aspect, the application provides a class incremental behavior recognition model construction method based on knowledge importance, characterized in that the method comprises:

[0011] Step (1), obtaining a first model suitable for incremental learning and trained;

[0012] Step (2), for each sample in original data {D 1 ,…,D b-1}, calculating the prediction value of the first model for each sample, verifying the prediction value, obtaining the judgment accuracy of the first model for each class of sample, and forming a knowledge importance factor β of the first model for original data {D 1 ,…,D b-1} based on the judgment accuracy of the first model for each class of sample.

[0013] Step (3), constructing a loss function based on the knowledge importance factor β, substituting the loss function into the first model, and performing class increment on the first model according to the target incremental class number, as a second model.

[0014] In a preferred implementation, the first model calculates the prediction value of each sample based on the following formula:

[0015]

[0016] wherein, T is a hyperparameter, representing a temperature value of distillation; is an element in , and represents the probability output of the first model when predicting the sample x belonging to each class.

[0017] In another preferred implementation, the knowledge importance factor β is: wherein, represents the prediction value of each sample of the teacher model, representing in b th The number of categories of old data in the incremental step.

[0018] In another preferred implementation, the method further comprises cosine normalization on the extracted feature vectors based on the samples.

[0019] In another preferred implementation, the loss function is:

[0020] L(x,y)=(1-β)L CE (x,y)+βL KD (x), (3)

[0021] Wherein, L CE represents the cross-entropy loss, L KD represents the knowledge distillation loss.

[0022] In another preferred implementation, the method further comprises, for any sample, if the predicted value of the first model for it is less than 0, setting its predicted value to 0.

[0023] In another preferred implementation, after each category increment, steps (2)-(3) are repeated, and the second model in the last cycle is taken as the first model.

[0024] According to another aspect of the present application, a method of behavior recognition using the model constructed by the method is provided, comprising:

[0025] Substituting new sample data into the second model for behavior recognition, and updating the new sample data to the original data, and repeating steps (2)-(3).

[0026] According to another aspect of the present application, a computer readable storage medium having a computer program stored thereon is provided, wherein the program, when executed by a processor, implements the method.

[0027] According to another aspect of the present application, a computer device is provided, comprising a memory and a processor, and a computer program capable of running on the processor is stored on the memory, characterized in that the processor implements the method when executing the program.

[0028] The technical effects achieved by the present application are:

[0029] The application aims at the problems of catastrophic forgetting and class imbalance in long-tail data set in the incremental learning process, first proposes a judgment method of using weight subsets to obtain mixed weights so that the teacher model can identify important knowledge autonomously, retains important knowledge in old data, realizes that the student network can automatically update and adapt according to the judgment effect of the teacher network in the incremental learning process, does not completely follow the old knowledge obtained from the teacher network, solves the problem of continuous errors caused by the error of the teacher network, and to some extent, alleviates the problem of catastrophic forgetting; at the same time, by limiting the range of the weight subset, the problem of class imbalance in the long-tail data set is also solved. The class incremental behavior recognition method based on knowledge importance has obvious improvement in accuracy compared with the existing most advanced method in the incremental process.

[0030] Compared with the prior art, when the incremental behavior recognition is carried out by using the method of the application, the forgetting effect of the model in the incremental learning process is smaller, and the recognition accuracy is obviously improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a model structure diagram of the application;

[0032] Figure 2 is a precision comparison diagram of the method in the embodiment of the application and other class incremental learning algorithms on the sensor-based behavior recognition data set DSADS.

[0033] Figure 3 is a precision comparison diagram of the method in the embodiment of the application and other class incremental learning algorithms on the sensor-based behavior recognition data set OPPORTUNITY.

[0034] Figure 4 is a precision comparison diagram of the method in the embodiment of the application and other class incremental learning algorithms on the sensor-based behavior recognition data set HARUSDS.

[0035] Figure 5 is a precision comparison diagram of the method in the embodiment of the application and other class incremental learning algorithms on the sensor-based behavior recognition data set PAMAP2. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the application more clear and understandable, the application will be further described in detail below with reference to the drawings. It should be understood that the specific examples described herein are only used to explain the application, and are not used to limit the application.

[0037] In this embodiment, the model construction and recognition test of behavior recognition are carried out by taking the behavior recognition dataset based on accelerometer, magnetometer and gyroscope data as an example. The model for category incremental learning established in this embodiment is as shown in Figure 1

[0038] In general, the teacher model trained by the old training data is first obtained in this embodiment, the classification prediction value of each sample in the original data (both test data and training data) is calculated by using the teacher model, the prediction value is verified (the label is taken as the basis for the data containing the label, and the determined classification is taken as the basis for the data not containing the label), the judgment accuracy of each class sample of the teacher model is obtained, and the knowledge importance factor β of the teacher model for the original data set {D 1 ,…,D b-1} is formed based on the judgment accuracy of each class sample of the teacher model. Then, the loss function is constructed based on the knowledge importance factor β, and the loss function is substituted into the teacher model as a student model. The new training data is input into the student model, and the above steps are repeated to dynamically adjust the knowledge importance factor β and the corresponding loss function, and the original data set is updated at the same time.

[0039] It should be noted that the teacher model in the present application is a model suitable for incremental learning, and the output node or the output node of the full connection layer thereof can be adjusted according to the total amount of categories of the training data. For example, if the category of the training data is 3 categories, the output node of the full connection layer is 3, and in the incremental process, if the category of the training data increases by 2 categories, the output node of the full connection layer increases by 2. The construction of such a model belongs to the prior art in the field, and will not be described in detail here. In the following examples, a CNN model with incremental learning capability is used as a teacher model.

[0040] Next, the data set description, feature extraction, Cosine regularization and knowledge distillation module method based on knowledge importance are introduced in detail respectively.

[0041] Data set description

[0042] In this embodiment, the behavior recognition test of the present application method and the prior art method is carried out on four data sets respectively. The four data sets are

[0043] (1) DSADS data set (see Altun, K., Barshan, B., & Ergin, S. (2004). Activity recognition using inertial sensors in a wearable wireless device. In ISWC (pp. 238-247). IEEE. ​O. (2010). Comparative study on classifying human activities with miniature inertial and magnetic sensors. Pattern Recognition, 43(10), 3605 - 3620.);

[0044] (2) OPPORTUNITY dataset (see Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., K., G.,... & Millan, J.D.R. (2010, June). Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh international conference on networked sensing systems (INSS) (pp. 233 - 240). IEEE.);

[0045] (3) HARUSDS dataset (Anguita, D., Ghio, A., Oneto, L., Parra, X., & Reyes - Ortiz, J.L. (2012, December). Human activity recognition on smartphones using a multiclass hardware - friendly support vector machine. In International workshop on ambient assisted living (pp. 216 - 223). Springer, Berlin, Heidelberg.);

[0046] (4) PAMAP2 dataset (Reiss, A., & Stricker, D. (2012, June). Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers (pp. 108 - 109). IEEE.) was conducted.

[0047] The DSADS dataset consists of 19 daily activities and sports of 8 users (4 males and 4 females). Three types of sensors (triaxial accelerometer, triaxial gyroscope, and triaxial magnetometer) are placed on five different body parts including torso, right arm, left arm, right leg, and left leg. Therefore, each piece of sensor data contains 45-dimensional readings. All users perform the activities in their own habitual way without any constraints.

[0048] The OPPORTUNITY dataset consists of daily activities of 4 users. Each user performs 6 rounds of experiments (including 5 rounds of daily activities and 1 round of training). Three types of sensors are deployed in a smart home environment to collect the users' activity data. Since the focus of the experiments in this example is on activity recognition, this example uses data from wearable sensors. In this dataset, there are a large number of missing values for the Bluetooth sensors. Therefore, only data from 5 Xsens inertial test units mounted on the sports jacket and 2 commercial inertial sensors mounted on the feet are used for the experiments. Each inertial test unit contains an accelerometer, a gyroscope, and a magnetometer. In the OPPORTUNITY dataset, activities are classified into 4 levels: motion patterns, low-level activities, mid-level activities, and high-level activities. The experiments in this example use the left-hand channel data in the mid-level activities to verify the effectiveness of the method in this chapter.

[0049] The HARUS DS dataset consists of activity data of 30 users. A smartphone fixed on the waist of the user is used to collect triaxial accelerometer and triaxial gyroscope readings for 6 activities (walking, going upstairs, going downstairs, sitting, standing, and lying). The data in this dataset has been preprocessed by the authors and divided into training and test sets. The experiments in this example directly use the data processed by the authors. Since the division of the dataset by the authors does not conform to the experimental goals in this chapter, this section merges the training and test sets and re-divides the dataset.

[0050] The PAMAP2 dataset consists of 12 different motion activities collected from 9 users. Each user wears 3 inertial sensor units and a heart rate detector. Since there are many missing values in the readings of the heart rate detector, this method only uses the data collected by the inertial sensor units for the experiments. Each inertial sensor unit consists of two accelerometers, a magnetometer, and a gyroscope. Therefore, the experiments use data collected by 12 sensors.

[0051] Feature extraction: Since the HARUSDS dataset provides features, only the DSADS, OPPORTUNITY, and PAMAP2 datasets are used for feature extraction in this embodiment. In the embodiments of the present application, the three-axis data from the same sensor are first synthesized into one-axis data using the formula . The purpose of this operation is to eliminate the negative effects caused by different sensor orientations. Then, a sliding window mechanism is used to segment the sensor readings.

[0052] In this embodiment, the inventors set the window size of the DSADS, OPPORTUNITY, and PAMAP2 datasets to 5 seconds, 2 seconds, and 5.12 seconds, respectively, with a 50% overlap between consecutive windows. For each window, the inventors extract 27 features from the time and frequency domains of a single sensor.

[0053] The extracted features are subjected to a regularization operation, which is a routine means in the art and will not be described here.

[0054] Next, the knowledge distillation module based on knowledge importance is used to distill the data after the regularization operation.

[0055] In this embodiment, an incremental step (adding new data, especially new class data) with n old classes and m new classes is used as an example. The knowledge distillation method is used for an old trained model (teacher model) that can classify n old classes to obtain a new student model that can classify n+m classes through distillation loss and cross-entropy loss.

[0056] Suppose there are B batches of training data {D 1 ,…,D B}, and when there are b new class data, in the b th th incremental step, there are where and represent the input data and the corresponding label, respectively, and n b is the number of examples in the set D b . In the b th th class incremental step, the goal is to learn knowledge from new data D b while preserving the knowledge learned from the old data {D 1 ,…,D b-1}. In each incremental step, the trained model will be validated on all currently visible classes.

[0057] The existing state-of-the-art method updates the parameters of the feature extraction layer and the fully connected layer using the total loss defined in equation (1) in the background technology.

[0058] The inventors have found that the student model training process based on the teacher model is similar to the learning process of a human student. Even in the process of a human student learning from a teacher, there are cases where the teacher teaches wrong knowledge. When the student who has learned the wrong knowledge teaches other students, the whole knowledge chain is wrong. Therefore, in the training process of the student model, how to let the student model judge which knowledge taught by the teacher model is correct in the learning process, and then learn the correct knowledge from the teacher model and forget the wrong knowledge, is very helpful to alleviate the most critical catastrophic forgetting problem in the incremental learning process.

[0059] Therefore, the embodiment adopts the method of category incremental learning based on knowledge importance to help the student model judge which knowledge taught by the teacher model is correct and more important in the learning process, and to adaptively learn the correct knowledge and forget the wrong knowledge.

[0060] Specifically, in the process of learning new data samples b th In the processing step, for each sample in the old data {D 1 ,…,D b-1}, the predicted value of the teacher model for each sample is calculated as:

[0061]

[0062] wherein T is a hyperparameter, representing the temperature value of distillation; is an element in , each element represents the classification output obtained by using the teacher model to classify any sample x (the value obtained after the classification output is passed through an activation function is between 0-1, representing the probability value of whether it belongs to a certain category). After category increment, it is represents the classification output of the current model, C b represents the number of categories of new data in the b th increment step.

[0063] Based on the predicted value of each sample by the teacher model, the classification accuracy of the teacher model is calculated class by class, so as to obtain the accuracy of the judgment of the teacher model on each sample and each class, and take it as the weight value of the degree of belief of the student model in the teacher model. The weight values of the same class samples form a weight subset, the values in the weight subset are averaged to obtain the judgment accuracy of the teacher model on this class, and the values after averaging the weight subsets of different classes form a mixed weight subset, the weights in the mixed weight subset are averaged to obtain the judgment accuracy of the teacher model on several classes (the first average: the weight values of all samples in the same class are calculated to obtain the judgment accuracy of the teacher model on this class. The second average: if there are multiple classes in the old data, the judgment accuracy of the teacher model on the class is calculated respectively, and then the judgment accuracy of the teacher model on several classes is obtained by averaging the judgment accuracy of the teacher model on the class; if there is only one class in the old data, the average is not needed, and the judgment accuracy of the teacher model on the class is the judgment accuracy of the teacher model on the class), as the knowledge importance factor:

[0064]

[0065] The teacher model is incrementally set, the number of output nodes of the full connection layer is adjusted (to m+n), and the knowledge importance factor is used to judge the classification ability of the teacher model on the old data in the current incremental step, so that the loss of new and old knowledge of the model can be balanced, and the final loss determined by the application is optimized as:

[0066] L(x,y)=(1-β)L CE (x,y)+βL KD (x), (3)

[0067] Where the cross-entropy loss (Cross-Entropy Loss) L CE is:

[0068]

[0069] Where δ c＝y is an indicator function, p c (x) is the output probability corresponding to the c th class.

[0070] The knowledge distillation loss (Knowledge Distillation Loss) L KD is:

[0071]

[0072] Where

[0073] In the existing method of using knowledge distillation to realize incremental learning, as described in the background art, a single weight λ is used to balance the cross-entropy loss and the knowledge distillation loss for all classes in the incremental process. In the present application, this method is improved. As shown in formulas (2) and (3), the value of β is dynamically adjusted based on the current data set. For the classes that the teacher model can accurately predict, the value of β is relatively large, and the weight of the knowledge distillation loss compared with the cross-entropy loss is also large, which means that the student model has a higher degree of belief in the teacher model. When the predicted value of the teacher model for any sample is less than 0.5, it means that the teacher model is generally wrong in predicting the old data, and therefore β = 0 is set to completely ignore the prediction result of the teacher model for the old data.

[0074] In this embodiment, the method of the present application (our) is compared with the most advanced class incremental learning method, and the comparative methods include:

[0075] (1) LwF method (see Li, Z., & Hoiem, D. (2017). Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12), 2935-2947.);

[0076] (2) BiC (see Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., & Fu, Y. (2019). Large scale incremental learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 374-382),

[0077] (3) WA (see Zhao, B., Xiao, X., Gan, G., Zhang, B., & Xia, S. T. (2020). Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 13208-13217).);

[0078] (4) CIRF (see Hu, C., Chen, Y., Hu, L., & Peng, X. (2018). A novel random forests based class incremental learning method for activity recognition. Pattern Recognition, 78, 277-290.).

[0079] Comparison Figures 2-5 From the tests on the four public activity recognition datasets, it can be seen that the method of the present application has a smaller decline in accuracy than the other four comparison methods during the class incremental process, which shows that the method of the present application has a better effect of mitigating the catastrophic forgetting problem compared with the most advanced class incremental learning method.

[0080] In a preferred implementation, the present application performs an improved regularization operation on the dataset. Those skilled in the art can use conventional regularization operations in the art, or can use the following improved regularization operation.

[0081] Cosine regularization is an additional (optional) process in the method of the present application. The inventors found that there is a problem of imbalance between new and old class data in the process of class incremental learning, which is not only caused by the long-tail dataset problem, but also caused by the difference between new and old class data. Due to the imbalance between new and old classes, the embedding and bias of new classes are much larger than those of old classes. This leads to a bias in the prediction of new classes.

[0082] To solve this problem, the present application proposes to use cosine normalization, which is based on cosine similarity, not simple dot product. For the feature data processed above, take a simple CNN network as an example, first pass through the convolutional layer, then the pooling layer, and finally pass through the fully connected layer to the classifier. is the l2 regularization vector of the classifier parameter W (the sum of the absolute values of each element in the vector), is the l2 regularization vector of the feature v obtained by feature extraction. The l2 regularization vector and By measure the cosine similarity between the two regularization vectors as the cosine normalization vector of the feature, which is used in the subsequent model. It is worth noting that due to cosine normalization, the value before the loss function softmax is in the same range (i.e. [-1, 1]), so it is comparable.

[0083] Although the principles of the present application have been described in connection with the preferred embodiments thereof with reference to the drawings, it should be understood that the application is not limited to the construction and arrangements of the preferred embodiments as set forth above and above and that several other changes, modifications, equivalents, substitutions and alterations can be made thereto without departing from the spirit and scope of the present application.

Claims

1. A method for constructing a class-incremental behavior recognition model based on knowledge importance, characterized in that, The method comprises: Step (1), obtaining a first model suitable for incremental learning and trained; Wherein the first model calculates its prediction value for each sample based on the following formula: （1） wherein, , is a hyperparameter, expressed as a temperature value of distillation; is an element of, denotes the probability output of the first model that a sample x, each sample comprising data of a user behavior or motion acquired by a sensor, belongs to each class when the first model makes a prediction for this sample; Step (2), for each example in the original data , calculate the predicted value of each example by the first model, check the predicted value, obtain the judgment accuracy of the first model for each type of example, and form the knowledge importance factor β of the first model for the original data based on the judgment accuracy of the first model for each type of example. wherein the knowledge importance factor β is: wherein, denotes the predicted value of the first model for each example, represents the number of classes of the old data at the number of classes of the old data in the incremental step; Step (3), constructing a loss function based on the knowledge importance factor β, substituting the loss function into the first model, and performing class increment on the first model according to the target number of incremental classes as a second model.

2. The method of claim 1, wherein, The method further comprises performing cosine normalization on the feature vector extracted based on the sample.

3. The method of claim 1, wherein, The loss function is: （3） wherein, denotes the cross-entropy loss, denotes the knowledge distillation loss.

4. The method of claim 1, wherein, The method further comprises, for any sample, if the prediction value of the first model for the sample is less than 0, setting the prediction value of the sample to 0.

5. The method of claim 1, wherein, After each class increment, steps (2)-(3) are repeated, and the second model in the last cycle is taken as the first model.

6. A method of behavior recognition using a model constructed by the method of any one of claims 1-5, comprising: Substituting the sample data to be measured into the second model for behavior recognition, updating new sample data to the original data, and repeating steps (2)-(3).

7. A computer readable storage medium having stored thereon a computer program, wherein, The program is executed by a processor to implement the method of any one of claims 1-6.

8. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1-6. The processor executes the program to implement the method of any one of claims 1-6.

Images