Augmentable behavior recognition system and method based on data enhancement and knowledge distillation

Abstract

The present invention discloses an augmented behavior recognition system and method based on data augmentation and knowledge distillation, comprising: a data acquisition unit for collecting sensor data during user movement; a data preprocessing unit for outputting preprocessed behavior data; a storage unit for storing representative sample sets of old behavior categories and old behavior recognition models; a behavior recognition unit for identifying and classifying the preprocessed behavior data; an augmented training unit for outputting a new behavior recognition model; the new behavior recognition model recognizes old behavior category data or new behavior category data and outputs a behavior recognition result. The present invention ensures continuous learning performance and resource cost advantages during the continuous learning process of behavior recognition; at the same time, it conforms to the dynamic extensibility of user behavior in real life and can maintain sustainable augmented recognition performance for unknown new categories of behavior.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent perception technology and artificial intelligence technology, and specifically refers to an augmentable behavior recognition system and method based on data enhancement and knowledge distillation. Background Art

[0002] In real-life scenarios, people make a variety of movements, each corresponding to specific behaviors. Identifying these behaviors helps people record their daily lives, analyze behavioral habits, and build character profiles. However, collecting activity data requires specialized sensors, which used to be cumbersome and difficult. With the rapid development and widespread adoption of smart mobile devices, people have become accustomed to carrying them with them. Using the inertial measurement units (IMUs) of these devices, we can collect user behavior data, perform behavior recognition, and serve downstream tasks.

[0003] Behavior recognition becomes increasingly complex as activity scenarios change. For example, a person is born with only the ability to crawl, roll over, and sleep. As they age, they gradually learn to walk, eat, run, and eventually read, write, and play basketball. As activity scenarios become increasingly diverse, the number of behaviors that need to be recognized increases. Traditional neural network models can only recognize pre-trained behavior categories, but the number of behavior scenarios and types is dynamically increasing. This dynamic growth makes previous activity samples difficult to obtain due to privacy concerns. Therefore, how to ensure that the model, based on current activity samples, does not forget previously recognized behaviors and can also recognize new behaviors is a challenge. There is an urgent need for behavior recognition models to possess continuous recognition and classification capabilities to adapt to changing life scenarios.

[0004] Traditional neural network-based models used in incremental behavior recognition tasks often suffer from the following technical deficiencies:

[0005] 1. The trained (old) behavior recognition model can only recognize known (old) behavior categories, but cannot recognize unknown (new) behavior categories;

[0006] 2. Old behavior data is difficult to obtain due to privacy issues. Using only new behavior data to train and update the old behavior recognition model will cause the model to catastrophically forget the recognition tasks in the old domain.

[0007] 3. As the types of behaviors that need to be identified continue to expand, the number of parameters in the neural network model increases accordingly. Summary of the Invention

[0008] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an augmentable behavior recognition system and method based on data enhancement and knowledge distillation to solve the problem that the behavior categories are constantly increasing in the existing dynamic environment and the model cannot accurately identify new behavior categories.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] The present invention provides an augmentable behavior recognition system based on data enhancement and knowledge distillation, comprising: a data acquisition unit, a data preprocessing unit, a storage unit, a behavior recognition unit and an augmentation training unit;

[0011] A data acquisition unit, used to collect data from the three-axis accelerometer and three-axis gyroscope of the inertial sensor when the user is exercising;

[0012] A data preprocessing unit resamples the acquired accelerometer data and gyroscope data, clips the data length, normalizes the data value, and outputs the preprocessed behavioral data;

[0013] A storage unit, used to store old (i.e., known) behavior category representative sample sets and old (i.e., trained) behavior recognition models;

[0014] A behavior recognition unit is used to recognize and classify the preprocessed behavior data. For new behavior categories, the number of output categories of the classifier is changed, and samples of the new (i.e., unknown) behavior category and representative samples of the old behavior category are used as training samples to generate an augmented class training task, which is sent to the augmented class training unit.

[0015] The augmentation training unit obtains the old behavior recognition model stored in the storage unit, performs augmentation training on the old behavior recognition model using samples of the new behavior category and some representative samples of the old behavior category, and outputs a new behavior recognition model; the new behavior recognition model recognizes the old behavior category data or the new behavior category data and outputs a behavior recognition result.

[0016] Furthermore, the data preprocessing unit includes: a data resampling module, a data normalization and standardization module;

[0017] Data resampling module: aligns the behavioral data acquired by the data acquisition unit; performs linear interpolation on each behavioral data with inconsistent lengths. This operation uses an interpolation function and sets a fixed sampling length. The interpolation function will generate behavioral data with a fixed sampling length to achieve alignment of all behavioral data;

[0018] Data normalization and standardization module: Calculate the mean μ and variance σ of all resampled data, apply μ and σ to all training data, and use the following formula to normalize the data to [-1, 1] so that it obeys the standard normal distribution:

[0019]

[0020] Where D o is the normalized behavioral data, D i is the original behavioral data, μ represents the mean of the training data, and σ represents the variance of the training data; all test data are normalized and standardized using the mean and variance.

[0021] Furthermore, the storage unit stores the old behavior category representative sample set D old and the old behavior recognition model M o , where the old behavior category represents the sample set D old The capacity is set to a fixed number of samples: after each incremental training, the central sample of the new behavior category training sample is calculated by clustering method, and the representative training samples are selected and stored in the storage unit. At the same time, the number of representative samples of the old behavior category is reduced, and the training samples with large differences from the central sample are deleted. The total number of representative samples of the old behavior category and the new behavior category is kept constant, and the old behavior category representative sample set D is updated with the current old behavior category representative sample and the new behavior category representative sample. old .

[0022] Furthermore, the behavior recognition unit obtains the output behavior data of the data preprocessing unit; in the model training stage, if there are new categories in the behavior data, the number of output categories of the classifier is changed, and samples of the new behavior category and some representative samples of the old behavior category are used as training samples to generate a category-increasing training task, which is sent to the category-increasing training unit; otherwise, the behavior recognition result is output; in the testing stage, a classification result is output based on the behavior data.

[0023] Furthermore, the behavior recognition unit includes: a feature extraction module and a classifier module;

[0024] The feature extraction module encodes the behavior data processed by the data preprocessing unit and outputs a 256-dimensional feature vector;

[0025] The classifier module performs inner product calculation on the encoded 256-dimensional feature vector and the hidden layer feature matrix of the classifier, and outputs an N-dimensional vector, where N represents the number of behavior categories, and the category corresponding to the largest value on the N-dimensional vector is the classification result.

[0026] Furthermore, the feature extraction module encodes the behavioral data processed by the data preprocessing unit and outputs a 256-dimensional feature vector, which contains important feature information of the behavioral data and serves as the input of the classifier module; the feature extraction module includes: 1 position encoding module and 6 serially connected encoder modules; the position encoding module adds position information to the data from the data preprocessing unit and re-encodes it into a 256-dimensional feature vector as the input of the first encoder module; the latter encoder module uses the 256-dimensional feature vector of the previous encoder module as input and outputs a 256-dimensional feature vector after encoding; the 256-dimensional feature vectors output by the first 5 encoder modules are all called hidden layer feature vectors, and the hidden layer feature vectors output by the 5 encoder modules are retained. At the same time, the 5 encoder modules also output a hidden layer self-attention matrix (corresponding to the output of the multi-head self-attention module below).

[0027] Furthermore, the position encoding module uses a sine function and a cosine function to encode the relative position and absolute position of the input data, and the formula is expressed as follows:

[0028]

[0029]

[0030] Where, d model Indicates the number of dimensions of the input data (i.e. the total number of all inertial sensor coordinate axes), i∈[1, d model ] represents the i-th dimension, po represents the position of the data at the current moment in the entire input data window (i.e. the po-th sampling data).

[0031] Furthermore, the encoder module includes: a multi-head self-attention module, a forward transfer neural network module and a pooling module;

[0032] Multi-head self-attention module: It consists of multiple different self-attention modules; the self-attention module maps the input behavior data to the Query query vector, Key key vector and Value value vector through the convolution layer, and then calculates QK T , and then multiply it with V to get the output feature vector; among them, QK calculated by the multi-head self-attention module T The matrix contains the weight information of the eigenvector, which is recorded as the self-attention matrix; the formula is as follows:

[0033]

[0034] Where Q represents the Query vector, K represents the Key vector, V represents the Value vector, softmax is the activation function, and d represents the dimension of the input data;

[0035] Forward pass neural network module: Use two fully connected layers to perform matrix operations on the feature vectors output by the multi-head self-attention module and convert them into new feature vectors. The formula is as follows:

[0036] f(F)=max(0,FW1+b1)W2+b2

[0037] Where F represents the input feature vector, W1 represents the weight coefficient of the first fully connected layer, b1 represents the bias term of the first fully connected layer, W2 represents the weight coefficient of the second fully connected layer, and b2 represents the bias term of the second fully connected layer;

[0038] Pooling module: Use a 1-dimensional maximum pooling layer to calculate, that is, calculate the maximum output value for each element in a fixed-shape window (also called pooling window) of the input data.

[0039] Furthermore, the classifier module is used to perform inner product calculation on the 256-dimensional feature vector output by the feature extraction module and the hidden layer feature matrix of the classifier, and output an N-dimensional vector, where N represents the number of behavior categories, and the category corresponding to the maximum value on the N-dimensional vector is the classified behavior category; for each behavior category, multiple representative feature vectors are used to calculate the similarity between the 256-dimensional feature vector extracted by the feature extraction module and the behavior category, and multiple representative feature vectors are collectively referred to as a representative feature vector cluster; each behavior category has K representative feature vectors Δ c，k , where c represents the behavior category; calculate the input 256-dimensional feature vector X and the representative feature vector Δ of the behavior category c，k The similarity m C，k , and then for all m c，k Find the average similarity The formula is as follows:

[0040]

[0041] The representative feature vector cluster optimizes the similarity of each sample with its true category representation, and the nearest component analysis is used to optimize the loss. The formula is as follows:

[0042]

[0043] in, Represents the probability distribution corresponding to the true label category; Represents the probability distribution of the i-th (i≠y) class label; δ is a hyperparameter used to set the threshold; η is a hyperparameter used to adjust the model parameters; exp represents the exponential function with e as the base, Represents the feature loss value.

[0044] Furthermore, the augmented class training unit obtains the old behavior class representative sample set D from the storage unit. old , old behavior recognition model M o And the additional training task T i training samples; the augmented class training unit is used to train the old behavior recognition model M o Copy and get the initial new behavior recognition model M t , while fixing the old behavior recognition model M o Parameters; the old behavior category represents the sample set D old The training samples of the augmented training task are simultaneously input into the old behavior recognition model M o and the new behavior recognition model M t In the old behavior recognition model M o The output of the new behavior recognition model M t The output of the loss function is calculated, and the calculation result is used to guide the new behavior recognition model M t training, so that the new behavior recognition model M t It can identify old behavior categories and new behavior categories of the augmented training task; the augmented training unit outputs a small amount of training samples of the augmented training task, which are used to update the old behavior category representative sample set D stored in the storage unit. old , the new behavior recognition model M finally trained t Denoted as model M c And use it to replace the old behavior recognition model M o ;

[0045] After the incremental training task is completed, if a new incremental training task T is generated i+1 , then repeat the above process so that the model can recognize both old and new behavior categories.

[0046] Furthermore, the class enhancement training unit includes a knowledge distillation module and a data enhancement module;

[0047] The knowledge distillation module obtains the old behavior recognition model M o The output of the new behavior recognition model M t The output of the loss function is used to calculate the loss function, and the calculation result is used to guide the new behavior recognition model M t The knowledge distillation module includes: hidden layer feature vector distillation module and hidden layer self-attention matrix distillation module; input the behavior data into the new behavior recognition model M t and the old behavior recognition model M o In the new behavior recognition model M t and the old behavior recognition model M o Each will output its own hidden layer feature vector and hidden layer self-attention matrix;

[0048] The data enhancement module uses the old behavior category representative sample set D old Input to the new behavior recognition model M t In the output feature vector, the feature vectors of two samples are randomly selected and linearly combined according to the probability distribution. At the same time, the labels of each sample are linearly combined according to the same probability distribution. The feature vector after linear combination is added with random noise and label for back propagation to train the new behavior recognition model M. t , enhancing the generalization performance of the model.

[0049] Furthermore, the hidden layer feature vector distillation module is divided into two parts, one part distills the dimensional information of the hidden layer feature vector, and the other part distills the time sequence information of the hidden layer feature vector;

[0050] The dimensional information of the hidden layer feature vector is distilled as follows: t Output hidden layer feature vector and the old behavior recognition model M o Output hidden layer feature vector Calculate the loss for back propagation and train the new behavior recognition model M t , so that the new behavior recognition model M t Get the old behavior recognition model M o The knowledge of the old behavior category is expressed as follows:

[0051]

[0052] in, Represents the old behavior recognition model M o The output hidden layer feature vector, Represents the new behavior recognition model M t The output hidden layer feature vector; W represents the feature dimension, T represents the feature vector length, and l represents the number of samples in the batch;

[0053] The temporal information distillation of the hidden layer feature vector is as follows: the hidden layer feature vector output by the behavior recognition model has a temporal dependency, the temporal dependency of the hidden layer feature vector is distilled, and the law is found from the temporal correlation; the new behavior recognition model M t and the old behavior recognition model M o The hidden layer feature vectors of are each subjected to matrix transposition and multiplication, and then distilled. The formula is as follows:

[0054]

[0055]

[0056]

[0057] Among them, D KL represents the calculation of KL divergence, that is, the difference between two vectors; |T| represents the length of the feature vector; represents the feature vector dimension, ( and Introduction to distillation of dimensional information in hidden layer feature vectors); Represents the old behavior recognition model M o The hidden layer feature vector The weighted attention matrix obtained by matrix transposition and multiplication; Represents the new behavior recognition model M t The hidden layer feature vector The weighted attention matrix obtained by matrix transposition and multiplication; represents the calculated loss of timing correlation;

[0058] The input of the hidden layer self-attention matrix distillation module is the new action recognition model M t and the old behavior recognition model M o The hidden layer self-attention matrix outputted separately is expressed as follows:

[0059]

[0060] in, Represents the old behavior recognition model M o The output hidden layer self-attention matrix, Represents the new behavior recognition model M t The output hidden layer self-attention matrix, MSE (,) represents the calculation of the average Euclidean distance between two vectors, Γ represents the number of multi-head self-attention modules, Represents the loss value of the hidden layer self-attention matrix.

[0061] Furthermore, the data enhancement module includes data enhancement and noise enhancement based on linear combination;

[0062] The data enhancement based on linear combination is specifically as follows: during the class enhancement training process, the representative sample set in the storage unit is input into the new behavior recognition model M t , new behavior recognition model M t The output hidden layer feature vectors are linearly combined to obtain new feature vectors, thereby improving the generalization of the model by generating more samples; the formula is as follows:

[0063]

[0064]

[0065] Among them, x iand x j Represents the input to the new behavior recognition model M t The hidden layer feature vector, y i and y j Corresponding to x i and x j The classification label, represents the eigenvector after linear combination, Represents the label after linear combination, λ represents the hyperparameter, which obeys the normal distribution.

[0066] Furthermore, the noise enhancement is specifically to enhance the representation ability of the hidden layer feature vector, improve the generalization of the behavior recognition model, and the feature vector after linear combination Add random noise values to the new behavior recognition model M t Ability to maintain high recognition performance in the presence of interference; the formula is as follows:

[0067]

[0068] in, represents the eigenvector after linear combination, β is a hyperparameter that controls the influence of noise on the feature, x noise is noise, which follows a random distribution. Represents the hidden layer feature vector after adding noise.

[0069] The present invention provides an augmentable behavior recognition method based on data enhancement and knowledge distillation, comprising the following steps:

[0070] 1) Collecting the three-axis accelerometer and three-axis gyroscope data of the inertial sensor when the user is moving;

[0071] 2) Preprocess the collected behavioral data, crop the window, resample, and normalize the data using the standard normal distribution;

[0072] 3) Identify and judge the normalized behavior data. If there is new category behavior data, proceed to step 4); if it is within the range of identifiable behavior categories, output the recognition result and end;

[0073] 4) Collect samples of new behavior categories, train and update the new behavior recognition model, and obtain the new behavior recognition model M c And a small number of representative samples of new behavior categories, and add a small number of representative samples of new behavior categories to the representative sample set D of old behavior categories old ;

[0074] 5) Use the new behavior recognition model M c Identify the normalized behavior data and output the behavior recognition results.

[0075] Furthermore, in step 3), the behavioral data is encoded, feature extraction is performed, and a 256-dimensional feature vector is output; the inner product of the encoded 256-dimensional feature vector and the hidden layer feature matrix of the classifier is calculated to output an N-dimensional vector, where N represents the number of behavioral categories, and the category corresponding to the largest value on the N-dimensional vector is the classification result; if new category behavioral data is added, the category addition training task is started; conversely, the dimension with the largest probability on the N-dimensional vector represents the recognized category.

[0076] Furthermore, in step 4), the class-enhancing training task T is used. i Training samples of the old behavior recognition model M o and the old behavior category representative sample set D old Start the incremental training to obtain a new behavior recognition model M that can recognize old behavior categories and new behavior categories c and a small number of representative samples of new behavior categories; use a small number of representative samples of new behavior categories to update the representative sample set D of the old behavior categories old , delete the old behavior recognition model M o , store the new behavior recognition model M c , new behavior recognition model M c The old action recognition model M becomes the next category-increasing task o ;

[0077] For the old behavior recognition model M o Copy and get the initial new behavior recognition model M t , while fixing the old behavior recognition model M o Parameters, and then use the old behavior category to represent the sample set D old and augmentation training task T i The training samples are simultaneously input into the old behavior recognition model M o and the new behavior recognition model M t In the old behavior recognition model M o The output is used to guide the new behavior recognition model M t training, so that the new behavior recognition model M t Ability to recognize new behavioral categories in the incremental training task while maintaining high recognition performance for behavioral categories previously recognized in the incremental training task, that is, recognition of new behavioral categories without forgetting previously recognized behavioral categories;

[0078] Old behavior recognition model M o The output is used to guide the new behavior recognition model M t The training method is to calculate the old behavior recognition model M o The output hidden layer feature vector and hidden layer self-attention matrix are combined with the new behavior recognition model M tThe difference between the output hidden layer feature vector and the hidden layer self-attention matrix, i.e. knowledge distillation;

[0079] Calculate the old behavior recognition model M o Output hidden layer feature vector and new behavior recognition model M t The method of outputting the hidden layer feature vector is to calculate the KL divergence of the two;

[0080] Calculate the old behavior recognition model M o Output hidden layer self-attention matrix and new behavior recognition model M t The method of outputting the hidden layer self-attention matrix is to calculate the average Euclidean distance between the two;

[0081] Use the old behavior category to represent the sample set D old Input to the new behavior recognition model M t Perform data augmentation to generate training samples for old behavior categories.

[0082] Furthermore, the data enhancement method is based on linear combination of data enhancement and noise enhancement.

[0083] Furthermore, the old behavior category representative sample set D old The storage of old behavior categories represents the sample set D old The capacity is set to a fixed number of samples. After each incremental training, the center sample of the new behavior category training sample is calculated by clustering method, and the representative training samples are selected and stored in the storage unit. At the same time, the number of representative samples of the old behavior category is reduced, and the training samples with large differences from the center sample are deleted. The total number of representative samples of the old behavior category and the new behavior category is kept constant, and the representative sample set D is updated with the current old category samples and new category samples. old .

[0084] Beneficial effects of the present invention:

[0085] The present invention adopts a behavior recognition model based on the Transformer encoder, which is superior to traditional convolutional neural networks, recurrent neural networks and long short-term memory network models in recognition rate. First, an initial behavior recognition model can be obtained through initial training. Then, the initial (i.e., old) behavior recognition model is trained and fine-tuned in the new task, and a new behavior recognition model is obtained using knowledge distillation and data enhancement, wherein the new behavior recognition model and the initial behavior recognition model maintain the same structure and scale. In addition, the new behavior recognition model is used as the initial behavior recognition model when learning the next new task. This continuous cycle allows the behavior recognition model to continuously learn new tasks and always retains the memory of previously learned tasks.

[0086] When performing incremental behavior recognition based on the aforementioned behavior recognition model, each new task can be based on the previous behavior recognition model, avoiding repeated training of the entire behavior recognition model and improving its training efficiency. This incremental behavior recognition process ensures continuous learning performance and leverages resource cost savings. It also conforms to the dynamic scalability of user behavior in real life, maintaining sustainable incremental recognition performance for unknown new behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 Schematic diagram of the system of the present invention.

[0088] Figure 2 Schematic diagram of the feature extraction module.

[0089] Figure 3 Schematic diagram of knowledge distillation.

[0090] Figure 4 Schematic diagram of data augmentation. DETAILED DESCRIPTION

[0091] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0092] Reference Figure 1 As shown, the present invention is an augmentable behavior recognition system based on data enhancement and knowledge distillation, comprising: a data acquisition unit, a data preprocessing unit, a storage unit, a behavior recognition unit and an augmentation training unit;

[0093] A data acquisition unit, used to collect data from the three-axis accelerometer and three-axis gyroscope of the inertial sensor when the user is exercising;

[0094] A data preprocessing unit resamples the acquired accelerometer data and gyroscope data, clips the data length, normalizes the data value, and outputs the preprocessed behavioral data;

[0095] Wherein, the data preprocessing unit includes: a data resampling module, a data normalization and standardization module;

[0096] Data resampling module: aligns the behavioral data acquired by the data acquisition unit; performs linear interpolation on each behavioral data with inconsistent lengths. This operation uses an interpolation function and sets a fixed sampling length. The interpolation function will generate behavioral data with a fixed sampling length to achieve alignment of all behavioral data;

[0097] Data normalization and standardization module: Calculate the mean μ and variance σ of all resampled data, apply μ and σ to all training data, and use the following formula to normalize the data to [-1, 1] so that it obeys the standard normal distribution:

[0098]

[0099] Where D o is the normalized behavioral data, D i is the original behavioral data, μ represents the mean of the training data, and σ represents the variance of the training data; all test data are normalized and standardized using the mean and variance.

[0100] A storage unit, used to store a representative sample set of old behavior categories and an old behavior recognition model;

[0101] The storage unit stores the old behavior category representative sample set D old and the old behavior recognition model M o , where the old behavior category represents the sample set D old The capacity is set to a fixed number of samples: after each incremental training, the central sample of the new behavior category training sample is calculated by clustering method, and the representative training samples are selected and stored in the storage unit. At the same time, the number of representative samples of the old behavior category is reduced, and the training samples with large differences from the central sample are deleted. The total number of representative samples of the old behavior category and the new behavior category is kept constant, and the old behavior category representative sample set D is updated with the current old behavior category representative sample and the new behavior category representative sample. old .

[0102] A behavior recognition unit is used to recognize and classify the preprocessed behavior data. For new behavior categories, the number of output categories of the classifier is changed, and samples of the new behavior category and representative samples of the old behavior category are used as training samples to generate an augmented class training task, which is sent to the augmented class training unit.

[0103] The behavior recognition unit obtains the output behavior data from the data preprocessing unit. During the model training phase, if the behavior data contains new categories, the number of output categories of the classifier is changed. Samples of the new behavior categories and some representative samples of the old behavior categories are used as training samples to generate a category-increasing training task, which is sent to the category-increasing training unit. Otherwise, the behavior recognition result is output. During the testing phase, a classification result is output based on the behavior data.

[0104] Specifically, the behavior recognition unit includes: a feature extraction module and a classifier module;

[0105] The feature extraction module encodes the behavior data processed by the data preprocessing unit and outputs a 256-dimensional feature vector;

[0106] The classifier module performs inner product calculation on the encoded 256-dimensional feature vector and the hidden layer feature matrix of the classifier, and outputs an N-dimensional vector, where N represents the number of behavior categories. The category corresponding to the largest value in the N-dimensional vector is the classification result.

[0107] The feature extraction module encodes the behavioral data processed by the data preprocessing unit and outputs a 256-dimensional feature vector, which contains important feature information of the behavioral data and serves as the input of the classifier module; the feature extraction module includes: 1 position encoding module and 6 serially connected encoder modules; the position encoding module adds position information to the data from the data preprocessing unit and re-encodes it into a 256-dimensional feature vector as the input of the first encoder module; the latter encoder module uses the 256-dimensional feature vector of the previous encoder module as input and outputs a 256-dimensional feature vector after encoding; the 256-dimensional feature vectors output by the first 5 encoder modules are all called hidden layer feature vectors, and the hidden layer feature vectors output by the 5 encoder modules are retained. At the same time, the 5 encoder modules also output a hidden layer self-attention matrix (corresponding to the output of the multi-head self-attention module below).

[0108] The position encoding module uses a sine function and a cosine function to encode the relative position and absolute position of the input data. The formula is as follows:

[0109]

[0110]

[0111] Where, d model Indicates the number of dimensions of the input data (i.e. the total number of all inertial sensor coordinate axes), i∈[1, d model ] represents the i-th dimension, po represents the position of the data at the current moment in the entire input data window (i.e. the po-th sampling data).

[0112] The encoder module includes: a multi-head self-attention module, a forward transfer neural network module and a pooling module;

[0113] Multi-head self-attention module: It consists of multiple different self-attention modules; the self-attention module maps the input behavior data to the Query query vector, Key key vector and Value value vector through the convolution layer, and then calculates QK T , and then multiply it by V to get the output feature vector. Among them, QK calculated by the multi-head self-attention module TThe matrix contains the weight information of the eigenvector, which is recorded as the self-attention matrix; the formula is as follows:

[0114]

[0115] Where Q represents the Query vector, K represents the Key vector, V represents the Value vector, softmax is the activation function, and d represents the dimension of the input data;

[0116] Forward pass neural network module: Use two fully connected layers to perform matrix operations on the feature vector output by the multi-head self-attention module and convert it into a new feature vector. The formula is as follows:

[0117] f(F)=max(0,FW1+b1)W2+b2

[0118] Where F represents the input feature vector, W1 represents the weight coefficient of the first fully connected layer, b1 represents the bias term of the first fully connected layer, W2 represents the weight coefficient of the second fully connected layer, and b2 represents the bias term of the second fully connected layer;

[0119] Pooling module: Use a 1-dimensional maximum pooling layer to calculate, that is, calculate the maximum output value for each element in a fixed-shape window (also called pooling window) of the input data.

[0120] The above feature extraction module refers to Figure 2 shown.

[0121] The classifier module is used to perform inner product calculation on the 256-dimensional feature vector extracted from the feature extraction module and the hidden layer feature matrix of the classifier, and output an N-dimensional vector, where N represents the number of behavior categories. The category corresponding to the maximum value on the N-dimensional vector is the classified behavior category; for each behavior category, multiple representative feature vectors are used to calculate the similarity between the 256-dimensional feature vector extracted by the feature extraction module and the behavior category. Multiple representative feature vectors are collectively referred to as a representative feature vector cluster; each behavior category has K representative feature vectors Δ c，k , where c represents the behavior category; calculate the input 256-dimensional feature vector X and the representative feature vector Δ of the behavior category c，k The similarity m c，k , and then for all m c，k Find the average similarity The formula is as follows:

[0122]

[0123] The representative feature vector cluster optimizes the similarity of each sample with its true category representation, and the nearest component analysis is used to optimize the loss. The formula is as follows:

[0124]

[0125] in, Represents the probability distribution corresponding to the true label category; Represents the probability distribution of the i-th (i≠y) class label; δ is a hyperparameter used to set the threshold; η is a hyperparameter used to adjust the model parameters; exp represents the exponential function with e as the base, Represents the feature loss value;

[0126] The augmentation training unit obtains the old behavior recognition model stored in the storage unit, performs augmentation training on the old behavior recognition model using samples of the new behavior category and some representative samples of the old behavior category, and outputs a new behavior recognition model; the new behavior recognition model recognizes the old behavior category data or the new behavior category data and outputs a behavior recognition result;

[0127] The augmented class training unit obtains the old behavior class representative sample set D from the storage unit. old , old behavior recognition model M o And the additional training task T i training samples; the augmented class training unit is used to train the old behavior recognition model M o Copy and get the initial new behavior recognition model M t , while fixing the old behavior recognition model M o Parameters; the old behavior category represents the sample set D old The training samples of the augmented training task are simultaneously input into the old behavior recognition model M o and the new behavior recognition model M t In the old behavior recognition model M o The output of the new behavior recognition model M t The output of the loss function is calculated, and the calculation result is used to guide the new behavior recognition model M t training, so that the new behavior recognition model M t It can recognize both old behavior categories and new behavior categories of the augmented training task; the augmented training unit outputs a small amount of training samples of the augmented training task, which is used to update the representative sample set D stored in the storage unit. old , the new behavior recognition model M finally trained t Denoted as model M c And use it to replace the old behavior recognition model M o ;

[0128] After the incremental training task is completed, if a new incremental training task T is generated i+1 , then repeat the above process so that the model can recognize both old behavior categories and new behavior categories continuously.

[0129] Specifically, the class enhancement training unit includes a knowledge distillation module and a data enhancement module;

[0130] The knowledge distillation module obtains the old behavior recognition model M o The output of the new behavior recognition model M t The output of the loss function is used to calculate the loss function, and the calculation result is used to guide the new behavior recognition model M t The knowledge distillation module includes: hidden layer feature vector distillation module and hidden layer self-attention matrix distillation module; input the behavior data into the new behavior recognition model M t and the old behavior recognition model M o In the new behavior recognition model M t and the old behavior recognition model M o Each will output its own hidden layer feature vector and hidden layer self-attention matrix;

[0131] The data enhancement module uses the old behavior category representative sample set D old Input to the new behavior recognition model M t In the output feature vector, the feature vectors of two samples are randomly selected and linearly combined according to the probability distribution. At the same time, the labels of each sample are linearly combined according to the same probability distribution. The feature vector after linear combination is added with random noise and label for back propagation to train the new behavior recognition model M. t , enhancing the generalization performance of the model.

[0132] The hidden layer feature vector distillation module is divided into two parts, one part distills the dimensional information of the hidden layer feature vector, and the other part distills the temporal information of the hidden layer feature vector;

[0133] The dimensional information of the hidden layer feature vector is distilled as follows: t Output hidden layer feature vector and the old behavior recognition model M o Output hidden layer feature vector Calculate the loss for back propagation and train the new behavior recognition model M t , so that the new behavior recognition model M t Get the old behavior recognition model M o The knowledge of the old behavior category is expressed as follows:

[0134]

[0135] in, Represents the old behavior recognition model M o The output hidden layer feature vector, Represents the new behavior recognition model M tThe output hidden layer feature vector; W represents the feature dimension, T represents the feature vector length, and l represents the number of samples in the batch;

[0136] The temporal information distillation of the hidden layer feature vector is as follows: the hidden layer feature vector output by the behavior recognition model has a temporal dependency, the temporal dependency of the hidden layer feature vector is distilled, and the law is found from the temporal correlation; the new behavior recognition model M t and the old behavior recognition model M o The hidden layer feature vectors of are each subjected to matrix transposition and multiplication, and then distilled. The formula is as follows:

[0137]

[0138]

[0139]

[0140] Among them, D KL represents the calculation of KL divergence, that is, the difference between two vectors; |T| represents the input feature length; represents the feature vector dimension, ( and Introduction to distillation of dimensional information in hidden layer feature vectors); Represents the old behavior recognition model M o The hidden layer feature vector The weighted attention matrix obtained by matrix transposition and multiplication; Represents the new behavior recognition model M t The hidden layer feature vector The weighted attention matrix obtained by matrix transposition and multiplication; represents the calculated loss of timing correlation;

[0141] The input of the hidden layer self-attention matrix distillation module is the new action recognition model M t and the old behavior recognition model M o The hidden layer self-attention matrix outputted separately is expressed as follows:

[0142]

[0143] in, Represents the old behavior recognition model M o The output hidden layer self-attention matrix, Represents the new behavior recognition model M t The output hidden layer self-attention matrix, MSE (,) represents the calculation of the average Euclidean distance between two vectors, Γ represents the number of multi-head self-attention modules, Represents the loss value of the hidden layer self-attention matrix; the above knowledge distillation refers to Figure 3 shown.

[0144] The data enhancement module includes data enhancement and noise enhancement based on linear combination;

[0145] The data enhancement based on linear combination is specifically as follows: during the class enhancement training process, the representative sample set in the storage unit is input into the new behavior recognition model M t , new behavior recognition model M t The output hidden layer feature vectors are linearly combined to obtain new feature vectors, thereby improving the generalization of the model by generating more samples; the formula is as follows:

[0146]

[0147]

[0148] Among them, x i and x j Represents the input to the new behavior recognition model M t The hidden layer feature vector, y i and y j Corresponding to x i and x j The classification label, represents the eigenvector after linear combination, Represents the label after linear combination, λ represents the hyperparameter, which obeys the normal distribution.

[0149] The noise enhancement is specifically to enhance the representation ability of the hidden layer feature vector, improve the generalization of the behavior recognition model, and the feature vector after linear combination Add random noise values to the new behavior recognition model M t Ability to maintain high recognition performance in the presence of interference; the formula is as follows:

[0150]

[0151] in, represents the eigenvector after linear combination, β is a hyperparameter that controls the influence of noise on the feature, x noise is noise, which follows a random distribution. Represents the hidden layer feature vector after adding noise; the above data enhancement reference Figure 4 shown.

[0152] The present invention provides an augmentable behavior recognition method based on data enhancement and knowledge distillation, comprising the following steps:

[0153] 1) Collecting the three-axis accelerometer and three-axis gyroscope data of the inertial sensor when the user is moving;

[0154] 2) Preprocess the collected behavioral data, crop the window, resample, and normalize the data using the standard normal distribution;

[0155] 3) Identify and judge the normalized behavior data. If there is new category behavior data, proceed to step 4); if it is within the range of identifiable behavior categories, output the recognition result and end;

[0156] 4) Collect samples of new behavior categories, train and update the new behavior recognition model, and obtain the new behavior recognition model M c And a small number of representative samples of new behavior categories, and add a small number of representative samples of new behavior categories to the representative sample set D of old behavior categories old ;

[0157] 5) Use the new behavior recognition model M c Identify the normalized behavior data and output the behavior recognition results.

[0158] In step 3), the behavioral data is encoded, feature extraction is performed, and a 256-dimensional feature vector is output; the inner product of the encoded 256-dimensional feature vector and the hidden layer feature matrix of the classifier is calculated to output an N-dimensional vector, where N represents the number of behavioral categories, and the category corresponding to the largest value on the N-dimensional vector is the classification result; if new category behavioral data is added, the category addition training task is started; conversely, the dimension with the largest probability on the N-dimensional vector represents the recognized category.

[0159] Wherein, in step 4), the class-enhancing training task T is used. i Training samples of the old behavior recognition model M o and the old behavior category representative sample set D old Start the incremental training to obtain a new behavior recognition model M that can recognize old behavior categories and new behavior categories c and a small number of representative samples of new behavior categories; use a small number of representative samples of new behavior categories to update the representative sample set D of the old behavior categories old , delete the old behavior recognition model M o , store the new behavior recognition model M c , new behavior recognition model M c The old action recognition model M becomes the next category-increasing task o ;

[0160] Behavior recognition model M o Copy and get the initial new behavior recognition model M t , while fixing the old behavior recognition model M oParameters, and then use the old behavior category to represent the sample set D old and augmentation training task T i The training samples are simultaneously input into the old behavior recognition model M o and the new behavior recognition model M t In the old behavior recognition model M o The output is used to guide the new behavior recognition model M t training, so that the new behavior recognition model M t Ability to recognize new behavioral categories in the incremental training task while maintaining high recognition performance for behavioral categories previously recognized in the incremental training task, that is, recognition of new behavioral categories without forgetting previously recognized behavioral categories;

[0161] Old behavior recognition model M o The output is used to guide the new behavior recognition model M t The training method is to calculate the old behavior recognition model M o The output hidden layer feature vector and hidden layer self-attention matrix are combined with the new behavior recognition model M t The difference between the output hidden layer feature vector and the hidden layer self-attention matrix, i.e. knowledge distillation;

[0162] Calculate the old behavior recognition model M o Output hidden layer feature vector and new behavior recognition model M t The method of outputting the hidden layer feature vector is to calculate the KL divergence of the two;

[0163] Calculate the old behavior recognition model M o Output hidden layer self-attention matrix and new behavior recognition model M t The method of outputting the hidden layer self-attention matrix is to calculate the average Euclidean distance between the two;

[0164] Use the old behavior category to represent the sample set D old Input to the new behavior recognition model M t Perform data augmentation to generate training samples for old behavior categories.

[0165] The data enhancement method is based on linear combination of data enhancement and noise enhancement.

[0166] Among them, the old behavior category represents the sample set D old The storage of old behavior categories represents the sample set D oldThe capacity is set to a fixed number of samples. After each incremental training, the center sample of the new behavior category training sample is calculated by clustering method, and the representative training samples are selected and stored in the storage unit. At the same time, the number of representative samples of the old behavior category is reduced, and the training samples with large differences from the center sample are deleted. The total number of representative samples of the old behavior category and the new behavior category is kept constant, and the representative sample set D is updated with the current old category samples and new category samples. old .

[0167] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principles of the present invention. These improvements should also be considered as the scope of protection of the present invention.

Claims

1. An augmentable behavior recognition system based on data augmentation and knowledge distillation, characterized by: include: Data acquisition unit, data preprocessing unit, storage unit, behavior recognition unit and category enhancement training unit; A data acquisition unit, used to collect data from the three-axis accelerometer and three-axis gyroscope of the inertial sensor when the user is moving; A data preprocessing unit resamples the acquired accelerometer data and gyroscope data, clips the data length, normalizes the data value, and outputs the preprocessed behavioral data; A storage unit, used to store a representative sample set of old behavior categories and an old behavior recognition model; A behavior recognition unit is used to recognize and classify the preprocessed behavior data. For new behavior categories, the number of output categories of the classifier is changed, and samples of the new behavior category and representative samples of the old behavior category are used as training samples to generate an augmented class training task, which is sent to the augmented class training unit. The augmentation training unit obtains the old behavior recognition model stored in the storage unit, performs augmentation training on the old behavior recognition model using samples of the new behavior category and some representative samples of the old behavior category, and outputs a new behavior recognition model; the new behavior recognition model recognizes the old behavior category data or the new behavior category data and outputs a behavior recognition result; The behavior recognition unit includes: a feature extraction module and a classifier module; The feature extraction module encodes the behavior data processed by the data preprocessing unit and outputs a 256-dimensional feature vector; The classifier module calculates the inner product of the encoded 256-dimensional feature vector and the hidden layer feature matrix of the classifier, and outputs an N-dimensional vector, where N represents the number of behavior categories. The category corresponding to the largest value in the N-dimensional vector is the classification result; The class enhancement training unit includes a knowledge distillation module and a data enhancement module; The knowledge distillation module obtains the old behavior recognition model M o The output of the new behavior recognition model M t The output of the loss function is used to calculate the loss function, and the calculation result is used to guide the new behavior recognition model M t The knowledge distillation module includes: hidden layer feature vector distillation module and hidden layer self-attention matrix distillation module; input the behavior data into the new behavior recognition model M t and the old behavior recognition model M o In the new behavior recognition model M t and the old behavior recognition model M o Each will output its own hidden layer feature vector and hidden layer self-attention matrix; The data enhancement module uses the old behavior category representative sample set D old Input to the new behavior recognition model M t In the output feature vector, the feature vectors of two samples are randomly selected and linearly combined according to the probability distribution. At the same time, the labels of each sample are linearly combined according to the same probability distribution. The feature vector after linear combination is added with random noise and label for back propagation to train the new behavior recognition model M. t , enhance the generalization performance of the model; The hidden layer feature vector distillation module is divided into two parts, one part distills the dimensional information of the hidden layer feature vector, and the other part distills the temporal information of the hidden layer feature vector; The dimensional information of the hidden layer feature vector is distilled as follows: t Output hidden layer feature vector and the old behavior recognition model M o Output hidden layer feature vector Calculate the loss for back propagation and train the new behavior recognition model M t , so that the new behavior recognition model M t Get the old behavior recognition model M o The knowledge of the old behavior category is expressed as follows: in, Represents the old behavior recognition model M o The output hidden layer feature vector, Represents the new behavior recognition model M t The output hidden layer feature vector; W represents the feature dimension, T represents the feature vector length, and l represents the number of samples in the batch; The temporal information distillation of the hidden layer feature vector is as follows: the hidden layer feature vector output by the behavior recognition model has a temporal dependency, the temporal dependency of the hidden layer feature vector is distilled, and the law is found from the temporal correlation; the new behavior recognition model M t and the old behavior recognition model M o The hidden layer feature vectors of are each subjected to matrix transposition and multiplication, and then distilled. The formula is as follows: Among them, D KL represents the calculation of KL divergence, that is, the difference between two vectors; |T| represents the length of the feature vector; represents the feature vector dimension; Represents the old behavior recognition model M o The hidden layer feature vector The weighted attention matrix obtained by matrix transposition and multiplication; Represents the new behavior recognition model M t The hidden layer feature vector The weighted attention matrix obtained by matrix transposition and multiplication; represents the calculated loss of timing correlation; The input of the hidden layer self-attention matrix distillation module is the new behavior recognition model M t and the old behavior recognition model M o The hidden layer self-attention matrix outputted separately is expressed as follows: in, Represents the old behavior recognition model M o The output hidden layer self-attention matrix, Represents the new behavior recognition model M t The output hidden layer self-attention matrix, MSE (,) represents the average Euclidean distance between two vectors, Γ represents the number of multi-head self-attention modules, Represents the loss value of the hidden layer self-attention matrix; The data enhancement module includes data enhancement and noise enhancement based on linear combination; The data enhancement based on linear combination is specifically as follows: during the class enhancement training process, the representative sample set in the storage unit is input into the new behavior recognition model M t , new behavior recognition model M t The output hidden layer feature vectors are linearly combined to obtain new feature vectors, thereby improving the generalization of the model by generating more samples; the formula is as follows: Among them, x i and x j Represents the input to the new behavior recognition model M t The hidden layer feature vector, y i and y j Corresponding to x i and x j The classification label, represents the eigenvector after linear combination, Represents the label after linear combination, λ represents the hyperparameter, which obeys the normal distribution.

2. The augmentable behavior recognition system based on data augmentation and knowledge distillation according to claim 1 is characterized in that: The data preprocessing unit includes: a data resampling module, a data normalization and standardization module; Data resampling module: aligns the behavioral data acquired by the data acquisition unit; performs linear interpolation on each behavioral data with inconsistent lengths. This operation uses an interpolation function and sets a fixed sampling length. The interpolation function will generate behavioral data with a fixed sampling length to achieve alignment of all behavioral data; Data normalization and standardization module: Calculate the mean μ and variance σ of all resampled data, apply μ and σ to all training data, and use the following formula to normalize the data to [-1, 1] so that it obeys the standard normal distribution: Where D o is the normalized behavioral data, D i is the original behavioral data, μ represents the mean of the training data, and σ represents the variance of the training data; all test data are normalized and standardized using the mean and variance.

3. The augmentable behavior recognition system based on data augmentation and knowledge distillation according to claim 1 is characterized in that: The storage unit stores the old behavior category representative sample set D old and the old behavior recognition model M o , where the old behavior category represents the sample set D old The capacity is set to a fixed number of samples: after each incremental training, the central sample of the new behavior category training sample is calculated by clustering method, and the representative training samples are selected and stored in the storage unit. At the same time, the number of representative samples of the old behavior category is reduced, and the training samples with large differences from the central sample are deleted. The total number of representative samples of the old behavior category and the new behavior category is kept constant, and the old behavior category representative sample set D is updated with the current old behavior category representative sample and the new behavior category representative sample. old .

4. The augmentable behavior recognition system based on data augmentation and knowledge distillation according to claim 1 is characterized in that: The feature extraction module encodes the behavioral data processed by the data preprocessing unit and outputs a 256-dimensional feature vector, which contains important feature information of the behavioral data and serves as the input of the classifier module; the feature extraction module includes: 1 position encoding module and 6 serially connected encoder modules; the position encoding module adds position information to the data from the data preprocessing unit and re-encodes it into a 256-dimensional feature vector as the input of the first encoder module; the latter encoder module uses the 256-dimensional feature vector of the previous encoder module as input and outputs a 256-dimensional feature vector after encoding; the 256-dimensional feature vectors output by the first 5 encoder modules are all called hidden layer feature vectors, and the hidden layer feature vectors output by the 5 encoder modules are retained. At the same time, the 5 encoder modules also output a hidden layer self-attention matrix.

5. The augmentable behavior recognition system based on data augmentation and knowledge distillation according to claim 1 is characterized in that: The classifier module is used to perform inner product calculation on the 256-dimensional feature vector output by the feature extraction module and the hidden layer feature matrix of the classifier, and output an N-dimensional vector, where N represents the number of behavior categories. The category corresponding to the maximum value on the N-dimensional vector is the classification result; for each behavior category, multiple representative feature vectors are used to calculate the similarity between the 256-dimensional feature vector extracted by the feature extraction module and the behavior category. Multiple representative feature vectors are collectively called a representative feature vector cluster; each behavior category has K representative feature vectors Δ c,k , where c represents the behavior category; calculate the input 256-dimensional feature vector X and the representative feature vector Δ of the behavior category c,k The similarity m c,k , and then for all m c,k Find the average similarity The formula is as follows: The representative feature vector cluster optimizes the similarity of each sample with its true category representation, and the nearest component analysis is used to optimize the loss. The formula is as follows: in, Represents the probability distribution corresponding to the true label category; Represents the probability distribution of the i-th class label; δ is a hyperparameter used to set the threshold; η is a hyperparameter used to adjust the model parameters; exp represents the exponential function with e as the base, Represents the feature loss value.

6. The augmentable behavior recognition system based on data augmentation and knowledge distillation according to claim 1 is characterized in that: The augmented training unit obtains the old behavior category representative sample set D from the storage unit. old , old behavior recognition model M o And the additional training task T i training samples; the augmented class training unit is used to train the old behavior recognition model M o Copy and get the initial new behavior recognition model M t , while fixing the old behavior recognition model M o Parameters; the old behavior category represents the sample set D old The training samples of the augmented training task are simultaneously input into the old behavior recognition model M o and the new behavior recognition model M t In the old behavior recognition model M o The output of the new behavior recognition model M t The output of the loss function is calculated, and the calculation result is used to guide the new behavior recognition model M t training, so that the new behavior recognition model M t Ability to identify old behavioral categories and new behavioral categories in incremental training tasks; The augmented training unit outputs a small amount of training samples for augmented training tasks, which are used to update the old behavior category representative sample set D stored in the storage unit. old , the new behavior recognition model M finally trained t Denoted as model M c And use it to replace the old behavior recognition model M o ; After the incremental training task is completed, if a new incremental training task T is generated i+1 , then repeat the above process so that the model can recognize both old and new behavior categories.

7. A method for augmentable behavior recognition based on data augmentation and knowledge distillation, based on the system of claim 1, characterized in that: The steps are as follows: 1) Collecting the three-axis accelerometer and three-axis gyroscope data of the inertial sensor when the user is moving; 2) Preprocess the collected behavioral data, crop the window, resample, and normalize the data using the standard normal distribution; 3) Identify the normalized behavior data. If new behavior category data exists, proceed to step 4); If it is within the range of recognizable behavior categories, the recognition result is output and the process ends; 4) Collect new behavior category samples, train and update the new behavior recognition model, and obtain the new behavior recognition model M c And a small number of representative samples of new behavior categories, and add a small number of representative samples of new behavior categories to the representative sample set D of old behavior categories old ; 5) Use the new behavior recognition model M c Identify the normalized behavior data and output the behavior recognition results.

8. The augmentable behavior recognition method based on data enhancement and knowledge distillation according to claim 7 is characterized in that: In step 4), the training samples of the augmented training task are used, and the old behavior recognition model M o and the old behavior category representative sample set D old Start the incremental training to obtain a new behavior recognition model M that can recognize old behavior categories and new behavior categories c and a small representative sample of new behavioral categories; Use a small number of representative samples of new behavior categories to update the representative sample set D of the old behavior category old , delete the old behavior recognition model M o , store the new behavior recognition model M c , new behavior recognition model M c The old action recognition model M becomes the next category-increasing task o ; For the old behavior recognition model M o Copy and get the initial new behavior recognition model M t , fix the old behavior recognition model M o Parameters, and then use the old behavior category to represent the sample set D old The training samples of the augmented training task are simultaneously input into the old behavior recognition model M o and the new behavior recognition model M t In the old behavior recognition model M o The output is used to guide the new behavior recognition model M t training, so that the new behavior recognition model M t Ability to identify old behavioral categories and new behavioral categories in incremental training tasks; Old behavior recognition model M o The output is used to guide the new behavior recognition model M t The training method is to calculate the old behavior recognition model M o The output hidden layer feature vector and hidden layer self-attention matrix are combined with the new behavior recognition model M t The difference between the output hidden layer feature vector and the hidden layer self-attention matrix, i.e. knowledge distillation; Calculate the old behavior recognition model M o Output hidden layer feature vector and new behavior recognition model M t The difference between the output hidden layer feature vectors is calculated by KL divergence between the two; Calculate the old behavior recognition model M o Output hidden layer self-attention matrix and new behavior recognition model M t The difference between the output hidden layer self-attention matrices is calculated by calculating the average Euclidean distance between the two; Use the old behavior category to represent the sample set D old Input to the new behavior recognition model M t Perform data augmentation to generate training samples for old behavior categories.

Images