Human activity identification method based on instance migration and attention distillation network

Abstract

The invention relates to a human activity identification method based on instance migration and an attention distillation network. The method comprises the following steps: carrying out replacement sampling on target domain samples in a target domain sample data set to generate a plurality of sub-data set training base learners; evaluating and calculating the weight of the trained base learner, and adding the source domain samples of which the weight is greater than a threshold value into a candidate source domain set; pre-training a teacher network in the human activity recognition model by adopting the candidate source domain set; and merging the target domain sample data set and the candidate source domain set, inputting the features of the samples into a student network and a teacher network in the human activity identification model, calculating classification loss and distillation loss, and finally updating parameters of the student network through back propagation. Obtaining a trained human activity recognition model of the distillation learning network based on the attention mechanism, inputting the user acceleration data into the trained human activity recognition model for activity recognition, and outputting the activity type of the user; therefore, the accuracy of human activity identification is improved.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a method for human activity recognition based on instance migration and attention distillation network. Background Art

[0002] With the continuous development of computer science, human activity recognition has become a hot research field. Human activity recognition analysis has great research significance in the fields of medicine, security, human-computer interaction, etc. At present, most activity recognition algorithms are based on wireless sensors to collect activity signals and use signal processing algorithms for recognition, such as machine learning.

[0003] In traditional machine learning, for some more complex problems or scenarios, the model usually needs to be trained on a large amount of labeled data to achieve good performance. However, in many practical applications, it is difficult and expensive to obtain enough labeled data. Faced with the contradiction between model complexity and data scarcity, transfer learning provides a way of thinking, which simulates the human ability to "learn from one example and apply it to other situations" and uses prior knowledge and related examples encountered before to overcome the problem of insufficient current data. Formally speaking, transfer learning aims to borrow knowledge learned in the source domain to improve the learning effect in the target domain, where the source domain usually has some similarity with the target domain or contains some transferable knowledge.

[0004] According to different carriers of knowledge transfer, transfer learning methods can be divided into several categories such as instance-based transfer, feature-based transfer, and model-based transfer, but there are some shortcomings. For example, instance-based methods usually compare based on the similarity between samples. When the number of source domain instances is large, the time complexity of the algorithm will be relatively large; feature-based methods often require domain knowledge when aligning or screening features, and important information may be lost during the mapping process of feature space, thereby increasing the risk of negative transfer; the effectiveness of pre-trained models in model-based methods depends on the similarity between domains, and too large differences may lead to poor transfer effects.

[0005] Therefore, the current transfer learning method is used to train the human activity recognition model based on the machine learning algorithm, resulting in a low accuracy rate of human activity recognition. Summary of the invention

[0006] Based on this, it is necessary to provide a human activity recognition method based on instance transfer and attention distillation network that can improve the accuracy of human activity recognition in response to the above technical problems.

[0007] A human activity recognition method based on instance transfer and attention distillation network, the method comprising:

[0008] Get the user's acceleration data;

[0009] Input the acceleration data into a trained human activity recognition model based on an attention mechanism distillation learning network for activity recognition, and output the user's activity type;

[0010] The training method of the human activity recognition model is:

[0011] Performing sampling with replacement on the target domain samples in the target domain sample data set to generate multiple sub-data sets, and training a base learner on each sub-data set to obtain a trained base learner;

[0012] Evaluate the trained base learner, calculate the weight of the trained base learner according to the evaluation result to screen source domain samples, and add the source domain samples whose weight is greater than a threshold into the candidate source domain set;

[0013] Pre-training a teacher network in a human activity recognition model using a candidate source domain set to learn domain-invariant features and freeze parameters of the teacher network;

[0014] The target domain sample dataset is merged with the candidate source domain set, and the features of the samples in the merged dataset are input into the student network and teacher network in the human activity recognition model. The features of the samples are reweighted using the attention mechanism, and the classification loss and distillation loss are calculated respectively. Finally, the parameters of the student network are updated through back propagation to obtain a trained human activity recognition model based on the distillation learning network with attention mechanism.

[0015] In one embodiment, the step of performing sampling with replacement on the target domain samples in the target domain sample data set to generate multiple sub-data sets, and training a base learner on each sub-data set to obtain a trained base learner includes:

[0016] Step 1.1, collect the acceleration data of the target population as the target domain samples, and construct the target domain sample dataset T, {T|(x1, y1), (x2, y2)....(x n ,y n )}, where x i Represents the characteristics of the i-th target domain sample, y i represents the true label of the i-th target domain sample, i∈1, 2, 3…n, n is the number of target domain samples, and the target domain sample dataset T is self-sampled with replacement to generate N T Sub-dataset T i , N T is a hyperparameter, each sub-dataset T i The number of samples is the same as the number of samples in the target domain sample data set, that is, {T i |(xi 1, y i 1), (x i 2, y i 2)....(x i n ,y i n )};

[0017] Step 1.2: Select a deep learning or machine learning model structure to implement the classification task and initialize the model parameters. T Sub-dataset T i Parallel training, get N T Different trained base learners i .

[0018] In one embodiment, evaluating the trained base learner, calculating the weight of the trained base learner according to the evaluation result to screen source domain samples, and adding the source domain samples with the weight greater than a threshold to a candidate source domain set includes:

[0019] Step 2.1: For each trained base learner i , use the sample data set T of the target domain for evaluation, and obtain the error rate e of the corresponding trained base learner on the target domain i , according to the error rate e i Calculate the trained base learner Learner i The screening weight w i , screening weight w i The expression is:

[0020]

[0021] Step 2.2: Collect acceleration data of other people as source domain samples to form a source domain sample dataset S, {S|(x1, y 1) ,(x2,y2)....(x m ,y m )}, where x j Represents the features of the j-th source domain sample, y j represents the true label of the jth source domain sample, j∈1, 2, 3…m, m is the number of source domain samples, and each time a single source domain sample (x j ,y j ), the feature x of the source domain sample j Input to each trained base learner i In the example above, we get the prediction result of the source domain sample. Combine the prediction results of the source domain sample in all base learners and their corresponding weights w i , calculate the voting rate of the source domain sample vote_rate(x j ), the voting rate of the source domain sample vote_rate(x j ) is:

[0022]

[0023] Wherein, I(·) is a conditional judgment function, and the conditional judgment of the conditional judgment function is:

[0024]

[0025] Step 2.3: According to the set screening threshold σ, if the voting rate of the current source domain sample vote_rate(x j ) is greater than the threshold σ, the source domain sample is added to the candidate source domain set S c Otherwise, it is considered that the source domain sample has a negative effect on the target domain training and is not included in the candidate source domain set S c Finally, the selected candidate source domain set S c contains s samples, that is, {S c |(x1,y1),(x2,y2)....(x s ,y s )}.

[0026] In one embodiment, the step of using a candidate source domain set to pre-train a teacher network in a human activity recognition model, learning domain-invariant features, and freezing parameters of the teacher network includes:

[0027] Step 3.1: Use the candidate source domain set S c The teacher network TeaNet of the human activity recognition model is pre-trained, where for the candidate source domain set S c The characteristics of the source domain samples x j , j∈1, 2, 3...s, first enter the feature extraction layer of the teacher network TeaNet Then we get the feature layer embedding Feature layer embedding Then the classification layer The final predicted value is obtained

[0028] Step 3.2: Based on the final predicted value and true label y j The classification loss is calculated and the parameters of the teacher network are updated. The optimization goal of updating the parameters of the teacher network is:

[0029]

[0030] in, and Corresponding to the teacher network feature extraction layer The classification layer The parameters to be optimized; E ( x j, y j)～Sc [.] represents the candidate source domain set S c The mathematical expectation of the sampled loss values, which is based on the variables Calculated, indicating the predicted value The loss function value between the actual value y of the label;

[0031] Step 3.3: Freeze the parameters of the teacher network to obtain the pre-trained teacher network TeaNet.

[0032] In one of the embodiments, the target domain sample data set is merged with the candidate source domain set, the features of the samples in the merged data set are input into the student network and the teacher network in the human activity recognition model, the features of the samples are reweighted using the attention mechanism, and finally the classification loss and the distillation loss are calculated respectively, and finally the parameters of the student network are updated by back propagation to obtain the trained human activity recognition model based on the attention mechanism distillation learning network, including:

[0033] Step 4.1, merge the target domain sample dataset with the candidate source domain set to obtain a merged dataset CmbSet, where the number of samples in the merged dataset is N=n+s;

[0034] Step 4.2: Initialize the parameters of the student network StuNet in the human activity recognition model. The parameter structure of the student network is mainly divided into three parts: the feature extraction layer parameters of the student network Attention layer parameters and classification layer parameters

[0035] Step 4.3: For the sample (x i ,y i ), i∈1, 2, 3...N, first input feature x i to the student network StuNet, through the feature extraction layer of the student network Get preliminary embedding features

[0036] Step 4.4: Calculate preliminary embedding features using additive attention The attention weight α qk , after the attention layer of the student network After the action of

[0037] Step 4.5: Reweight the features Input to the classification layer of the student network Get the predicted value of the student network and calculate the classification loss L of the student network cls , the classification loss L cls The expression is:

[0038]

[0039] Among them, N is the total number of samples in the merged data set CmbSet, y i is the true label of the i-th sample, x i is the feature of the i-th sample, For the x i Prediction value after passing through the student network;

[0040] Step 4.6, the feature x i At the same time, it is input into the pre-trained teacher network TeaNet, and after feature extraction, it is obtained calculate and Distillation loss L dst , the calculation formula is:

[0041]

[0042] in, is feature x i The output value of the feature extraction layer of the teacher network, is feature x i Output value of the feature extraction layer of the student network;

[0043] Step 4.7. Finally, calculate the total loss of the student network and back-propagate the updated parameters. The optimization goal of back-propagation updated parameters is:

[0044]

[0045] in, Represents the mathematical expectation of the total loss calculated by sampling from the merged dataset CmbSet. The total loss is composed of the student network classification loss L cls and distillation loss L dst It consists of two parts, where λ is the distillation coefficient of the teacher network, ranging from 0 to 1;

[0046] Step 4.8: Repeat steps 4.2 to 4.7 until the preset maximum number of training batches is exceeded. Save the model and evaluate it in the test set to obtain a trained human activity recognition model based on the attention mechanism-based distillation learning network.

[0047] In one embodiment, step 4.4 comprises:

[0048] First, according to the embedded features Initialize the Query (Q), Key (K), and Value (V) vectors respectively, where the Q moment vector represents the important features to be concerned about; the K vector represents the associated features in the feature space; the V vector represents the feature value associated with the K vector. Initially, the values ​​of Q, K, and V are equal to Right now:

[0049] Embedding features Perform linear transformation to obtain the transformed Query (Q′) and Key (K′) vectors, respectively, expressed as:

[0050]

[0051]

[0052] Among them, W q and W k is a learnable linear transformation matrix used to map input features to a new feature space, b q and b k is the bias term;

[0053] The linearly transformed Q′ and K′ are added together, and the similarity score e between the qth component in the Q vector and the kth component in the K vector is calculated through a learnable weight vector v qk , which measures the extent to which the feature at position q in the Q vector focuses on the feature at position k in the K vector, and the similarity score e qk The expression is:

[0054] e qk =v T tanh(Q′+K′)

[0055] Among them, q and k represent the position index of Q vector and K vector, v T It is the vector after the weight vector v is transposed. Tanh is the hyperbolic tangent activation function, which maps the input to the range of (-1, 1). The calculation formula of the hyperbolic tangent activation function is:

[0056]

[0057] Among them, x represents the variable of the hyperbolic tangent activation function, that is, x refers to Q′+K′;

[0058] The similarity score e qk Normalize to get the attention weight α between different components in the feature qk , which represents the importance of the qth component to the kth component in the embedded feature, t represents the feature vector dimension, and the expression is:

[0059]

[0060] Use attention weight α qk Embedding features Perform weighted summation to obtain the re-weighted features Reweighted Features The calculation formula is:

[0061]

[0062] Among them, t is the number of components of the embedded feature, α qk is the calculated attention weight, V k is the component of the V vector at position k.

[0063] The human activity recognition method based on instance transfer and attention distillation network generates multiple sub-datasets by sampling with replacement on the target domain samples in the target domain sample data set, and trains a base learner on each sub-dataset to obtain a trained base learner; the trained base learner is evaluated, and the weight of the trained base learner is calculated according to the evaluation result to screen the source domain samples, and the source domain samples with weights greater than a threshold are added to the candidate source domain set; the candidate source domain set is used to pre-train the teacher network in the human activity recognition model to learn the invariant features in the domain and freeze the parameters of the teacher network; the target domain sample data set is merged with the candidate source domain set, and the features of the samples in the merged data set are input into the student network in the human activity recognition model The network and the teacher network use the attention mechanism to reweight the features of the samples, and finally calculate the classification loss and distillation loss respectively. Finally, the parameters of the student network are updated through back propagation to obtain a trained human activity recognition model based on the distillation learning network of the attention mechanism, and then screen out samples with higher relevance to the target domain to optimize the knowledge transfer of source domain and target domain data, improve the performance of the model in the target domain and reduce the negative transfer phenomenon, so that the trained model has higher accuracy and stronger generalization, and then obtain the user's acceleration data; input the acceleration data into the trained human activity recognition model based on the distillation learning network of the attention mechanism for activity recognition, and output the user's activity type; thereby improving the accuracy of human activity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of a process of a human activity recognition method based on instance transfer and attention distillation network in one embodiment;

[0065] Figure 2 A schematic diagram of a training process of a human activity recognition model in one embodiment;

[0066] Figure 3 is a structural diagram of a human activity recognition model in one embodiment;

[0067] Figure 4 This is the data distribution in one embodiment using H as the source domain and D as the target domain;

[0068] Figure 5 The loss values ​​of three different settings of an experiment with H as the source domain and D as the target domain are compared in one embodiment, specifically including the following three groups of data processing strategies: (1) using only target domain data; (2) directly merging the source domain and target domain data; (3) filtering the source domain samples and then merging them with the target domain. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] In one embodiment, Figure 1 As shown, a human activity recognition method based on instance transfer and attention distillation network is provided, including the following steps:

[0071] Step S220, obtaining the user's acceleration data.

[0072] The acceleration data of the user may be acceleration data of the user through a sensor.

[0073] Step S240: Input the acceleration data into the trained human activity recognition model of the attention mechanism-based distillation learning network for activity recognition, and output the user's activity type.

[0074] Among them, activity types may include lying, walking, ascending and descending, etc.

[0075] Among them, the training method of the human activity recognition model based on the distillation learning network of the attention mechanism is as follows: first, the target domain data is sampled with replacement to generate multiple sub-datasets of the same size as the original target domain data set. Based on a specific model, training is performed on each sub-dataset, multiple base learners are obtained and saved, and the screening weight of each base learner is obtained by evaluating the original target domain data. Then, each source domain sample is input into all base learners for prediction in turn, and the final voting rate is calculated according to the weight. According to the preset screening threshold, the source domain samples greater than the threshold are finally put into the candidate set. The screened source domain candidate set and the original target domain data set are merged as the basis for subsequent training. First, the teacher network in the distillation network is pre-trained with the source domain data to fully learn the invariant features in the domain, and the parameters of the teacher network are frozen after the training is completed. Next, the student network of the main part of the network is trained, and the data in the mixed training set are input into the student and teacher networks in parallel and then the losses are calculated respectively. First, the data is fed into the feature embedding layer, attention learning layer and classification layer of the network to calculate the classification loss of the student network. Then the data will also enter the feature extraction layer of the teacher network to obtain the feature embedding of the teacher network. The teacher network feature embedding and the student network feature embedding of the same data are calculated to obtain the distillation loss. Finally, the student network classification loss and distillation loss are added together to calculate the total loss, and then back propagation is performed to update the student network parameters.

[0076] In one embodiment, Figure 2 As shown, the training method of the human activity recognition model is:

[0077] The target domain samples in the target domain sample data set are sampled with replacement to generate multiple sub-data sets, and a base learner is trained on each sub-data set to obtain a trained base learner; the trained base learner is evaluated, and the weight of the trained base learner is calculated according to the evaluation result to screen the source domain samples, and the source domain samples with weights greater than a threshold are added to the candidate source domain set; the candidate source domain set is used to pre-train the teacher network in the human activity recognition model to learn the invariant features in the domain and freeze the parameters of the teacher network; the target domain sample data set is merged with the candidate source domain set, and the features of the samples in the merged data set are input into the student network and the teacher network in the human activity recognition model, and the features of the samples are re-weighted using the attention mechanism, and finally the classification loss and the distillation loss are calculated respectively, and finally the parameters of the student network are updated through back propagation to obtain a trained human activity recognition model based on the distillation learning network of the attention mechanism.

[0078] In one embodiment, the step of performing sampling with replacement on the target domain samples in the target domain sample data set to generate multiple sub-data sets, and training a base learner on each sub-data set to obtain a trained base learner includes:

[0079] Step 1.1, collect the acceleration data of the target population as the target domain samples, and construct the target domain sample dataset T, {T|(x1, y1), (x2, y2)....(x n ,y n )}, where x i Represents the characteristics of the i-th target domain sample, y i represents the true label of the i-th target domain sample, i∈1, 2, 3…n, n is the number of target domain samples, and the target domain sample dataset T is self-sampled with replacement to generate N T Sub-dataset T i , N T is a hyperparameter, each sub-dataset T i The number of samples is the same as the number of samples in the target domain sample data set, that is, {T i |(x i 1, y i 1), (x i 2, y i 2)....(x i n y i n )}.

[0080] Step 1.2: Select a deep learning or machine learning model structure to implement the classification task and initialize the model parameters. T Sub-dataset T i Parallel training, get N T Different trained base learners i .

[0081] Among them, the division of the population can be based on gender, age group, race and other characteristics to divide similar people into the same population.

[0082] Among them, due to the limited number of samples, using an overly complex base model will lead to overfitting, so a simple random forest classifier was selected as the base learner.

[0083] It should be understood that the use of a weighted base classifier to screen source domain samples will reduce the complexity when the amount of data is large compared to the traditional method based on pairwise comparison.

[0084] In one embodiment, evaluating the trained base learner, calculating the weight of the trained base learner according to the evaluation result to screen source domain samples, and adding the source domain samples with weights greater than a threshold to a candidate source domain set includes:

[0085] Step 2.1: For each trained base learner i, use the sample data set T of the target domain for evaluation, and obtain the error rate e of the corresponding trained base learner on the target domain i , according to the error rate e i Calculate the trained base learner Learner i The screening weight w i , screening weight w i The expression is:

[0086]

[0087] Step 2.2: Collect acceleration data of other people as source domain samples to form a source domain sample dataset S, {S|(x1, y1), (x2, y2)....(x m ,y m )}, where x j Represents the features of the j-th source domain sample, y j represents the true label of the jth source domain sample, j∈1, 2, 3…m, m is the number of source domain samples, and each time a single source domain sample (x j ,y j ), the feature x of the source domain sample j Input to each trained base learner i In the example above, we get the prediction result of the source domain sample. Combine the prediction results of the source domain sample in all base learners and their corresponding weights w i , calculate the voting rate of the source domain sample vote_rate(x j ), the voting rate of the source domain sample vote_rate(x j ) is:

[0088]

[0089] Among them, I(·) is a conditional judgment function. The conditional judgment involved in the above calculation of voting rate is as follows: when the predicted value of the sample label is equal to the true value of the label, the result of the function is 1, otherwise the result is 0. The conditional judgment of the conditional judgment function is:

[0090]

[0091] Step 2.3: According to the set screening threshold σ, if the voting rate of the current source domain sample vote_rate(x j ) is greater than the threshold σ, the source domain sample is added to the candidate source domain set S c Otherwise, it is considered that the source domain sample has a negative effect on the target domain training and is not included in the candidate source domain set S cFinally, the selected candidate source domain set S c contains s samples, that is, {S c |(x1,y1),(x2,y2)....(x s, y s )}.

[0092] In an example, suppose the sample feature of the source domain is 38 dimensions, the true label is 3, N T is 20, 15 base learners predict the sample as category 3, and the weights are all 0.05, then vote_rate is calculated as 0.75 according to the following formula. The sample vote_rate is 0.75. Assuming the screening threshold σ is 0.6, the sample is added to the candidate source domain set.

[0093] In one embodiment, the method of using the candidate source domain set to pre-train a teacher network in a human activity recognition model, learning domain-invariant features and freezing parameters of the teacher network includes:

[0094] Step 3.1: Use the candidate source domain set S c The teacher network TeaNet of the human activity recognition model is pre-trained, where for the candidate source domain set S c The characteristics of the source domain samples x j , j∈1, 2, 3...s, first enter the feature extraction layer of the teacher network TeaNet Then we get the feature layer embedding Feature layer embedding Then the classification layer The final predicted value is obtained

[0095]

[0096] Step 3.2: Based on the final predicted value and true label y j The classification loss is calculated and the parameters of the teacher network are updated. The optimization goal of updating the parameters of the teacher network is:

[0097]

[0098] in, and Corresponding to the teacher network feature extraction layer The classification layer The parameters to be optimized; E ( x j, y j)～Sc [.] represents the candidate source domain set S c The mathematical expectation of the sampled loss values, which is based on the variables Calculated, indicating the predicted value The loss function value between the actual value y of the label;

[0099] Step 3.3: Freeze the parameters of the teacher network to obtain the pre-trained teacher network TeaNet.

[0100] Among them, the candidate source domain set S is used c The teacher network TeaNet of the human activity recognition model is pre-trained with the main purpose of learning domain-invariant features, which enables the student network StuNet to distill knowledge more efficiently.

[0101] Among them, the optimization goal of updating the parameters of the teacher network is to minimize the empirical risk of the teacher network model in the screened source domain distribution.

[0102] Among them, the variable It can be any loss function suitable for the classification task, with variables Taking the cross entropy loss as an example, its calculation method is as follows:

[0103]

[0104] Among them, s is the candidate source domain set S c The number of samples in y j is the true label of the jth sample, x j is the feature of the jth sample, For the x j Predicted values ​​after passing through the teacher network.

[0105] Among them, since the teacher network TeaNet only performs preliminary pre-training on the source domain and subsequently calculates the distillation loss, in the subsequent update of the student network, although the data will enter the teacher network, the parameters of the teacher network TeaNet during back propagation and It will not be updated.

[0106] In one embodiment, the target domain sample data set is merged with the candidate source domain set, the features of the samples in the merged data set are input into the student network and the teacher network in the human activity recognition model, the features of the samples are reweighted using the attention mechanism, and finally the classification loss and the distillation loss are calculated respectively, and finally the parameters of the student network are updated by back propagation to obtain a trained human activity recognition model based on the attention mechanism distillation learning network, including:

[0107] Step 4.1: Merge the target domain sample dataset with the candidate source domain set to obtain a merged dataset CmbSet. The number of samples in the merged dataset is N=n+s.

[0108] Step 4.2: Initialize the parameters of the student network StuNet in the human activity recognition model. The parameter structure of the student network is mainly divided into three parts: the feature extraction layer parameters of the student network Attention layer parameters and classification layer parameters

[0109] Step 4.3: For the sample (x i ,y i ), i ∈1, 2, 3...N, first input feature x i to the student network StuNet, through the feature extraction layer of the student network Get preliminary embedding features

[0110] Step 4.4: Calculate preliminary embedding features using additive attention The attention weight α qk , after the attention layer of the student network After the action of

[0111] Step 4.5: Reweight the features Input to the classification layer of the student network Get the predicted value of the student network and calculate the classification loss L of the student network cls , the classification loss L cls The expression is:

[0112]

[0113] Among them, N is the total number of samples in the merged data set CmbSet, y i is the true label of the i-th sample, x i is the feature of the i-th sample, For the x i Predicted values ​​after passing through the student network.

[0114] Step 4.6, the feature x i At the same time, it is input into the pre-trained teacher network TeaNet, and after feature extraction, it is obtained The student network will try to imitate the feature extraction ability of the teacher network to strengthen itself, so it needs to calculate and Distillation loss L dst , the calculation formula is:

[0115]

[0116] in, is feature x i The output value of the feature extraction layer of the teacher network, is feature x i Output value of the feature extraction layer of the student network.

[0117] Among them, any similarity measurement function is applicable here, and the MSE error function is selected in the experiment to measure the distillation loss between two features.

[0118] Step 4.7. Finally, calculate the total loss of the student network and back-propagate the updated parameters. The optimization goal of back-propagation updated parameters is:

[0119]

[0120] Among them, E ( x i, y i)～CmbSet [.] represents the mathematical expectation of the total loss calculated by sampling from the merged dataset CmbSet. The total loss is composed of the student network classification loss L cls and distillation loss L dst It consists of two parts, where λ is the distillation coefficient of the teacher network, which ranges from 0 to 1.

[0121] Step 4.8: Repeat steps 4.2 to 4.7 until the preset maximum number of training batches is exceeded. Save the model and evaluate it in the test set to obtain a trained human activity recognition model based on the attention mechanism-based distillation learning network.

[0122] It should be understood that the use of source domain dataset pre-training improves the model's ability to find invariant features in the domain, and then the mixed dataset is used to train the main structure of the network, the generalization of the model is improved based on classification loss and distillation loss, and the attention mechanism is used to further improve the model's ability to extract high-dimensional features.

[0123] In one embodiment, step 4.4 includes:

[0124] First, according to the embedded features Initialize the Query (Q), Key (K), and Value (V) vectors respectively, which represent the input vectors required for feature autocorrelation calculation. Among them, the Q moment vector represents the important features to be concerned about; the K vector represents the associated features in the feature space; the V vector represents the feature value associated with the K vector. Initially, the values ​​of Q, K, and V are equal to Right now:

[0125] Embedding features Perform linear transformation to obtain the transformed Query (Q′) and Key (K′) vectors, respectively, expressed as:

[0126]

[0127]

[0128] Among them, W q and W k is a learnable linear transformation matrix used to map input features to a new feature space, b q and b k is the bias term;

[0129] The linearly transformed Q′ and K′ are added together, and the similarity score e between the qth component in the Q vector and the kth component in the K vector is calculated through a learnable weight vector v qk , which measures the extent to which the feature at position q in the Q vector focuses on the feature at position k in the K vector, and the similarity score e qk The expression is:

[0130] e qk =v T tanh(Q′+K′)

[0131] Among them, q and k represent the position index of Q vector and K vector, v T It is the vector after the weight vector v is transposed. Tanh is the hyperbolic tangent activation function, which maps the input to the range of (-1, 1). The calculation formula of the hyperbolic tangent activation function is:

[0132]

[0133] Among them, x represents the variable of the hyperbolic tangent activation function, that is, x refers to Q′+K′;

[0134] The similarity score e qk Normalize to get the attention weight α between different components in the feature qk , which represents the importance of the qth component to the kth component in the embedded feature, t represents the feature vector dimension, and the expression is:

[0135]

[0136] Use attention weight α qk Embedded features Perform weighted summation to obtain the re-weighted features Reweighted Features The calculation formula is:

[0137]

[0138] Among them, t is the number of components of the embedded feature, α qk is the calculated attention weight, V k is the component at position k in the V vector, that is, the actual characteristic component at position k (when initialized ).

[0139] Among them, the main network structure of the candidate source domain set pre-trained human activity recognition model is as follows Figure 3 As shown in the figure, there are four data sets as an example. Each data set can be regarded as a separate data domain, and pairs are paired. For example, the samples with H as the source domain and D as the target domain are visually distributed after PCA dimensionality reduction. Figure 4 As shown in the figure, it consists of a student network StuNet and a teacher network TeaNet. Taking H>D as an example, the amount of original data in the source domain H is 6616. After screening using the target domain D, the final candidate set Sc only contains 2280 data. Therefore, the merged CmbSet has a total of 4680 data.

[0140] The human activity recognition method based on instance transfer and attention distillation network generates multiple sub-datasets by sampling with replacement on the target domain samples in the target domain sample data set, and trains a base learner on each sub-dataset to obtain a trained base learner; the trained base learner is evaluated, and the weight of the trained base learner is calculated according to the evaluation result to screen the source domain samples, and the source domain samples with weights greater than a threshold are added to the candidate source domain set; the candidate source domain set is used to pre-train the teacher network in the human activity recognition model to learn the invariant features in the domain and freeze the parameters of the teacher network; the target domain sample data set is merged with the candidate source domain set, and the features of the samples in the merged data set are input into the student network in the human activity recognition model The network and the teacher network use the attention mechanism to reweight the features of the samples, and finally calculate the classification loss and distillation loss respectively. Finally, the parameters of the student network are updated through back propagation to obtain a trained human activity recognition model based on the distillation learning network of the attention mechanism, and then screen out samples with higher relevance to the target domain to optimize the knowledge transfer of source domain and target domain data, improve the performance of the model in the target domain and reduce the negative transfer phenomenon, so that the trained model has higher accuracy and stronger generalization, and then obtain the user's acceleration data; input the acceleration data into the trained human activity recognition model based on the distillation learning network of the attention mechanism for activity recognition, and output the user's activity type; thereby improving the accuracy of human activity recognition.

[0141] In order to verify the effect of this application, multiple human activity datasets such as USC-HAD, UCI-HAR, DSADS, and PAMAP were collected. This application preprocessed the data and selected the sensor parts common to all datasets and four common activity categories (lying, walking, rising, and falling). At the same time, a total of 38 features in the time domain and frequency domain were extracted for a single sensor using the sliding window technology.

[0142] 1. Data preparation

[0143] The data provided by the four datasets USC-HAD, UCI-HAR, DSADS, and PAMAP are used as experimental objects. The preprocessed dataset information is shown in Table 1.

[0144] Table 1 Dataset information

[0145]

[0146] Since the experimental conditions and data distribution are different, this application regards each data set as a different domain, and the subsequent experiments will be based on 4 domains. At the same time, the data of the target domain is divided into training set and test set in a ratio of 4:1. 80% of the target domain training set and the screened candidate source domain training set S c The merged models are then rebuilt. The final trained model will be used to verify the classification effect in the test set. The main body of the experiment is divided into two parts: classification performance comparison experiment and ablation experiment, with classification accuracy as the evaluation indicator.

[0147] 2. Comparative Experiment

[0148] 1) Classification performance comparison experiment

[0149] The method of this application and the classic TrAdaboost method in instance transfer and the MAML method in small sample learning were used to conduct paired domain transfer experiments on 4 data sets. The method of this application and the comparison method used the same parameters to compare the classification accuracy of the trained model on the test set. Table 2 shows the parameter settings of this experiment.

[0150] Table 2 Hyperparameter settings

[0151]

[0152]

[0153] In addition, this application uses a 3-layer MLP as the basic network architecture (backbone), and its structure and dimensions are shown in Table 3, where each layer uses ReLu as the activation function and adds a dropout layer with a probability of 0.5.

[0154] Table 3 Backbone structure settings

[0155]

[0156] Comparison results:

[0157] Table 4 Comparison results of classification accuracy (%)

[0158]

[0159] According to the comparison results in Table 4, it can be seen that the method of the present application has high accuracy and can have strong generalization in most scenarios in recognition tasks when data is scarce.

[0160] 2) Ablation experiment

[0161] In order to test the effectiveness of source domain sample screening and attention mechanism, experiments were conducted with the following five settings. The settings that did not use direct merge annotation all used the same data screening steps as the method of this application (backbone was also merged after screening), respectively verifying the effectiveness of the data screening part and the attention distillation network part. The final results are shown in Table 5.

[0162] Ablation results:

[0163] Table 5 Ablation test accuracy of this method (%)

[0164]

[0165] Combined with Figure 5 It can be seen from the ablation experiment results in Table 5 that both the source domain data screening method and the attention distillation network involved in this application have a certain positive effect on the task, and the method of this application has the highest accuracy in most of the five groups of settings.

[0166] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A human activity recognition method based on instance transfer and attention distillation network, characterized in that: The method comprises: Get the user's acceleration data; Input the acceleration data into a trained human activity recognition model based on an attention mechanism distillation learning network for activity recognition, and output the user's activity type; The training method of the human activity recognition model is: Performing sampling with replacement on the target domain samples in the target domain sample data set to generate multiple sub-data sets, and training a base learner on each sub-data set to obtain a trained base learner; Evaluate the trained base learner, calculate the weight of the trained base learner according to the evaluation result to screen source domain samples, and add the source domain samples whose weight is greater than a threshold into the candidate source domain set; Pre-training a teacher network in a human activity recognition model using a candidate source domain set to learn domain-invariant features and freeze parameters of the teacher network; The target domain sample dataset is merged with the candidate source domain set, and the features of the samples in the merged dataset are input into the student network and teacher network in the human activity recognition model. The features of the samples are reweighted using the attention mechanism, and the classification loss and distillation loss are calculated respectively. Finally, the parameters of the student network are updated through back propagation to obtain a trained human activity recognition model based on the distillation learning network with attention mechanism.

2. The method according to claim 1, characterized in that: The step of performing sampling with replacement on the target domain samples in the target domain sample data set to generate multiple sub-data sets, and training a base learner on each sub-data set to obtain a trained base learner includes: Step 1.1, collect the acceleration data of the target population as the target domain samples, and construct the target domain sample dataset T, {T|(x1,y1),(x2,y2)....(x n, y n )}, where x i Represents the characteristics of the i-th target domain sample, y i represents the true label of the i-th target domain sample, i∈1, 2, 3…n, n is the number of target domain samples, and the target domain sample dataset T is self-sampled with replacement to generate N T Sub-dataset T i , N T is a hyperparameter, each sub-dataset T i The number of samples is the same as the number of samples in the target domain sample data set, that is, {T i |(x i 1, y i 1), (x i 2, y i 2)....(x i n ,y i n )}; Step 1.2: Select a deep learning or machine learning model structure to implement the classification task and initialize the model parameters. T Sub-dataset T i Parallel training, get N T Different trained base learners i .

3. The method according to claim 2, characterized in that Evaluating the trained base learner, calculating the weight of the trained base learner according to the evaluation result to screen source domain samples, and adding the source domain samples with the weight greater than a threshold to a candidate source domain set, including: Step 2.1: For each trained base learner i , use the sample data set T of the target domain for evaluation, and get the error rate e of the corresponding trained base learner on the target domain i , according to the error rate e i Calculate the trained base learner Learner i The screening weight w i , screening weight w i The expression is: Step 2.2: Collect acceleration data of other people as source domain samples to form a source domain sample dataset S, {S|(x1, y1), (x2, y2)....(x m ,y m )}, where x j Represents the features of the j-th source domain sample, y j represents the true label of the jth source domain sample, j∈1, 2, 3…m, m is the number of source domain samples, and each time a single source domain sample (x j ,y j ), the feature x of the source domain sample j Input to each trained base learner i In the example above, we get the prediction result of the source domain sample. Combine the prediction results of the source domain sample in all base learners and their corresponding weights w i , calculate the voting rate of the source domain sample vote_rate(x j ), the voting rate of the source domain sample vote_rate(x j ) is: Wherein, I(·) is a conditional judgment function, and the conditional judgment of the conditional judgment function is: Step 2.3: According to the set screening threshold σ, if the voting rate of the current source domain sample vote_rate(x j ) is greater than the threshold σ, the source domain sample is added to the candidate source domain set S c Otherwise, it is considered that the source domain sample has a negative effect on the target domain training and is not included in the candidate source domain set S c Finally, the selected candidate source domain set S c contains s samples, that is, {S c |(x1,y1),(x2,y2)....(x s y s )}.

4. The method according to claim 3, characterized in that: The method of using the candidate source domain set to pre-train a teacher network in a human activity recognition model, learning invariant features within the domain and freezing parameters of the teacher network includes: Step 3.1: Use the candidate source domain set S c The teacher network TeaNet of the human activity recognition model is pre-trained, where for the candidate source domain set S c The characteristics of the source domain samples x j , j∈1, 2, 3...s, first enter the feature extraction layer of the teacher network TeaNet Then we get the feature layer embedding Feature layer embedding Then the classification layer The final predicted value is obtained Step 3.2: Based on the final predicted value and true label y j The classification loss is calculated and the parameters of the teacher network are updated. The optimization goal of updating the parameters of the teacher network is: in, and Corresponding to the teacher network feature extraction layer The classification layer The parameters to be optimized; Represents the candidate source domain set S c The mathematical expectation of the sampled loss values, which is based on the variables Calculated, indicating the predicted value The loss function value between the actual value y of the label; Step 3.3: Freeze the parameters of the teacher network to obtain the pre-trained teacher network TeaNet.

5. The method according to claim 1, characterized in that The target domain sample data set is merged with the candidate source domain set, the features of the samples in the merged data set are input into the student network and the teacher network in the human activity recognition model, the features of the samples are reweighted using the attention mechanism, and finally the classification loss and the distillation loss are calculated respectively, and finally the parameters of the student network are updated through back propagation to obtain the trained human activity recognition model based on the attention mechanism distillation learning network, including: Step 4.1, merge the target domain sample dataset with the candidate source domain set to obtain a merged dataset CmbSet, where the number of samples in the merged dataset is N=n+s; Step 4.2: Initialize the parameters of the student network StuNet in the human activity recognition model. The parameter structure of the student network is mainly divided into three parts: the feature extraction layer parameters of the student network Attention layer parameters and classification layer parameters Step 4.3: For the sample (x i ,y i ), i∈1, 2, 3...N, first input feature x i to the student network StuNet, through the feature extraction layer of the student network Get preliminary embedding features Step 4.4: Calculate preliminary embedding features using additive attention The attention weight α qk , after the attention layer of the student network After the action of Step 4.5: Reweight the features Input to the classification layer of the student network Get the predicted value of the student network and calculate the classification loss L of the student network cls , the classification loss L cls The expression is: Among them, N is the total number of samples in the merged data set CmbSet, y i is the true label of the i-th sample, x i is the feature of the i-th sample, For the x i Prediction value after passing through the student network; Step 4.6, the feature x i At the same time, it is input into the pre-trained teacher network TeaNet, and after feature extraction, it is obtained calculate and Distillation loss L dst , the calculation formula is: in, is feature x i The output value of the feature extraction layer of the teacher network, is feature x i Output value of the feature extraction layer of the student network; Step 4.

7. Finally, calculate the total loss of the student network and back-propagate the updated parameters. The optimization goal of back-propagation updated parameters is: in, Represents the mathematical expectation of the total loss calculated by sampling from the merged dataset CmbSet. The total loss is composed of the student network classification loss L cls and distillation loss L dst It consists of two parts, where λ is the distillation coefficient of the teacher network, ranging from 0 to 1; Step 4.8: Repeat steps 4.2 to 4.7 until the preset maximum number of training batches is exceeded. Save the model and evaluate it in the test set to obtain a trained human activity recognition model based on the attention mechanism-based distillation learning network.

6. The method according to claim 5, characterized in that The step 4.4 comprises: First, according to the embedded features Initialize the Query (Q), Key (K), and Value (V) vectors respectively, where the Q moment vector represents the important features to be concerned about; the K vector represents the associated features in the feature space; the V vector represents the feature value associated with the K vector. Initially, the values ​​of Q, K, and V are equal to Right now: Embedded features Perform linear transformation to obtain the transformed Query (Q′) and Key (K′) vectors, respectively, expressed as: Among them, W q and W k is a learnable linear transformation matrix used to map input features to a new feature space, b q and b k is the bias term; The linearly transformed Q′ and K′ are added together, and the similarity score e between the qth component in the Q vector and the kth component in the K vector is calculated through a learnable weight vector v qk , which measures the extent to which the feature at position q in the Q vector focuses on the feature at position k in the K vector, and the similarity score e qk The expression is: e qk =υ T tanh(Q′+K′) Among them, q and k represent the position index of Q vector and K vector, v T It is the vector after the weight vector v is transposed. Tanh is the hyperbolic tangent activation function, which maps the input to the range of (-1, 1). The calculation formula of the hyperbolic tangent activation function is: Among them, x represents the variable of the hyperbolic tangent activation function, that is, x refers to Q′+K′; The similarity score e qk Normalize to get the attention weight α between different components in the feature qk , which represents the importance of the qth component to the kth component in the embedded feature, t represents the feature vector dimension, and the expression is: Use attention weight α qk Embedded features Perform weighted summation to obtain the re-weighted features Reweighted Features The calculation formula is: Among them, t is the number of components of the embedded feature, α qk is the calculated attention weight, V k is the component of the V vector at position k.

Images