A user behavior recognition method based on multi-task multi-view incremental learning

Abstract

The present invention discloses a user behavior recognition method based on multi-task multi-view incremental learning, comprising the following steps: S1. collecting activity posture data, dividing the data into multiple parts, and determining the number of tasks and the number of views of each task according to the personnel participating in the training, and initializing the parameters of each stage; S2. extracting features for sensor data using a multi-task multi-view deep neural network; S3. updating parameters by calculating multiple loss functions and updating parameters through back propagation; S4. completing the training of all stages and obtaining the final prediction results. The advantage is that MTMVIS uses each layer of the multi-task multi-view deep neural network to extract features respectively, and uses the attention layer to weight the outputs of all layers of the multi-task multi-view deep neural network for each task as the final output layer. MTMVIS uses adaptive weight consolidation to mitigate the catastrophic forgetting problem and enhance model scalability.

Description

Technical Field

[0001] The present invention belongs to the technical field of user behavior prediction of smart devices, and specifically relates to a user behavior recognition method based on multi-task multi-view incremental learning. Background Art

[0002] In recent years, user behavior recognition through intelligent sensing has attracted more and more researchers' interest in academic and industrial fields. Various sensors (such as accelerometers, gyroscopes, etc.) embedded in powerful smartphones or smart wearable devices are used to identify user behavior and are widely used in medical services, commerce, security and other fields. User behavior recognition aims to detect user behavior in the real world, which can enable intelligent systems to help individuals improve their quality of life in areas such as healthcare and smart cities. In previous studies, user behavior recognition on specific datasets has achieved relatively good results.

[0003] However, existing methods for user behavior recognition have the following drawbacks: (1) Traditional methods collect data from various sensors at all time intervals and build a general offline model to recognize activities, which consumes a lot of space to store a large amount of training data. (2) Catastrophic forgetting is common in incremental learning. Since the measurements generated by sensors are very fast, when the model is trained on new activities in the changing phase, it may forget the previously learned knowledge. (3) Different people and different types of sensors usually present different forms for the same activity, namely task heterogeneity and view heterogeneity. Traditional incremental learning methods often ignore these problems. Summary of the invention

[0004] Based on the above problems, this application proposes a user behavior recognition method MTMVIS based on multi-task multi-view incremental learning to solve the above three major problems in user behavior recognition. MTMVIS first uses the online method of incremental learning to save time and space; by formalizing different groups of people into different tasks, and formalizing sensors located in different parts of the body into different views, the idea of ​​multi-task multi-view learning is used to solve task heterogeneity and view heterogeneity; plastic weight consolidation (EWC) can alleviate the problem of catastrophic forgetting in incremental learning. Its technical solution is,

[0005] A user behavior recognition method based on multi-task multi-view incremental learning includes the following steps:

[0006] S1. Collect activity posture data, divide the data into multiple parts, initialize the number of activity stages, determine the number of tasks and the number of views for each task according to the personnel participating in the training, and initialize the parameters of each stage;

[0007] S2. Extract features from sensor data using multi-task multi-view deep neural networks;

[0008] S3. Parameter update, by calculating multiple loss functions and updating the parameters through back propagation;

[0009] S4. Perform multiple rounds of iterations from S2 to S4 until the required number of iterations is reached, and then proceed to the next stage of training until all stage training is completed and the prediction results of the final stage are obtained.

[0010] Further preferably, in step S1, the type of activity posture is ζ, the number of times each posture appears is Γ, and the order of appearance is random, and the number of activity stages is M=ζ*Γ; a single activity posture of the user appears in each activity stage and the activity posture appearing in each stage is random and repeatable, and the number of tasks and the number of views of each task are determined according to the number of people participating in the activity and the body parts wearing sensors, and the parameters of each stage specifically include the model parameter θ m , the number of iterations R m , learning rate ρ m ,m∈{1,2,…,M} represents the mth activity stage.

[0011] Further preferably, in step S2, the data collected by the sensor is output to the data alignment layer for data alignment, the steps are as follows:

[0012] In each time interval of T seconds, there are S sensors in each part of the body, each sensor can generate N data, and the initial data vector of the sensor is set Where n∈{1,2,…,N}, s∈{1,2,…,S}, k represents the kth task, v represents the vth view, where k∈{1,2,…K}, v∈{1,2,…,V}, represents the nth piece of data generated by the sth sensor on the vth part of the kth user within T seconds; the data alignment layer includes the first attention layer and the second attention layer. The first attention layer aligns the data of different sensors located on the same part of the body, and performs weighted average according to the importance of different sensors located on the same part of the body to obtain a feature vector of fixed length The second attention layer weights different measurements generated in the same time interval to align different data generated in the same time interval and obtain a fixed-length feature vector

[0013] Further preferably, in step S2, the feature vector output by the data alignment layer is Input the fully connected layer to get a fixed-length feature vector

[0014] Further preferably, in step S2, the feature vector Input the task view shared layer and execute the MMoE algorithm. The specific process is as follows:

[0015] The MMoE network consists of E expert networks and a K*V gating network. First, the feature vector As the input of each expert network, the output of each expert network is obtained Where e represents the e-th expert network, and the gating network combines the outputs of multiple experts to obtain a specific feature vector for each task view.

[0016] Further preferably, in step S2, the feature representation Input view fusion layer, fuse the view features of each task through the third attention layer to obtain the feature vector of each task

[0017] Further preferably, in step S3, the feature vectors output by the data alignment layer are respectively The feature vector output by the fully connected layer The feature vector output by the task view shared layer After different attention layers, the feature vector is obtained Eigenvector and the eigenvector The feature vector Eigenvector The feature vector and the eigenvector Input adaptive output layer for weighting, each task gets a specific feature vector F k .

[0018] Further preferably, in step S4, the multi-view loss L is calculated by the regularization term in the view fusion stage. MV ; Prediction result F k and the true value y k Calculate the cross entropy loss L CL ; and calculate the multi-task loss L through homoscedastic uncertainty MT In addition, the EWC algorithm parameters are consolidated to overcome the catastrophic forgetting problem and obtain incremental loss Final loss function

[0019]

[0020] Using L F Calculate the gradient and update the model, that is

[0021] where r m ∈(1,2,...Rm ).

[0022] Beneficial Effects

[0023] The present invention uses a user behavior recognition framework based on multi-task multi-view incremental learning to solve the user behavior recognition problem in real scenarios. Existing methods usually train data in a centralized batch processing manner, which requires a very large storage space to save all user behavior data. If users are trained individually, the limited amount of data is an extremely serious problem. By introducing a multi-task framework, multiple users can learn from each other and thus learn a generalized model. In addition, since different body parts of different people have heterogeneity for the same activity, this multi-view problem is also the focus of our research. The present invention utilizes a multi-task multi-view learning method under the framework of incremental learning and proposes a more advanced user behavior prediction method.

[0024] MTMVIS (MT multi-task, MV multi-view, I incremental, Sstreaming data) solves the three major problems of limited storage space, heterogeneity between users, and heterogeneity of views in user behavior recognition. MTMVIS first uses the online method of incremental learning to save time and space; by formalizing different people into different tasks and sensors located in different parts of the body into different views, it uses the idea of ​​multi-task multi-view learning to solve task heterogeneity and view heterogeneity; adaptive plastic weight consolidation (EWC) can alleviate the problem of catastrophic forgetting in incremental learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the user location prediction process of the framework of the present invention.

[0026] Figure 2 Schematic diagram of the framework proposed by the present invention.

[0027] Figure 3 Schematic diagram of the multi-task multi-view deep neural network of the framework of the present invention.

[0028] Figure 4 Schematic diagram of simulation results of the framework of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the attached Figure 1-4 Describe this application in detail.

[0030] A user behavior recognition method based on multi-task multi-view incremental learning, the specific steps are as follows:

[0031] S1: Initialization. This step specifically includes:

[0032] S11. Collect activity posture data and divide the data into multiple parts, that is, initialize the number of activity stages. Since the activities are performed sequentially, the specific activities include standing, lying, walking, running, going upstairs, going downstairs, and jumping, with a total of seven activity postures. Each activity posture appears three times, and the order of appearance is random, so there are a total of M = 21 activity stages. The task for each activity stage is to predict these seven specific activity postures (experimental setting: during the training process, for the first few activity stages, since there are no labels for all activity postures, the default prediction results for activities that have not been trained are 100%. For example, in the first stage, the input of the activity is standing data, and the standing data is used for training and prediction. The test set contains seven activity postures, and the accuracy of the other six activity postures is 100%; in the second stage, the input is running data. We use running data for training and prediction, and the accuracy of the other five activity postures is 100%.); Use m∈{1,2,…,M} to represent the mth activity stage, and each activity stage contains t m There are time intervals T, and each time interval is set to 2.5 seconds.

[0033] S12. Determine the number of tasks and the number of views for each task. For the sake of generality, 4 males and 4 females were selected for the experiment. The acceleration of the chest, forearm, head, calf, thigh, upper arm and waist was recorded simultaneously for each activity. Since 7 different body parts were considered as 7 views, a total of 8 tasks with 7 views were defined.

[0034] S13. Initialize the parameters of each stage. The parameters of each stage specifically include the model parameters θ m , the number of iterations R m , learning rate ρ m .

[0035] S2: For each stage, Figure 3 As shown, a multi-task multi-view deep neural network is used to extract features for sensor data. In this step, it specifically includes:

[0036] S21. Data alignment: Output the data collected by the sensor to the data alignment layer for data alignment. First, in each time interval of T seconds, there are S sensors in each part of the body, and each sensor can generate N data. Set the initial data vector of the sensor Where n∈{1,2,…,N}, s∈{1,2,…,S}, k represents the kth task, v represents the vth view, where k∈{1,2,…K}, v∈{1,2,…,V}, It represents the nth piece of data generated by the sth sensor at the vth part of the kth user within T seconds.

[0037] The data alignment layer includes the first attention layer and the second attention layer. In the first attention layer, the data of different sensors located in the same part of the body are aligned, and the weighted average is performed according to the importance of different sensors located in the same part of the body to obtain a feature vector of fixed length. In the second attention layer, we weight different measurements generated in the same time interval to achieve alignment of different data generated in the same time interval and obtain a fixed-length feature vector

[0038] S22. In order to improve the learning ability of the model, a fully connected neural network with two hidden layers is used, using the feature vector obtained in step S21 As input, we get a fixed-length feature vector

[0039] S23. To model the relationship between different task views, in the task view sharing layer, use the feature vector obtained in step S22 As input, the MMoE algorithm is executed. The specific process is as follows: the MMoE network contains E expert networks and K*V gating networks, that is, each task view has its own unique gating network), where the gating network of each task view realizes the selective use of experts through different final output weights. First, use the feature vector obtained in step S22 As the input of each expert network, the output of each expert network is obtained Where e represents the e-th expert network. By combining the outputs of multiple experts, the gating network can obtain a customized feature vector for each task view.

[0040] S24. Obtain the feature representation of each task view. In order to obtain the unique feature representation of each task, it is necessary to perform view fusion on all views of each task. Specifically, at the view fusion layer, use the feature vector obtained in step S23 As input, the third attention layer is used to fuse the view features of each task to obtain the feature vector of each task.

[0041] S3: Each task obtains the prediction result of this stage by weighting the output of each layer of the multi-task multi-view deep neural network. In this step, Figure 2 As shown, specifically including:

[0042] S31. Align the feature vector output by the data layer The feature vector output by the fully connected layer The feature vector output by the task view shared layer The feature vectors are obtained by passing through different attention layers (fifth attention layer, sixth attention layer, seventh attention layer) respectively. Eigenvector and the eigenvector

[0043] S32. For each task, the feature vector Eigenvector Eigenvector and the eigenvector Input the adaptive output layer (the fourth attention layer) for weighting, and each task gets a specific feature vector F k .

[0044] S4: Parameter update. This step specifically includes:

[0045] S41. Calculate the multi-view loss L through the regularization term in the view fusion stage MV ; Prediction result F k and the true value y k Calculate the cross entropy loss L CL ; and calculate the multi-task loss L through homoscedastic uncertainty MT ,Right now where σ k is the standard deviation of the Gaussian distribution of the noise of the kth task, λ MT is a hyperparameter; in addition, the incremental loss is obtained by consolidating the EWC algorithm parameters to overcome the catastrophic forgetting problem, that is, in is the p-th diagonal value of the diagonal Fisher matrix calculated after the m-th stage of training for the k-th task, Represents the parameters of the m+1th stage, and the total loss is obtained

[0046] S42. Using L F Calculate the gradient and update the model, that is where r m ∈(1,2,...R m ).

[0047] Perform multiple rounds of iterations on S2-S4 until the required number of iterations is reached, and then proceed to the next stage of training until all stage training is completed and the prediction results of the final stage are obtained.

[0048] pass Figure 4It is proved that our technology has achieved excellent results under the four measurement indicators. As the stage progresses, all indicators decrease due to catastrophic forgetting, but our mechanism has the ability to overcome catastrophic forgetting, so the decline rate is relatively slow. In addition, since we have conducted each activity three times, multiple training can consolidate the parameters of related activities and improve the model effect. AUC refers to the area under the ROC curve (ROC refers to the receiver operating characteristic curve).

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A user behavior recognition method based on multi-task multi-view incremental learning, characterized in that: The following steps are included: S1. Collect activity posture data, divide the data into multiple parts, initialize the number of activity stages, determine the number of tasks and the number of views for each task according to the personnel participating in the training, and initialize the parameters of each stage; S2. Use a multi-task multi-view deep neural network to extract features for the data collected by the sensor. The steps are as follows: Assume that in each time interval of T seconds, there are S sensors in each part of the body. The data collected by the sensors are input into the data alignment layer of the deep neural network for data alignment to obtain a feature vector of fixed length The feature vector output by the data alignment layer Input the fully connected layer to get a fixed-length feature vector The feature vector Input the task view shared layer and execute the MMoE algorithm. Each task view can obtain a specific feature vector Represent the features Input view fusion layer, fuse the view features of each task through the third attention layer to obtain the feature vector of each task Align the feature vectors output by the data layer The feature vector output by the fully connected layer The feature vector output by the task view shared layer After different attention layers, the feature vector is obtained Eigenvector and the eigenvector The feature vector Eigenvector The feature vector and the eigenvector Input adaptive output layer for weighting, each task gets a specific feature vector F k ; S3. Parameter update, by calculating multiple loss functions and updating the parameters through back propagation; S4. Perform multiple rounds of iterations from S2 to S4 until the required number of iterations is reached, and then proceed to the next stage of training until all stage training is completed and the prediction results of the final stage are obtained.

2. According to claim 1, a user behavior recognition method based on multi-task multi-view incremental learning is characterized in that: In step S1, the type of activity posture is ζ, the number of times each posture appears is Γ, and the order of appearance is random, and the number of activity stages is M=ζ*Γ; a single activity posture of the user appears in each activity stage, and the activity postures appearing in each stage are random and repeatable, and the number of tasks and the number of views of each task are determined according to the number of people participating in the activity and the body parts wearing sensors, and the parameters of each stage specifically include the model parameter θ m , the number of iterations R m , learning rate ρ m ,m∈{1,2,...,M} represents the mth activity stage.

3. The user behavior recognition method based on multi-task multi-view incremental learning according to claim 1 is characterized in that: In step S2, the data collected by the sensor is input into the data alignment layer for data alignment. The steps are as follows: In each time interval of T seconds, there are S sensors in each part of the body, each sensor can generate N data, and the initial data vector of the sensor is set Where n∈{1,2,…,N}, s∈{1,2,…,S}, k represents the kth task, v represents the vth view, where k∈{1,2,…K}, v∈{1,2,…,V}, represents the nth piece of data generated by the sth sensor at the vth part of the kth user within T seconds; The data alignment layer includes the first attention layer and the second attention layer. The first attention layer aligns the data of different sensors located in the same part of the body, performs weighted average according to the importance of different sensors located in the same part of the body, and obtains a feature vector of fixed length. The second attention layer weights different measurements generated in the same time interval to achieve alignment of different data generated in the same time interval and obtain a fixed-length feature vector 4. The user behavior recognition method based on multi-task multi-view incremental learning according to claim 1 is characterized in that: In step S2, the feature vector Input the task view shared layer and execute the MMoE algorithm. The specific process is as follows: The MMoE network consists of E expert networks and a K*V gating network. First, the feature vector As the input of each expert network, the output of each expert network is obtained Where e represents the e-th expert network, and the gating network combines the outputs of multiple experts to obtain a specific feature vector for each task view.

5. The user behavior recognition method based on multi-task multi-view incremental learning according to claim 1 is characterized in that: In step S3, the multi-view loss L is calculated by the regularization term in the view fusion stage MV ; Prediction result F k and the true value y k Calculate the cross entropy loss L CL ; and calculate the multi-task loss L through homoscedastic uncertainty MT In addition, the EWC algorithm parameters are consolidated to overcome the catastrophic forgetting problem and obtain incremental loss The final loss function: Using L F Calculate the gradient and update the model, that is where r m ∈(1,2,...R m ).

Images