Cross-user activity recognition method based on source domain data screening and training feature constraint

Abstract

The application discloses a cross-user activity recognition method based on source domain data screening and training feature constraint. The data distribution of different users is calculated through optimal transmission distance, and source domain data screening is performed according to the distribution distance of the users, thereby reducing the problem of poor domain adaptation migration. Data enhancement is performed in the screened source domain data, so that the source domain data distribution is more continuous, and the influence of the reduction of the source domain data is alleviated. Then, a joint loss of a regularization random drop loss, a classification loss and a hierarchical feature maximum mean error is constructed to train a network model, and through the training feature constraint and the hierarchical feature maximum mean error alignment method, better domain adaptation effect is realized. The application improves the generalization capability of the model in cross-user activity recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human activity recognition, and particularly relates to a cross-user activity recognition method based on source domain data screening and training feature constraint. BACKGROUND

[0002] With the continuous development of Micro-Electro Mechanical System (MEMS), the size of various physiological signal acquisition sensors is increasingly reduced, various wearable terminals are increasingly popular, and human activity recognition technology based on wearable sensors plays a very important role in our life and is widely used in health monitoring, sports assistance, human-computer interaction and other fields. However, due to different exercise habits, physical conditions and sensor wearing methods of each user, the distribution of physiological signal data collected by wearable sensors will have great differences. The existing human activity recognition research based on deep learning often randomly divides the training set and the test set, so that the data of the same user may appear in the training set or the test set. The recognition model obtained by such training will greatly decrease in performance when encountering new user data, that is, the cross-user activity recognition problem.

[0003] Cross-user activity recognition is a task with practical significance and challenges in human activity recognition. The existing cross-user activity recognition solution regards the data of each user as a domain, and the data of all users in the training set forms a source domain, and the data of the test set users forms a target domain. By combining domain adaptation means, the distribution of the target domain data is aligned with the distribution of the source domain data to improve the generalization ability of cross-user activity recognition. However, the existing solution ignores the fact that the data of the user in the source domain has a large distribution gap with the target domain data. If the target domain data is aligned with such data in the source domain, negative transfer will occur, which will damage the performance of the domain adaptation technology. Moreover, when the feature distribution extracted by the feature extractor is relatively discrete, it is also not conducive to aligning the data of the source domain and the target domain by domain adaptation. SUMMARY

[0004] The purpose of the present application is to provide a cross-user activity recognition method based on source domain data screening and training feature constraint to solve the above-mentioned deficiencies of the prior art, and to achieve better domain adaptation performance by screening source domain data according to distribution difference and limiting the feature distribution extracted by the feature extractor, thereby further improving the generalization ability of the model in the cross-user activity recognition scenario.

[0005] In order to achieve the above-mentioned purpose, the technical solution of the present application is as follows:

[0006] A cross-user activity recognition method based on source domain data screening and training feature constraint, comprising:

[0007] obtain user data of different users, calculate distribution gap between different user data based on optimal transport distance, and obtain a distance matrix;

[0008] divide the user data into source domain user data and target user data, filter the source domain user data based on the distance matrix, and apply linear data enhancement to the filtered source domain user data;

[0009] perform data preprocessing and sliding window segmentation on the finally obtained source domain user data to obtain samples of a training set, and perform data preprocessing and sliding window segmentation on the target user data to obtain samples of a test set;

[0010] construct a convolutional neural network model including a feature extractor and an action classifier, input the samples of the training set twice to the feature extractor to extract first features and second features, calculate a regularization random dropout loss, and obtain an action classification by the classifier to calculate a classification loss;

[0011] align the hierarchical features of the samples of the test set corresponding to the target user data with the hierarchical features extracted by the feature extractor from the samples of the training set corresponding to the source domain user data, and calculate a hierarchical feature maximum mean error;

[0012] calculate a joint loss including the regularization random dropout loss, the classification loss, and the hierarchical feature maximum mean error, perform back propagation, complete training of the convolutional neural network model, input the user data to be identified into the trained convolutional neural network model, and obtain an identification result.

[0013] Further, the distribution gap between different user data is calculated based on the optimal transport distance, wherein the optimal transport distance calculation formula is as follows:

[0014]

[0015] wherein and represent the data distribution of different users, represent all joint probability distributions between two user data, a and b are different user data samples, is the optimal transport distance between two distributions, ‖a-b‖ represents the Euclidean distance, E (a,b)∈γ ‖a-b‖ refers to the expectation of ‖a-b‖ under the condition that (a-b) obeys a gamma distribution, inf represents the lower bound, γ distribution belongs to all joint probability distributions between two user data.

[0016] Furthermore, the step of filtering source domain user data based on the distance matrix and applying linear data augmentation to the filtered source domain user data includes:

[0017] Determine the screening threshold k;

[0018] Discard k user data entries that differ significantly from the target user data from the source domain user data;

[0019] Linear data augmentation is applied to the filtered source domain user data.

[0020] Furthermore, the regularized random dropout loss is calculated using the following formula:

[0021]

[0022] in This represents the loss due to regularized random dropout. and For the first feature and the second feature, D KL It is the KL divergence function;

[0023] The classification loss is calculated using the following formula:

[0024]

[0025] Where L c For classification loss, n is the number of the input training sample, x i and y i Training sample data and corresponding action labels are performed separately.

[0026] Furthermore, the hierarchical features include shallow features, mid-level features, and deep features, and the maximum mean error of the hierarchical features is calculated using the following formula:

[0027] L MMD =λ s MMD(S s ,S t )+λ m MMD(M s M t )+λ d MMD(D s D t )

[0028] Where λ s ,λ m ,λ d L is the control factor. MMD MMD(S) represents the maximum mean error of hierarchical features. s ,S t MMD(M) represents the maximum mean error corresponding to shallow features. sM t MMD(D) represents the maximum mean error corresponding to the middle-level features. s D t ) represents the maximum mean error corresponding to the deep feature.

[0029] Furthermore, the feature extractor includes a convolutional network with a preset number of layers, each layer of which includes a one-dimensional convolutional layer, an activation function layer, and a batch normalization layer; the action classifier includes a fully connected layer, an activation function layer, a random deactivation layer, and an output layer; the feature extractor maps the input data to a common feature space, extracts temporal and spatial features from the input data, flattens them, and uses them as input to the action classifier. In the action classifier, multiplication and addition operations are performed through the fully connected layer, and then the result is fed into the activation function to add nonlinearity. Finally, after passing through the random deactivation layer, it is fed into the output layer, classifying the input features into one of the predefined actions.

[0030] This application proposes a cross-user activity recognition method based on source domain data filtering and training feature constraints. It utilizes wearable sensor data, including but not limited to motion sensor data, to filter the source domain dataset. By training feature constraints and aligning hierarchical features with the maximum mean error, the method improves the model's generalization ability in new user scenarios. It has the following beneficial effects:

[0031] 1. Calculate the data distribution of different users by using the optimal transmission distance, and filter source domain data according to the distribution distance of users to reduce the problem of poor domain adaptation migration.

[0032] 2. Perform data augmentation on the selected source domain data to make the distribution of source domain data more continuous, while mitigating the impact of reduced source domain data.

[0033] 3. Constrain the feature distribution extracted by the model to achieve better domain adaptation.

[0034] 4. By aligning the maximum mean error of hierarchical features, better domain adaptation is achieved, improving the model's generalization ability in cross-user activity recognition. Attached Figure Description

[0035] Figure 1 This is a flowchart of the cross-user identification method based on source domain data screening and training feature constraints in this application.

[0036] Figure 2 This is a schematic diagram of the source domain user data filtering mechanism in this application.

[0037] Figure 3 This is a schematic diagram of the hierarchical feature alignment network structure of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] In one embodiment, such as Figure 1 As shown, a cross-user activity recognition method based on source domain data filtering and training feature constraints is proposed, including:

[0040] Step S1: Obtain user data from different users, calculate the distribution gap between different user data based on the optimal transmission distance, and obtain the distance matrix.

[0041] This embodiment obtains user data X from different users through wearable sensors. These wearable sensors may include inertial sensors and / or electrophysiological sensors. For example, inertial sensors can collect user motion data, and electrophysiological sensors can collect user electrophysiological data. User data X can consist of data from individual inertial sensors or a combination of inertial sensor data and electrophysiological data, and is labeled with an action tag Y and a user tag G. Electrophysiological data includes, but is not limited to, ECG data.

[0042] The distribution gap between different user data is calculated based on the optimal transmission distance, where the formula for calculating the optimal transmission distance is as follows:

[0043]

[0044] in and This represents the data distribution among different users. Let a and b represent all joint probability distributions between two user data sets, where a and b are samples of different user data. The optimal transmission distance between the two distributions is determined by calculating the data distribution differences between different users sequentially and saving the results in matrix format to obtain the distance matrix. E (a,b) Expressing expectation, ||ab| represents Euclidean distance. (a,b)∈γ "|ab|" refers to the expectation of (ab) under the condition that it follows a γ distribution, and inf refers to the infimum of the following formula (expectation) (representing the largest lower bound among all elements in a set). The γ distribution refers to all joint probability distributions between two user data sets.

[0045] Step S2: Divide the user data into source domain user data and target user data, filter the source domain user data based on the distance matrix, and apply linear data augmentation to the filtered source domain user data.

[0046] In this embodiment, to improve action recognition performance in new user scenarios, user data is divided into source domain user data and target user data. Source domain user data is used for training, and target user data is used for testing. During dataset testing, each user's data is sequentially used as the target domain user, while other users are used as source domain users for testing. The results of each test are averaged to obtain the final result, thus achieving accurate model performance. In practical applications, training with new user data as the target domain is sufficient to improve the model's generalization ability in new user scenarios.

[0047] In a specific embodiment, such as Figure 2 As shown, source domain user data is filtered based on the distance matrix, and linear data augmentation is applied to the filtered source domain user data, including:

[0048] Step S2.1: Determine the screening threshold k.

[0049] The purpose of this embodiment is to filter out data from the source domain user data that differs significantly from the target user data. To this end, a filtering threshold k is first determined. This threshold k can be obtained by analyzing the dataset used in this embodiment, or it can be set manually.

[0050] For example, when analyzing the dataset used in this embodiment, it is possible to count how many different user data are in the dataset, and then view the distance matrix of the dataset. When there are k user data that are significantly different from the data of other users, the filtering threshold is determined to be k.

[0051] Step S2.2: Discard k user data entries from the source domain user data that differ significantly from the target user data.

[0052] For example, when the filtering threshold k equals 2, discard the two user data that are furthest from the target user data in the distance matrix.

[0053] Step S2.3: Apply linear data augmentation to the filtered source domain user data.

[0054] The specific linear data augmentation uses the following formula:

[0055]

[0056]

[0057] Where (x) i ,y i ) and (x j ,y j ) represents a randomly selected sample from the filtered source domain user data, and λ is the selection coefficient in the beta distribution, used to determine the degree of linear mixing between the two samples. The sample obtained using linear data augmentation.

[0058] It should be noted that (x i ,y i (x) can be the i-th segment of user data obtained by sliding window segmentation and its corresponding label, (x) j ,y j The data segment (j) can be the same user data or another user data segmented by sliding window, along with its corresponding label. This application is not limited to the various forms of linear data augmentation that are possible.

[0059] This embodiment controls the distribution of source domain user data through filtering and linear data augmentation, and applies linear data augmentation to the remaining source domain user data to make the distribution of source domain user data more continuous.

[0060] Step S3: Perform data preprocessing and sliding window segmentation on the final source domain user data to obtain training set samples, and perform data preprocessing and sliding window segmentation on the target user data to obtain test set samples.

[0061] In this embodiment, data preprocessing involves standardization, and the formula is shown below:

[0062]

[0063] Where the vector μ is the mean of the training sample x, σ is the standard deviation of the training sample, and ι is a local minimum.

[0064] The final source domain user data obtained is X s The target user data is X g For source domain user data X s Data preprocessing and sliding window segmentation are performed to obtain the training set samples. in This represents the specific source domain user data, action tags, and user tags of the i-th segment of the source domain user data after it has been segmented by a sliding window.

[0065] For target user data X g Samples of the test set for data preprocessing and sliding window segmentation

[0066] Step S4: Construct a convolutional neural network model including a feature extractor and an action classifier. Input the samples of the training set into the feature extractor twice to extract the first feature and the second feature. Calculate the regularized random dropout loss. Then, pass the first feature through the classifier to obtain the action classification and calculate the classification loss.

[0067] In this embodiment, the constructed convolutional neural network model includes a feature extractor F and an action classifier C. The feature extractor F is composed of a convolutional network with a preset number of layers (e.g., 9 layers), and each convolutional network layer includes: a one-dimensional convolutional layer, an activation function layer, and a batch normalization layer. The action classifier C consists of a fully connected layer, an activation function layer, a random deactivation layer, and an output layer. The feature extractor maps the input data to a common feature space, extracts temporal and spatial features from the input data, flattens them, and uses them as input to the action classifier. In the action classifier, multiplication and addition operations are performed through a fully connected layer, and the result is then fed into an activation function to add non-linearity. Finally, after passing through a random deactivation layer, the result is fed into the output layer, classifying the input features into one of the predefined actions.

[0068] During training, the training set data By inputting data into a convolutional neural network model in batches, we can obtain the first feature and action classification, and calculate the loss L for action classification of user data from the source domain. c The calculation formula is as follows:

[0069]

[0070] Where L c For classification loss, n is the number of the input training sample, x i and y i Training sample data and corresponding action labels are performed separately.

[0071] This application also includes training set data. The second feature is obtained by inputting batches of data into the feature extractor of the convolutional neural network model, and the regularized random dropout loss is calculated.

[0072]

[0073] in and To input the training set into the model, two features are obtained: the first feature and the second feature. KL It is the KL divergence function, i.e., the formula for calculating KL divergence (Kullback-Leibler Divergence, KLD).

[0074] This embodiment constrains the extracted feature distribution using a regularized random dropout method, inputting the training set data twice into the feature extractor to calculate the regularized random dropout loss.

[0075] Step S5: Extract hierarchical features from the test set samples corresponding to the target user data using a feature extractor, align the hierarchical features extracted by the feature extractor with the samples from the training set corresponding to the source domain user data using the maximum mean error, and calculate the maximum mean error of the hierarchical features.

[0076] In this embodiment, the feature extractor F extracts hierarchical features from the input samples. For example, the feature extractor has a 9-layer network, with the first 3 layers extracting shallow features, the middle 3 layers extracting mid-level features, and the last 3 layers extracting deep features. This application is not limited to the specific extraction form of hierarchical features; the following explanation uses a hierarchical feature composed of shallow, mid-level, and deep features as an example.

[0077] like Figure 3 As shown, feature extractor F extracts unlabeled data from the target user. Hierarchical features S t M t D t (represented by shallow features, mid-level features, and deep features), corresponding to the hierarchical features S in the source domain user data. s M s D s Maximum Mean Discrepancy (MMD) alignment is performed. The formula for calculating the maximum mean discrepancy is as follows:

[0078]

[0079] in For the source domain user data feature distribution, For the target user data feature distribution, Let n denote the regenerated Hilbert space, φ(·) be a mapping method for projecting samples into the regenerated Hilbert space. s and n t These represent the amount of source domain user data and the amount of target user data, respectively.

[0080] Therefore, the maximum mean error of the hierarchical features is L MMD The following formula can be used for calculation:

[0081] L MMD =λ s MMD(S s ,S t )+λ m MMD(M s M t )+λ d MMD(D s D t )

[0082] Where λ s ,λ m ,λ d These are the control factors for aligning the maximum mean error of hierarchical features.

[0083] Step S6: Calculate the joint loss including regularized random dropout loss, classification loss, and maximum mean error of hierarchical features, perform backpropagation to complete the training of the convolutional neural network model, input the user data to be identified into the trained convolutional neural network model, and obtain the recognition result.

[0084] The joint loss L of the convolutional neural network model in this embodiment obj :

[0085] L obj =L C +L R +L MMD

[0086] Imminent Classification Loss L C Regularized random dropout loss L R Maximum mean error of hierarchical features L MMD Combining these, we obtain the overall loss function of the model. After calculating the joint loss, we then apply L... obj Backpropagation updates the model's parameters. Training network models using loss functions is a well-established technique in this field and will not be elaborated upon here.

[0087] After the convolutional neural network model is trained, it can be used to identify the user data to be identified.

[0088] Compared to other domain adaptation methods, the cross-user activity recognition method proposed in this embodiment, based on source domain data filtering and training feature constraints, reduces the impact of undesirable migration on domain adaptation through source domain data filtering, enhances the performance of domain adaptation through source domain data linear augmentation and training feature constraints, and finally achieves the generalization capability of activity recognition in cross-user scenarios through a hierarchical feature domain adaptation method.

[0089] This embodiment selects the Physical Activity Monitoring Data Set (PAMAP2), a challenging dataset in the field of human activity recognition, to verify the effectiveness of the proposed method. The PAMAP2 dataset collects data from 9 different users wearing 3 inertial sensors and 1 heart rate sensor, performing 18 different actions. The inertial sensors collect data at a frequency of 100Hz, and the heart rate sensor at a frequency of 9Hz. The users participating in the data collection performed activities according to their individual habits. This embodiment uses Leave-One-Out (LOSO) to test the generalization performance of the proposed method in cross-user scenarios for activity recognition. Specifically, all data from one user out of N users is selected as the test set, and the data from the other users is used as the training set. This process is repeated N times, and the average recognition accuracy is calculated. Table 1 compares the generalization performance of the convolutional neural network-based method, the recurrent neural network-based method, the adversarial learning domain adaptation method, and the proposed method. The results are presented in the form of mean ± standard deviation.

[0090] Classification method Accuracy (%) F1-score (%) Convolutional neural network based method 81.23±12.16 75.28±10.90 Recurrent neural network based method 78.18±13.57 70.29±12.93 Domain adaptation method combined with adversarial learning 85.69±10.76 77.84±11.69 The technical solutions of the present application 88.17±4.73 79.10±9.04

[0091] Table 1

[0092] The results in Table 1 show that the accuracy and F1 score of the technical solution in this application are better than those of other existing methods.

[0093] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for cross-user activity recognition based on source domain data filtering and training feature constraints, characterized in that, The cross-user activity recognition method based on source domain data filtering and training feature constraints includes: Obtain user data from different users, calculate the distribution difference between the data of different users based on the optimal transmission distance, and obtain the distance matrix; User data is divided into source domain user data and target user data. Source domain user data is filtered based on the distance matrix, and linear data augmentation is applied to the filtered source domain user data. The source domain user data is preprocessed and segmented using a sliding window to obtain the training set samples, and the target user data is preprocessed and segmented using a sliding window to obtain the test set samples. Construct a convolutional neural network model that includes a feature extractor and an action classifier. Input the samples of the training set into the feature extractor twice to extract the first feature and the second feature. Calculate the regularized random dropout loss. Then, pass the first feature through the action classifier to obtain the action classification and calculate the classification loss. The hierarchical features of the test set samples corresponding to the target user data are extracted by the feature extractor, and the hierarchical features of the training set samples corresponding to the source domain user data are aligned with the maximum mean error of the extracted hierarchical features to calculate the maximum mean error of the hierarchical features. The joint loss, including regularized random dropout loss, classification loss, and maximum mean error of hierarchical features, is calculated, backpropagation is performed, the training of the convolutional neural network model is completed, the user data to be identified is input into the trained convolutional neural network model, and the identification result is obtained. The step of filtering source domain user data based on a distance matrix and applying linear data augmentation to the filtered source domain user data includes: Determine the screening threshold k; Discard k user data entries that differ significantly from the target user data from the source domain user data; Linear data augmentation is applied to the filtered source domain user data.

2. The cross-user activity recognition method based on source domain data filtering and training feature constraints according to claim 1, characterized in that, The distribution gap between different user data is calculated based on the optimal transmission distance, where the formula for calculating the optimal transmission distance is as follows: ； in and This represents the data distribution among different users. Represents all joint probability distributions between two user data sets. and For different user data samples, The optimal transmission distance between the two distributions. Represents Euclidean distance. It means Obedience Under the condition of distribution The expectation is given by inf, which represents the infimum. refer to The distribution belongs to all joint probability distributions between two user data sets.

3. The cross-user activity recognition method based on source domain data filtering and training feature constraints according to claim 1, characterized in that, The regularized random dropout loss is calculated using the following formula: ； in This represents the loss due to regularized random dropout. and The first feature and the second feature, It is the KL divergence function; The classification loss is calculated using the following formula: ； in For classifying losses, Enter the training sample ID. and The training samples and corresponding action labels are trained separately. F represents the feature extractor and C represents the action classifier.

4. The cross-user activity recognition method based on source domain data filtering and training feature constraints according to claim 1, characterized in that, The hierarchical features include shallow features, mid-level features, and deep features. The maximum mean error of the hierarchical features is calculated using the following formula: ； in As a control factor, This represents the maximum mean error of the hierarchical features. This represents the maximum mean error corresponding to shallow features. This represents the maximum mean error corresponding to the middle-level features. This represents the maximum mean error corresponding to the deep features.

5. The cross-user activity recognition method based on source domain data filtering and training feature constraints according to claim 1, characterized in that, The feature extractor includes a convolutional network with a preset number of layers, each layer of which includes a one-dimensional convolutional layer, an activation function layer, and a batch normalization layer. The action classifier includes a fully connected layer, an activation function layer, a random deactivation layer, and an output layer. The feature extractor maps the input data to a common feature space, extracts temporal and spatial features from the input data, flattens them, and uses them as input to the action classifier. In the action classifier, multiplication and addition operations are performed through a fully connected layer, and the result is then fed into an activation function to add non-linearity. Finally, after passing through a random deactivation layer, the result is fed into the output layer, classifying the input features into one of the predefined actions.

Images