A training method of a neural network classification model for identifying human behaviors

Abstract

The application provides a neural network classification model training method, the model comprising a feature extractor for extracting sample features and a classifier for classifying samples according to sample features, the method comprising one or more rounds of training of an initial model trained based on source domain samples and labels in the following manner: S1, obtaining a training set comprising source domain sample pairs and target domain sample pairs, any sample pair comprising two samples, each sample being sensor data collected from a person, the two samples corresponding to the same person and the same behavior category of the person when collected; S2, training the current model one or more times using the training set, updating the parameters of the model based on a predetermined loss function each time, the loss function being configured to: penalize classification bias of source domain samples, penalize edge distribution difference and conditional distribution difference between sample features of source domain and target domain samples, and penalize time series distribution difference corresponding to the two samples in each sample pair.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of health big data and transfer learning, in particular to a training method of a neural network classification model for identifying human behaviors. BACKGROUND

[0002] Due to the small volume, low power consumption, portability, and small data storage space, sensors are widely used in behavior recognition. For example, in Parkinson tremor recognition, subjects can use common smart phones or bracelets (or any electronic device with an accelerometer and gyroscope) to collect their postural tremor and resting tremor data, and then diagnose the degree of illness of the subjects according to the tremor amplitude, which will greatly save the time cost of the subjects and doctors.

[0003] However, the sensor dataset generally faces the problem of small labeled data and low dataset quality. Unlike pictures, audio, and text, which can be easily labeled after data collection, sensor data is often difficult for humans to understand directly, which makes it difficult to label sensor data after data collection, and a large amount of data is unlabeled. Since the advent of transfer learning, it has been widely used in human behavior recognition with a lack of data labeling because it can apply the knowledge learned from one field to another new field. Specifically, it regards the data labeled by professionals as the source domain and the unlabeled data as the target domain, finds the similarity between the source domain and the target domain through instance transformation, feature extraction, and subspace search, and then applies the model learned from the source domain to the target domain. In transfer learning, similarity can be calculated by measuring the difference in joint probability distribution between the source domain and the target domain. With the rise of deep learning, the mainstream transfer learning algorithm extracts the features of the source domain and the target domain through a deep neural network, calculates the difference in marginal distribution between the source domain and the target domain through a domain discriminator, and calculates the difference in conditional distribution between different classes through a class discriminator, so as to make the joint probability distribution difference as small as possible to improve the model recognition accuracy of the target domain.

[0004] At the same time, the collection time of sensor data is generally long, and is susceptible to interference during data collection. The interference can be caused by displacement of the sensor wearing position, or the state of the subject collected by the sensor changes, so that the initial data distribution is different in different time periods, and the time series distribution difference phenomenon occurs. Unlike noise that affects only a few data points and has a random distribution, time series distribution difference follows a certain distribution that changes over time, which affects a large number of data points in a time period. At the same time, since the cause of the time series distribution difference is unpredictable, it is not feasible to fit its distribution, which makes the time series distribution difference exist universally in the sensor data set and difficult to eliminate. Therefore, when performing transfer learning on sensor time series data, based on the existing transfer learning method, it is difficult to achieve as good results as performing transfer learning on image data sets. SUMMARY

[0005] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a training method of a neural network classification model for identifying human behaviors.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] According to a first aspect of the present application, a training method of a neural network classification model for identifying human behaviors is provided, the model comprising a feature extractor for extracting sample features from samples and a classifier for classifying samples according to sample features, the method comprising training the model with samples in a source domain and labels to obtain an initial model, and performing one or more rounds of iterative training based on the initial model in the following manner: S1, obtaining a training set, including source domain sample pairs generated based on source domain samples and target domain sample pairs generated based on target domain samples, wherein any sample pair includes two samples, each sample is sensor data collected from a person, the two samples correspond to the same person and the same behavior category of the person when collected, but the two samples are collected at different times, and each sample in the source domain in the training set is provided with a label indicating the behavior category of the corresponding person when collected; S2, training the current model one or more times using the training set, updating the parameters of the model based on a predetermined loss function each time, the loss function being configured to: penalize the classification deviation of the source domain samples, penalize the marginal distribution difference and conditional distribution difference between the sample features corresponding to the source domain and target domain samples, and penalize the time series distribution difference between the two samples in the source domain and target domain sample pairs, the time series distribution difference being related to the similarity of the confidence probability vector of the classifier classifying the two samples in the sample pair.

[0008] In some embodiments of the present application, the timing distribution difference is determined in the following manner: the confidence probability of each category of the sample is obtained from the sample features of the sample by using the classifier, and is spliced into a confidence probability vector corresponding to the sample; the similarity between the confidence probability vectors corresponding to the two samples in the sample pair is calculated, and the timing distribution difference is determined according to the similarity between the two samples corresponding to all sample pairs used in one training.

[0009] In some embodiments of the present application, the timing distribution difference is determined in the following manner:

[0010]

[0011] wherein, m s represents the total number of sample pairs of the source domain used in one training, x si represents the i th sample pair of the source domain, represents one sample in the i th sample pair x si of the source domain, represents the other sample in the i th sample pair x si of the source domain, sim(·) represents the cosine similarity, C(·) represents the confidence probability of each category output by the classifier, F(·) represents the output of the feature extractor, m t represents the total number of sample pairs of the target domain used in one training, x ti represents the i th sample pair of the target domain, represents one sample in the i th sample pair x ti of the target domain, represents the other sample in the i th sample pair x ti of the target domain.

[0012] In some embodiments of the present application, the penalty on the conditional distribution difference between the sample features of the sample pairs of the source domain and the target domain includes: determining the conditional distribution difference between all samples of the source domain and the target domain according to the sample features of all samples of the source domain and the target domain used in each training, the labels of all samples of the source domain and the pseudo labels of all samples of the target domain, wherein the pseudo label is the classification result obtained by classifying the samples in the target domain by using the current model.

[0013] In some embodiments of the present application, the conditional distribution difference is calculated in the following manner in one training:

[0014]

[0015] wherein, G c represents the conditional distribution difference between all samples of the source domain and the target domain used in one training, K represents the total number of categories, k represents the k th category, denotes the square of the distance in the Hilbert space, n s denotes the total number of samples of the source domain in one training, denotes the weight of the kth category in the source domain, denotes the total number of samples belonging to the kth category in the source domain, φ(·) denotes a feature mapping method, and F(·) denotes the output of a feature extractor, denotes the ith sample of the source domain, n t denotes the total number of samples of the target domain in one training, denotes the weight of the kth category in the target domain, denotes the total number of samples belonging to the kth category in the target domain, denotes the pseudo label of the ith sample of the target domain, denotes the ith sample of the target domain.

[0016] In some embodiments of the present application, the updating of the parameters of the model based on the predetermined loss function comprises updating the parameters of the model based on the total loss calculated based on the predetermined loss function, wherein the total loss is calculated in the following manner in one training:

[0017]

[0018] wherein J(·) denotes the classification bias of the samples of the source domain in one training, C(·) denotes the confidence probability of the output of the classifier for each category, F(·) denotes the output of the feature extractor, x s denotes the sample in the source domain, y s denotes the label of the sample of the source domain, α denotes the weight value of the edge distribution difference, G m (·) denotes the edge distribution difference between all samples of the source domain and the target domain used in one training, x t denotes the sample in the target domain, β denotes the weight value of the conditional distribution difference, G c (·) denotes the conditional distribution difference between all samples of the source domain and the target domain used in one training, σ denotes the weight value of the time sequence distribution difference, G t (·) denotes the time sequence distribution difference between the two samples of a sample pair of the source domain and the target domain used in one training, x i denotes the ith sample pair, denotes one sample in the sample pair x i denotes the other sample in the same sample pair x i α+β+σ=1. ​

[0019] In some embodiments of the present application, the two samples in each sample pair are collected at different times and the interval is less than or equal to a predetermined time length.

[0020] In some embodiments of the present application, the sample pairs are generated by: obtaining a data set of the source domain and the target domain, which includes a plurality of initial sensor data collected from different behaviors of different persons for a predetermined time length; dividing each initial sensor data of the source domain and the target domain into a plurality of segments of sensor data according to a predetermined data length; randomly selecting a segment of data from the plurality of segments of sensor data corresponding to each initial sensor data of the source domain as one sample, randomly selecting another segment of data from the plurality of segments of sensor data corresponding to the initial sensor data to which the one sample belongs as another sample, and forming a sample pair of the source domain by combining the two selected samples, and setting the same label for each sample in the sample pair as the label of the initial sensor data to which the sample belongs; and randomly selecting a segment of data from the plurality of segments of sensor data corresponding to each initial sensor data of the target domain as one sample, randomly selecting another segment of data from the plurality of segments of sensor data corresponding to the initial sensor data to which the one sample belongs as another sample, and forming a sample pair of the target domain by combining the two selected samples.

[0021] In some embodiments of the present application, the identifying the behavior category of the human body includes identifying a behavior category of Parkinson tremor, wherein the behavior category of Parkinson tremor includes no tremor, mild tremor, moderate tremor, and severe tremor.

[0022] In some embodiments of the present application, the identifying the behavior category of the human body includes identifying a behavior category of daily activities of the human body, wherein the behavior category of daily activities of the human body includes walking, climbing stairs, descending stairs, and sitting.

[0023] According to a second aspect of the present application, a method for identifying human behavior is provided, comprising: obtaining sensor data to be identified collected from a human body; using a feature extractor of a neural network classification model for identifying human behavior obtained by the training method of the first aspect of the present application to perform feature extraction on the sensor data to be identified, and identifying human behavior according to the extracted features through the classifier thereof to obtain a behavior category of the human body.

[0024] According to a third aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is configured to store executable instructions; the one or more processors are configured to implement the steps of the method of the first aspect of the present application and the second aspect of the present application by executing the executable instructions.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] The training set in the application includes sample pairs of source domains and target domains, any sample pair includes two samples, each sample is sensor data collected from a person, the persons corresponding to the two samples are the same, and the behavior categories of the persons when the samples are collected are the same, but the two samples are collected at different times, when the neural network classification model is trained by using the training set, the loss function used not only considers the classification bias, the edge distribution difference and the conditional distribution difference existing in the transfer learning, but also increases the calculation of the time series distribution difference of the two samples in the sample pair, the model is updated based on the loss function including the calculation of the time series distribution difference for multiple times, so that the model eliminates the sample feature difference caused by the different states of the persons corresponding to the two samples in the sample pair due to the different collection times to a certain extent, and the ability of the model to recognize and classify the sensor data collected from the wearable sensor is improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] The embodiments of the application are further described below with reference to the drawings, in which:

[0028] Figure 1 The structure schematic diagram of the neural network classification model for recognizing human behaviors according to one embodiment of the application;

[0029] Figure 2 The generation process schematic diagram of the sample of the source domain according to one embodiment of the application;

[0030] Figure 3 The flow chart of the method for training the neural network classification model according to one embodiment of the application;

[0031] Figure 4 The data processing process and principle schematic diagram when the neural network classification model is trained according to one embodiment of the application. DETAILED DESCRIPTION

[0032] In order to make the objects, technical schemes and advantages of the application clearer, the application is further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0033] As mentioned in the background section, the collection time of sensor data usually lasts for a long time, and is susceptible to interference during data collection. In turn, the same data has different data distributions at different time periods, resulting in the phenomenon of time series discrepancy. At the same time, the reason for the time series distribution discrepancy is unpredictable, and it is not feasible to fit its distribution, which makes the time series distribution discrepancy exist universally in the sensor data set and difficult to eliminate, resulting in that when the sensor time series data is subjected to transfer learning, it is difficult to achieve good results based on the existing transfer learning methods.

[0034] To solve the above problems, the inventors provide a training method for a neural network classification model for identifying human behaviors, which utilizes a training set including a plurality of sample pairs composed of source domain samples and a plurality of sample pairs composed of target domain samples, any one sample pair including two samples, wherein each sample is sensor data collected from a person, the two samples correspond to the same person and the same behavior category of the person when collected, but the two samples are collected at different times. When training the neural network classification model using the training set, not only the marginal distribution discrepancy and the conditional distribution discrepancy existing in transfer learning are considered, but also the time series distribution discrepancy of the two samples in the sample pair is calculated. The time series distribution discrepancy is related to the similarity of the confidence probability vector of the classifier for classifying the two samples in the sample pair. Based on the loss function including the calculation of the time series distribution discrepancy, the model is updated multiple times, so that the model eliminates the sample feature discrepancy caused by the different states of the person corresponding to the two samples due to different collection times to a certain extent, and improves the ability of the model to identify and classify the sensor data collected from the wearable sensor.

[0035] In order to better understand the present application, the present application will be described in detail below from the aspects of model structure, training set, model training, and application scenario in combination with the drawings and embodiments.

[0036] I. Model structure

[0037] According to an embodiment of the present application, referring to Figure 1 The present application provides a neural network classification model for identifying human behaviors, which includes a feature extractor for extracting sample features from samples and a classifier for classifying samples according to sample features, such as obtaining a sample from a target domain Target for feature extraction to obtain sample features f tThe classifier obtains a classification result according to the sample feature. The feature extractor can adopt an existing neural network, such as ResNet1D, LSTM, ConvLSTM, Cnn1D, etc. The classifier includes a one-layer fully connected neural network layer and a Softmax layer. The one-layer fully connected neural network layer is used for linear transformation of the sample feature of the sample to obtain a classification feature. The Softmax layer is used for calculating the confidence probability of each category according to the classification feature of the sample. The classifier can also be composed of a multi-layer fully connected neural network layer and a Softmax layer. The classifier finally outputs the category with the highest confidence probability as the predicted classification result. The model structure of the present application is shown in Figure 1 To more clearly understand how the losses of each part are calculated when the model is trained in the present application, the training process of the model and how the losses of each part are calculated in the training process are described in the form shown in Figure 4 The feature extractor and the classifier after the training are finally used to compose the neural network classification model.

[0038] II. Training set

[0039] According to an embodiment of the present application, the present application can generate a training set by using an existing sensor data set. The training set includes a source domain sample pair generated based on a source domain sample and a target domain sample pair generated based on a target domain sample. Each sample pair includes two samples, each sample is sensor data collected from a person, the persons corresponding to the two samples are the same, and the behavior category of the person when the samples are collected is the same, but the two samples are collected at different times and the interval is less than or equal to a predetermined time length. Each sample of the source domain is provided with a label indicating the behavior category of the corresponding person when the sample is collected.

[0040] According to an embodiment of the present application, the source domain sample pair generated based on the source domain sample and the target domain sample pair generated based on the target domain sample include the following steps a1, a2, a3 and a4:

[0041] In step a1, a data set of the source domain and the target domain is obtained, which includes a plurality of initial sensor data of a predetermined time length collected from different behavior categories of different persons.

[0042] According to one embodiment of the present application, different sensor data sets are selected for different human behavior application scenarios. For example, for human daily activities, the source domain adopts the UCIHAR dataset (Human activity recognition using smartphones dataset), which is collected from 30 subjects aged 19-48 years old, a smartphone is bound to the waist of the subject and the accelerometer sensor data is collected at a sampling frequency of 50HZ, at the same time a video recorder is used to record in real time for subsequent data labeling, the accelerometer sensor data in the experiment is used as the initial sensor data of the source domain; the target domain adopts the WISDM dataset (Wireless Sensor Data Mining), which places a smartphone in the pocket of 29 subjects and collects accelerometer sensor data of the subjects walking, jogging, climbing stairs, sitting and standing at a sampling frequency of 20HZ as the initial sensor data of the target domain. The source domain can also adopt the PAMAP2 dataset (Physical activity monitoring dataset), which collects 18 kinds of activity motions from 9 subjects, including 3 accelerometer sensor data (located on the arm, chest and ankle respectively) and 1 heart rate sensor data, the present application uses the accelerometer sensor data located on the arm as the initial sensor data of the source domain, and the sampling frequency is 100HZ, and the target domain can also adopt the UCIHAR dataset (Human activity recognition using smartphones dataset). Among the above three datasets UCIHAR, WISDM and PAMAP2, one can be selected as the dataset of the source domain, and one can be selected as the dataset of the target domain from the remaining two datasets. For example, for Parkinson tremor, the source domain adopts the TIM-Tremor dataset (Technology in Motion Tremor), which collects postural tremor and resting tremor data and video recordings of 55 patients using an accelerometer sensor attached to the back of the patient's hand.The target domain can use the PdAssist dataset (Objective and Quantified Symptom Assessment of Parkinson’s Disease via Smartphone), which was created by the dataset creator through the smartphone assistant developed by the dataset creator, hundreds of patients hold the accelerometer sensor of the left and right hands for static tremor test, and collect the Parkinson tremor data of the patients at a sampling frequency of 50HZ; the target domain can also use the IMU-Wild dataset (IMU Data Captured Unobtrusively in-the-wild by Parkinson’s Disease Patients and Healthy Controls), which collects tremor data of subjects in the “phone call” action under outdoor conditions, including accelerometer and gyroscope data collected at a sampling frequency of 50HZ for 45 subjects when the tremor associated with Parkinson’s disease occurs, including 31 patients and 14 healthy people; the target domain can also use the PD-BioStamp dataset (Parkinson’s Disease Accelerometry Dataset from Five Wearable Sensor Study). Among them, for Parkinson’s tremor, only the TIM-Tremor dataset is selected as the source domain dataset among the above four datasets TIM-Tremor, PdAssist, IMU-Wild, and PD-BioStamp, and the remaining datasets are imbalanced in the collection category, and any one of the remaining datasets is selected as the target domain dataset.

[0043] In step a2, each initial sensor data of the source domain and the target domain is divided according to a predetermined data length, to obtain a plurality of segments of sensor data corresponding to each initial sensor data.

[0044] According to one embodiment of the present application, before training, the data and labels of the data set of the source domain and the data of the data set of the target domain are collectively used as the training data set, during training, the data and labels of the data set of the source domain (for example, the UCI HAR data set) and the data of the data set of the target domain (for example, the WISDM data set) are used to train the model to predict the labels of the data of the data set of the target domain (the WISDM data set), during testing, the data and labels of the data set of the target domain (the WISDM data set) are used as the test data set, and the correctness of the predicted labels of the data of the data set of the target domain (the WISDM data set) is used as the evaluation standard. Among them, before training, the sampling frequencies of each data set of the source domain and the target domain need to be unified to 50HZ by linear interpolation, and then the initial sensor data of the source domain and the target domain after unifying the sampling frequency is segmented, wherein the predetermined data length of segmentation can be set according to experience. For example, the present application segments the initial sensor data of the source domain and the target domain according to a step length of 2.56 seconds without overlapping, and each segment of sensor data includes a standard length of 128 data points, so when the time for collecting each segment of initial sensor data is 10 seconds, the time interval between the two samples in the sample pair is less than or equal to the predetermined time length of 7.44 seconds. The present application processes the initial data set in the data set according to the predetermined data length segmentation in this embodiment to avoid long and uneven data segments, which makes it difficult for the model to learn, and lacks generalization and robustness.

[0045] In step a3, a piece of data is randomly selected from the multiple pieces of sensor data corresponding to each initial sensor data of the source domain as a sample, another piece of data is randomly selected from the multiple pieces of sensor data corresponding to the initial sensor data to which the sample belongs as another sample, and the two selected samples are combined to form a sample pair of the source domain, and each sample is set with the same label as the label of the initial sensor data to which it belongs.

[0046] According to one example of the present application, the generation method of the sample pair of the source domain is described. Referring to Figure 2 , Figure 2For the generation process of the samples of the source domain, firstly, initial sensor data of a predetermined time length collected under a behavior category of a person is acquired, the initial sensor data can be labeled as 1, which is used to indicate a behavior category of the corresponding person when the initial sensor data is collected; the initial sensor data is divided into three pieces of sensor data according to a predetermined data length, a first piece of sensor data is randomly selected as a sample 1, a second piece of sensor data is randomly selected as another sample 2, and the sample 1 and the sample 2 form a sample pair, since the initial sensor data to which the two samples belong is labeled as 1, the labels of the two samples are set as 1, and the selected first piece of sensor data and a third piece of sensor data can be respectively used as the sample 1 and a sample 3 to form a sample pair, or the selected second piece of sensor data and the third piece of sensor data can be respectively used as the sample 2 and the sample 3 to form a sample pair, since the initial sensor data to which the sample 1, the sample 2 and the sample 3 belong is labeled as 1, the labels of the three samples are all set as 1.

[0047] In step a4, a piece of data is randomly selected from the multiple pieces of sensor data corresponding to each initial sensor data of the target domain as a sample, another piece of data is randomly selected from the multiple pieces of sensor data corresponding to the initial sensor data to which the sample belongs as another sample, and the two selected samples form a sample pair of the target domain. In the embodiment of the present application, the sample pair of the target domain is formed according to the above manner, so that the two samples of the sample pair of the target domain can still be determined to be of the same label without setting the labels, so as to better train the neural network classification model to recognize the sensor data collected in different time periods and from the same person under the same behavior category.

[0048] III. Model training

[0049] According to an embodiment of the present application, referring to Figure 3 The present application provides a training method of a neural network classification model for recognizing human behaviors, the method comprises training a model by using samples in a source domain and labels to obtain an initial model, the initial model can realize recognition and classification of the samples, and one or more rounds of iterative training are performed according to steps S1 and S2 based on the initial model. In order to better understand the present application, each step will be described in detail below in combination with specific embodiments.

[0050] In step S1, a training set is obtained, including sample pairs generated based on samples of a source domain and sample pairs generated based on samples of a target domain, wherein any sample pair includes two samples, each sample is sensor data collected from a person, the two samples correspond to the same person and the same behavior category of the person when the samples are collected, but the two samples are collected at different times, and each sample of the source domain in the training set is provided with a label indicating the behavior category of the corresponding person when the sample is collected.

[0051] In step S2, the current model is trained one or more times using the training set, and the parameters of the model are updated based on a predetermined loss function each time the model is trained, and the loss function is configured to: punish the classification deviation of the samples of the source domain, punish the edge distribution difference and the conditional distribution difference between the sample features corresponding to the sample pairs of the source domain and the target domain, and punish the time sequence distribution difference between the two samples corresponding to the sample pairs of the source domain and the target domain, and the time sequence distribution difference is related to the similarity of the confidence probability vector of the classifier for classifying the two samples in the sample pair.

[0052] According to one embodiment of the present application, referring to Figure 4 , Figure 4 The data processing process for training the neural network classification model, in training the neural network classification model, a transfer learning module and a time sequence distribution adaptive network are introduced, the transfer learning module is used to calculate the edge distribution difference and the conditional distribution difference, the time sequence distribution adaptive network is used to calculate the time sequence distribution difference, and the classifier shown in the time sequence distribution adaptive network is actually a classifier in the neural network classification model; in other words, the classifier in the time sequence distribution adaptive network is only illustrative, and the classifier here is used to understand that the time sequence distribution difference is calculated based on the confidence probability vector of the classifier for classifying the two samples in the sample pair. In each training, the required sample pair is selected from the plurality of sample pairs of the source domain and the target domain and input into the model for training, and the data processing process of the model is described with one sample pair of the source domain and the target domain, the two samples of the sample pair in the source domain and and the two samples of the sample pair in the target domain and are input into the feature extractor in sequence for feature extraction, and the feature f s of the sample pair of the source domain and the feature f t of the sample pair of the target domain are output, wherein f s includes the sample feature f of the sample x and the sample feature f s2 of the sample x l si f t includes the sample feature f of the sample x jti and the sample feature f of the sample x t2 l ti the sample feature f of the sample of the source domain j si and f l si classification in the classifier in the input model to obtain the classification result thereof, and the classification deviation of the sample of the source domain is determined according to the sample classification result and the label of the sample, the sample feature f of the sample of the target domain j ti and f l ti classification in the classifier in the input model to obtain the classification result of each sample of the target domain, and the classification result of the sample of the current target domain is taken as the pseudo label of the sample, the migration learning module is used to determine the edge distribution difference and the conditional distribution difference between the sample features corresponding to the samples of the source domain and the target domain according to the sample features of the samples of the source domain and the target domain, and the time sequence distribution adaptive network is used to determine the time sequence distribution difference corresponding to two samples in a sample pair of the source domain and the target domain according to the sample features of the two samples. Wherein, the time sequence distribution adaptive network outputs the confidence probability of two samples under each category based on the sample features of the two samples in the sample pair through the classifier, and the confidence probability of the sample under each category is spliced into the confidence probability vector corresponding to the sample, such as the sample and the confidence probability vectors corresponding to each other are and the time sequence distribution difference corresponding to two samples in a sample pair is obtained according to the output confidence probability vector. Each time the model is updated by the loss calculated in the above manner, the edge distribution difference and the conditional distribution difference between the data sets of the target domain and the source domain are narrowed, and the time sequence distribution difference corresponding to two samples in a sample pair is reduced to a certain extent.

[0053] According to one embodiment of the present application, in one or more training, one training refers to a batch of training. For example, the sample pairs of the source domain and the sample pairs of the target domain used in a batch are both 32, then 128 samples are input for one training, and the total loss is calculated during training and the model is updated according to the total loss, at the same time, the classifier in the time sequence distribution adaptive network uses the corresponding classifier of the model after the model is updated in the next training.

[0054] According to one embodiment of the present application, in one training, the classification deviation of the sample of the source domain is calculated by using the cross entropy loss function, and the calculation method is as follows:

[0055]

[0056] ​wherein J(·) represents the classification bias of all samples of the source domain in one training, C(·) represents the confidence probability of the classifier output for each category, F(·) represents the output of the feature extractor, x s represents the sample in the source domain, y s represents the label of the sample of the source domain, n s represents the total number of samples of the source domain used in one training, K represents the total number of categories, and p represents probability, represents the i th sample of the source domain, represents the confidence probability of the classifier output that the sample is of the k th category, k) represents that the label of the sample belongs to the k th category, is 1, and the label of the sample does not belong to the k th category, is 0, represents the label of the i th sample of the source domain.

[0057] According to one embodiment of the present application, in one training, the edge distribution difference between the sample features corresponding to the samples of the source domain and the target domain is calculated by using the square of the Maximum Mean Discrepancy (MMD distance) as a loss function, and the calculation formula is as follows:

[0058]

[0059] wherein G m (·) represents the edge distribution difference between all samples of the source domain and the target domain used in one training, x t represents the sample of the target domain, represents the square of the distance in the Hilbert space, n s represents the total number of samples of the source domain in one training, and φ(·) represents a feature mapping method, and the kernel method is used in the MMD distance, represents the i th sample of the source domain, n t represents the total number of samples of the target domain in one training, represents the i th sample of the target domain, and H is a Reproducing Kernel Hilbert Space (RKHS). The edge distribution difference calculated by the transfer learning module using less labeled data in the embodiment of the present application can make the data distribution features of the target domain closer to the data distribution features of the source domain, and improve the classification prediction accuracy of the model on the unlabeled data in the target domain.

[0060] According to an embodiment of the present invention, the penalty for the conditional distribution difference between the sample features corresponding to the samples in the source domain and the target domain includes: determining the conditional distribution difference between all samples in the source domain and the target domain based on the sample features of all samples in the source domain and the target domain used in each training session, the labels of all samples in the source domain and the pseudo labels of all samples in the target domain, wherein the pseudo labels are the classification results obtained by classifying the samples in the target domain using the current model.

[0061] According to one embodiment of the present invention, during a single training iteration, the formula for calculating the conditional distribution difference between sample features corresponding to samples in the source domain and the target domain is as follows:

[0062]

[0063] Among them, G c This represents the conditional distribution difference among all samples in the source and target domains used in a single training iteration, where K represents the total number of classes and k represents the k-th class. This represents the weight of the k-th class in the source domain. This represents the total number of samples in the source domain belonging to the k-th category. This represents the weight of the k-th class in the target domain. This represents the total number of samples belonging to the k-th category in the target domain. This represents the pseudo-label of the i-th sample in the target domain. Since the labels of each sample in the source domain are known, we can count the number of labels for each class in the source domain. Since the true labels in the target domain are unknown, we obtain the pseudo-labels for all samples in the target domain during one training iteration using the classifier of the current model. And based on the pseudo-labels of all samples in the target domain used in one training session. Count the number of each category label in the target domain. Use To approximate substitution The embodiments of the present invention calculate the conditional distribution differences so that the sample distribution ratios of the source domain and the target domain in each category are approximately or the same, enabling the model after transfer learning to make better classification predictions for unlabeled data in the target domain.

[0064] According to one embodiment of the present application, the time sequence distribution difference is determined in the following manner: the confidence probability of each category of the sample is obtained from the sample features of the sample pair by using the classifier, and is spliced into the confidence probability vector corresponding to the sample pair; the similarity of the confidence probability vectors corresponding to the two samples in the sample pair is calculated, and the time sequence distribution difference is determined according to the similarity of the two samples corresponding to the sample pair in all sample pairs used in one training. The confidence probability vectors corresponding to the two samples in the sample pair from the same behavior category of the same person in the sample pair have higher similarity, the probability that the two samples of the sample pair obtain the same classification result is greater, and the recognition ability of the model to the two sensor data of the person in the collected state but with the same behavior category is improved.

[0065] According to one embodiment of the present application, the cosine similarity of the two confidence probability vectors is calculated according to the confidence probability vectors corresponding to the two samples in the sample pair of the source domain and the target domain used in each training, the smaller the cosine similarity value, the greater the similarity between the two confidence probability vectors, that is, the greater the similarity of the sample features of the two samples in the sample pair, therefore, the cosine similarity of the two confidence probability vectors is directly taken as the time sequence distribution difference, that is, in one training, the time sequence distribution difference is calculated in the following manner:

[0066]

[0067]

[0068] wherein, m s represents the total number of sample pairs of the source domain used in one training, x si represents the i th sample pair of the source domain, represents one sample in the i th sample pair x si of the source domain, represents the other sample in the i th sample pair x si of the source domain, sim(·) represents the cosine similarity, C(·) represents the confidence probability of each category output by the classifier, F(·) represents the output of the feature extractor, m t represents the total number of sample pairs of the target domain used in one training, x ti represents the i th sample pair of the target domain, represents one sample in the i th sample pair x ti of the target domain, represents the other sample in the i th sample pair x ti of the target domain. represents the confidence probability vector corresponding to the sample , represents the confidence probability vector corresponding to the sample , represents the confidence probability vector corresponding to the sample a corresponding confidence probability vector, representing a sample a corresponding confidence probability vector. Wherein the confidence probability vector corresponding to the sample is obtained by splicing the confidence probability of the sample in each category output by the classifier, for example, the confidence probability vector representing a sample confidence probability 0.1, confidence probability 0.08, confidence probability 0.12, and confidence probability 0.7. The embodiment of the application adds a unique time sequence distribution difference calculation for sensor data, further improving the classification prediction accuracy of the model for recognizing human behavior based on sensor data.

[0069] According to another embodiment of the application, the time sequence distribution difference can also be calculated in one training as follows:

[0070]

[0071] Wherein, the above time sequence distribution difference calculation method is to average the total time sequence distribution difference of all sample pairs to obtain the average time sequence distribution difference of each sample pair in the source domain and the average time sequence distribution difference of each sample pair in the target domain, and the sum of the average time sequence distribution differences of the source domain and the target domain is used as the time sequence distribution difference to avoid the influence of the model training speed caused by the more sample pairs leading to the larger time sequence distribution difference.

[0072] According to another embodiment of the application, the time sequence distribution difference can also be calculated in one training as follows:

[0073]

[0074] Wherein, γ represents the weight value corresponding to the time sequence distribution difference of the source domain, δ represents the weight value corresponding to the time sequence distribution difference of the target domain, and γ+δ=1.

[0075] Since the two samples of a sample pair come from the same behavior category of the same person, but the two samples are collected at different times, and the state of the person at different times will cause the sample feature difference between the two samples, therefore, the time sequence distribution difference between the two samples of all sample pairs is calculated by the above embodiments, so that the model considers the time sequence distribution difference of the sample pair when training and updating, and improves the classification accuracy of the model for the two samples in the same behavior category of the same person in the sample pair.

[0076] According to one embodiment of the present application, the updating of the parameters of the model based on the predetermined loss function comprises: updating the parameters of the model based on the total loss calculated according to the predetermined loss function, wherein the total loss is calculated as follows during one training:

[0077]

[0078] wherein J(·) represents the classification bias of the samples in the source domain during one training, C(·) represents the confidence probability of the classifier output for each category, F(·) represents the output of the feature extractor, x s represents the samples in the source domain, y s represents the label of the samples in the source domain, a represents the weight value corresponding to the edge distribution difference, G m (·) represents the edge distribution difference between all samples in the source domain and the target domain during one training, x t represents the samples in the target domain, b represents the weight value corresponding to the conditional distribution difference, G c (·) represents the conditional distribution difference between all samples in the source domain and the target domain during one training, s represents the weight value corresponding to the time sequence distribution difference, G t (·) represents the time sequence distribution difference between the two samples of a sample pair in the source domain and the target domain during one training, x i represents the i-th sample pair, represents one sample in the sample pair x i represents the other sample in the same sample pair x represents the other sample in the same sample pair x i , and a+b+s=1. The total loss calculated by the predetermined loss function of the present application includes the edge distribution difference, the conditional distribution difference, and the time sequence distribution difference specific to the sensor data, and the three differences are weighted and calculated. According to actual needs, a higher weight value can be set for the time sequence distribution difference to improve the classification prediction accuracy of the model for identifying human behaviors based on corresponding sensor data.

[0079] According to one embodiment of the present application, when the neural network classification model in the above embodiment is trained using a sensor data set related to Parkinson tremor, the obtained neural network classification model is used to identify the behavior category of Parkinson tremor of the human body, wherein the behavior category of Parkinson tremor includes no tremor, mild tremor, moderate tremor, and severe tremor.

[0080] According to an embodiment of the present application, when the neural network classification model in the above embodiment is trained using a sensor data set related to human daily activities, the obtained neural network classification model is used to identify the behavior category of human daily activities, wherein the behavior category of human daily activities includes walking, climbing stairs, descending stairs, and sitting. The method of the present application can be applied to a variety of different human behavior recognition tasks based on corresponding sensor data, and has a wide range of applications.

[0081] IV. Application scenarios

[0082] According to an embodiment of the present application, the present application also provides a method for identifying human behavior, comprising: obtaining sensor data to be identified collected from a human body; using a feature extractor of a neural network classification model for identifying human behavior obtained by the training method in the above embodiment to perform feature extraction on the sensor data to be identified, and identifying human behavior according to the extracted features through the classifier thereof to obtain a behavior category of the human body.

[0083] In order to verify the effect of the present application, the inventors conducted the following experiments.

[0084] Firstly, the data sets are selected. On the one hand, experiments are conducted for human daily activity recognition, and three human daily activity recognition (RAR) data sets are selected: PAMAP2 data set (denoted as data set P), WISDM data set (denoted as data set W), and UCIHAR data set (denoted as data set U). On the other hand, experiments are conducted for human Parkinson tremor recognition, and four Parkinson tremor recognition (PD) data sets are selected: TIM-Tremor data set (denoted as data set TI), PdAssist data set (denoted as data set PA), IMU-Wild data set (denoted as data set IW), and PD-BioStamp data set (denoted as data set BI).

[0085] Secondly, the data sets are processed in the manner in the above embodiment.

[0086] Then, a training method for comparison with the present application is selected. A variety of different transfer learning algorithms are selected, which are combined with the time series domain adaptation network (TSAN plug-in) in the above embodiment of the present application or other plug-ins that can be used to calculate the time series domain difference to verify the effectiveness of the training method of the present application. At the same time, ResNet (RESN) that is not trained on the target domain is used as a baseline model to test whether negative transfer occurs. The mainstream transfer learning algorithms selected in the experiment are shown in Table 1 below:

[0087] Table 1: Transfer learning methods

[0088]

[0089] Finally, the experimental results are shown in Tables 2 and 3 as follows:

[0090] Table 2: Results of human daily activity recognition

[0091]

[0092]

[0093] Table 3: Results of Parkinson tremor recognition

[0094]

[0095]

[0096] In Table 2 above, U->W means that the source domain uses dataset U and the target domain uses dataset W, U->P means that the source domain uses dataset U and the target domain uses dataset P, W->U means that the source domain uses dataset W and the target domain uses dataset U, W->P means that the source domain uses dataset W and the target domain uses dataset P, P->U means that the source domain uses dataset P and the target domain uses dataset U, and P->W means that the source domain uses dataset P and the target domain uses dataset U; in Table 3, TI->BI means that the source domain uses dataset TI and the target domain uses dataset BI, TI->PA means that the source domain uses dataset TI and the target domain uses dataset PA, and TI->IW means that the source domain uses dataset TI and the target domain uses dataset IW.

[0097] According to each method in Table 2 and Table 3, each method corresponds to an identification accuracy under a corresponding transfer learning task, and V_AVG represents the average of the identification accuracies of all algorithms on the task, and H_AVG represents the average of the identification accuracies of the algorithm on all tasks; the identification accuracy corresponding to the plug-in-free represents the identification accuracy of the original transfer learning algorithm, and the TSAN plug-in represents the identification accuracy of the original transfer learning algorithm plus the TSAN plug-in designed by the application for calculating the time sequence distribution difference. The identification accuracy of each method obtained by the experiment is the average result of three different random seeds, and the number after the ± symbol in the identification accuracy is the standard deviation. 6 tasks are verified on the human daily activity recognition dataset, including U->W, U->P, W->U, W->P, P->U, and P->W. Among them, the prediction accuracy after adding the TSAN plug-in is improved by 2.4% to 11% for a single task. For each transfer learning method, the prediction accuracy after adding the TSAN plug-in is improved by 2.9% to 8.5%. For all transfer learning methods on all tasks, the prediction accuracy after adding the TSAN plug-in is improved by 5.4 percentage points on average. 3 tasks are verified on the Parkinson tremor recognition dataset, including TI->BI, TI->PA, and TI->IW. Among them, the prediction accuracy after adding the TSAN plug-in is improved by 1.5% to 3.2% for a single task. For a single transfer learning method, the prediction accuracy after adding the TSAN plug-in is improved by 1.5% to 3%. For all transfer learning methods on all tasks, the prediction accuracy after adding the TSAN plug-in is improved by 2.2 percentage points on average. It can be seen that the identification accuracy of the transfer learning algorithm after adding the TSAN plug-in for calculating the time sequence distribution difference is higher than the identification accuracy of the original transfer learning algorithm without calculating the time sequence distribution difference, and the method of the application can be combined with various mainstream transfer learning algorithms to better adapt to human behavior recognition tasks.

[0098] It should be noted that although the above describes each step in a specific order, it does not mean that each step must be performed in the above specific order. In fact, some of these steps can be performed concurrently or even in reverse order, as long as the desired function can be achieved.

[0099] The application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for causing a processor to implement various aspects of the application.

[0100] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A computer readable storage medium can be any suitable medium tangibly embodying a set of computer instructions executable by a machine. More specific examples (a non-exhaustive list) of the computer readable storage medium include: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0101] Embodiments of the application have been described above, with examples of the description being exemplary, but not exhaustive, and are not limited to the disclosed embodiments. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The scope of the embodiments disclosed herein cover all technical equivalents that perform similar functions, possess similar structures, or achieve similar results. The selection of the terms to be utilized in this disclosure has been made with the intent of best explaining the principles of the embodiments, practical application, or improving the technology in the art, or enabling other of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a neural network classification model for recognizing human behavior, characterized in that, The model includes a feature extractor for extracting sample features from samples and a classifier for classifying samples based on sample features. The method includes training the model using samples and labels in the source domain to obtain an initial model, and performing one or more rounds of iterative training based on the initial model in the following manner: S1. Obtain the training set, which includes sample pairs of the source domain generated from samples of the source domain and sample pairs of the target domain generated from samples of the target domain. Each sample pair includes two samples, each of which is sensor data collected from a person. The two samples correspond to the same person and the same behavior category of the person when the data is collected, but the two samples are collected at different times. Each sample in the training set is labeled with a label indicating the behavior category of the person when the data is collected. S2. Train the current model once or multiple times using the training set. During each training iteration, update the model's parameters based on a predetermined loss function. The loss function is configured to: penalize the classification bias of samples in the source domain, penalize the marginal distribution difference and conditional distribution difference between the corresponding sample features of samples in the source and target domains, and penalize the temporal distribution difference between two samples in a sample pair in the source and target domains. The temporal distribution difference is related to the similarity of the confidence probability vectors of the classifier classifying the two samples in the sample pair. The temporal distribution difference is determined as follows: The classifier is used to obtain the confidence probability of the sample in each category based on the sample features and then concatenates them to form the confidence probability vector corresponding to the sample. The similarity is calculated based on the confidence probability vectors of the two samples in a sample pair, and the temporal distribution difference is determined based on the similarity of the two samples in all sample pairs used in a single training session. The temporal distribution difference is calculated as follows during a single training session: ， in, This represents the total number of source domain sample pairs used in a single training iteration. Represents the first term of the source domain One sample pair, Represents the first term of the source domain a sample pair One sample, Represents the first term of the source domain a sample pair Another sample, Represents cosine similarity. This represents the confidence probability output by the classifier for each class. This represents the output of the feature extractor. This represents the total number of sample pairs in the target domain used during a single training iteration. The first term of the target domain One sample pair, The first term of the target domain a sample pair One sample, Indicates the first in the target domain a sample pair Another sample.

2. The training method according to claim 1, characterized in that, The conditional distribution differences between the sample features corresponding to the samples in the penalty source domain and the target domain include: Based on the sample features of all samples in the source and target domains used in each training session, the labels of all samples in the source domain, and the pseudo-labels of all samples in the target domain, the conditional distribution differences between all samples in the source and target domains are determined, wherein the pseudo-labels are the classification results obtained by classifying the samples in the target domain using the current model.

3. The training method according to claim 2, characterized in that, During a single training session, the conditional distribution difference is calculated as follows: ， in, This represents the difference in conditional distribution among all samples in the source and target domains used during a single training iteration. Indicates the total number of categories. Indicates the first Categories This represents the square of the distance in Hilbert space. This represents the total number of samples in the source domain during one training iteration. , Indicates the first in the source domain The weights of each category, Indicates that the source domain belongs to the first The total number of samples in each category, Represents the first term of the source domain The label of each sample This represents a feature mapping method. This represents the output of the feature extractor. Represents the first term of the source domain One sample, This represents the total number of samples in the target domain during one training iteration. , Indicates the first in the target domain The weights of each category, Indicates that the target domain belongs to the first The total number of samples in each category, The first term of the target domain Pseudo-labels for each sample The first term of the target domain One sample.

4. The training method according to claim 1, characterized in that, The step of updating the model parameters based on a predetermined loss function includes updating the model parameters based on the total loss calculated using the predetermined loss function. The total loss is calculated during a single training iteration as follows: ， in, This represents the classification bias of the samples in the source domain during a single training iteration. This represents the confidence probability output by the classifier for each class. This represents the output of the feature extractor. Represents samples in the source domain. Labels representing samples from the source domain. This represents the weight values ​​corresponding to the differences in marginal distribution. This represents the difference in marginal distributions among all samples in the source and target domains used during a single training iteration. Represents samples in the target domain. This represents the weight values ​​corresponding to the differences in conditional distributions. This represents the difference in conditional distribution among all samples in the source and target domains used during a single training iteration. This represents the weight values ​​corresponding to the differences in time series distributions. This represents the temporal distribution difference between two samples in all sample pairs from the source and target domains used during a single training iteration. Indicates the first One sample pair, Indicates sample pairs One of the samples, Indicates the same sample pair Another sample in the sample, .

5. The training method according to claim 1, characterized in that, The two samples in each sample pair are collected at different times and the interval is less than or equal to the predetermined duration.

6. The training method according to claim 5, characterized in that, Sample pairs are generated in the following way: Acquire datasets for the source and target domains, which include multiple initial sensor data for predetermined durations collected from different individuals under different behavioral categories; Each initial sensor data in the source and target domains is divided into segments of sensor data of a predetermined length to obtain multiple segments of sensor data corresponding to each initial sensor data. In each initial sensor data segment in the source domain, a data segment is randomly selected as a sample. In the initial sensor data segment to which the sample belongs, another data segment is randomly selected as another sample. The two selected samples are combined to form a sample pair in the source domain. Each sample in the sample pair is assigned the same label as the initial sensor data to which it belongs. In each initial sensor data segment corresponding to the target domain, a data segment is randomly selected as a sample. In the initial sensor data segment corresponding to the sample, another data segment is randomly selected as another sample. The two selected samples are then combined to form a sample pair for the target domain.

7. The training method according to any one of claims 1-6, characterized in that, The identification of human behavior categories includes the identification of Parkinson's tremor behavior categories, wherein Parkinson's tremor behavior categories include no tremor, mild tremor, moderate tremor, and severe tremor.

8. The training method according to any one of claims 1-6, characterized in that, The identification of human behavior categories includes the identification of human daily activity behavior categories, which include walking, climbing stairs, descending stairs, and sitting still.

9. A method for recognizing human behavior, characterized in that, include: Acquire sensor data to be identified from the human body; The feature extractor of the neural network classification model for recognizing human behavior, obtained by the training method described in any one of claims 1-8, extracts features from the sensor data to be recognized, and the classifier identifies human behavior based on the extracted features to obtain the human behavior category.

10. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8 and 9.

11. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 8 and 9 by executing the executable instructions.

Images