Domain adaptive behavior recognition system and method based on data enhancement and pseudo-labeling

Abstract

The present invention discloses a domain adaptive behavior recognition system and method based on data enhancement and pseudo-label, including: a data acquisition module, which collects raw sensor data when a user moves; a data preprocessing module, which normalizes and segments the source domain sensor data and the collected target domain sensor data in the public data set to obtain source domain and target domain sample data; a behavior recognition module, which selects source domain sample data for training to obtain a domain generalization basic model, performs data enhancement on the target domain sample data to obtain enhanced target domain sample data, uses an iterative calculation pseudo-label algorithm and the enhanced target domain sample data to optimize the domain generalization basic model, and the classification result of the target domain sample data when the iteration terminates is the behavior recognition result. The present invention optimizes the classification model using unlabeled target domain sample data, reduces the dependence on the labeled samples in the target domain, and improves the accuracy and robustness of behavior recognition while reducing the labeling cost.

Description

Technical Field

[0001] The present invention belongs to the fields of intelligent perception technology and artificial intelligence technology, and specifically relates to a domain adaptive behavior recognition system and method based on data enhancement and pseudo-labeling. Background Art

[0002] At present, mobile smart devices and wearable smart devices have become an important part of people's daily life. People use these devices to detect and record their behavior. Sensors compatible with mobile devices and wearable devices save and record the data generated by user activities. These data are widely used in behavior recognition tasks, such as motion detection, human-computer interaction and other applications.

[0003] For the behavior recognition task based on wearable devices, the traditional solution is to train the behavior recognition model in a supervised way, that is, to train the classification model with a large amount of labeled sample data, and then use the model in actual applications to classify the perception data corresponding to the user behavior to achieve the goal of behavior recognition. However, the traditional method usually requires that the sample data in the training process and the data in the actual application follow the same data distribution. In actual application scenarios, due to differences in users, devices, environments, etc., the model often faces data domains with different distributions during training and actual deployment, resulting in a decrease in recognition performance.

[0004] In view of the above situation, an intuitive solution is to use labeled samples to train a classification model in a supervised manner for each data domain in the actual application. However, the cost of labeling unlabeled samples is often high, so this method is difficult to apply in actual situations. These problems often bring limitations to the behavior recognition tasks based on wearable devices and human-computer interaction services based on behavior recognition. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a domain adaptive behavior recognition system and method based on data enhancement and pseudo-labeling, so as to solve the problem that the existing behavior recognition model has poor performance when performing behavior recognition tasks in a new data domain (i.e., target domain) without labeled data; the present invention enables the model to use the unlabeled data in the target domain for adjustment and optimization, reduces the dependence on labeled samples in the target domain, and improves the accuracy and robustness of behavior recognition while reducing the labeling cost.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The present invention provides a domain adaptive behavior recognition system based on data enhancement and pseudo-labeling, comprising: a data acquisition module, a data preprocessing module and a behavior recognition module; wherein:

[0008] A data collection module is used to collect sensor data from a mobile device or wearable device when a user is exercising;

[0009] A data preprocessing module is used to process and obtain source domain sample data and target domain sample data, including: a data normalization module and a data segmentation module;

[0010] The data normalization module is used to normalize each dimension of the source domain sensor data in the public data set and the collected target domain sensor data (i.e., sensor data of the mobile device or wearable device) to the interval range of (0,1);

[0011] The data segmentation module is used to segment the normalized source domain sensor data and target domain sensor data into data segments of a uniform fixed length to obtain source domain sample data and target domain sample data;

[0012] Behavior recognition module, including: domain generalization basic model training module, multiple data enhancement module, iterative calculation pseudo label module;

[0013] The domain generalization basic model training module uses source domain sample data to train and obtain a domain generalization basic model;

[0014] The multiple data enhancement module is used to perform multiple data enhancements of different degrees on the target domain sample data;

[0015] The iterative calculation pseudo-label module is used to iteratively calculate pseudo-labels and optimize the domain generalization basic model, and the classification result when the iteration is terminated is output as the behavior recognition result.

[0016] Furthermore, the sensor data includes: accelerometer data, magnetometer data, and gyroscope data.

[0017] Furthermore, the data normalization module calculates the maximum value of each dimension of the sensor data in the source domain and the target domain, max(X nomal ) and the minimum value min(X nomal ), use the following formula to normalize the data of each dimension of the sensor to the (0, 1) interval:

[0018]

[0019] Where Y nomal is the normalized data output by the data normalization module, X nomal is the raw sensor data.

[0020] Furthermore, the data segmentation module uses a sliding window algorithm to segment all normalized sensor data into data segments of fixed window length, and the window overlap length is half of the window length.

[0021] Furthermore, the sliding window length in the sliding window algorithm is 256, the window overlap length is 128, and the step length is 128.

[0022] Furthermore, the domain generalization basic model training module extracts a sample data into a 256-dimensional feature vector by adding adaptive hyperparameters and data enhancement layers to the one-dimensional convolutional neural network, specifically: a convolutional neural network with 4 one-dimensional convolutional layers is used as the basic architecture, and a data enhancement layer using adaptive hyperparameters is added after some convolutional layers;

[0023] The domain generalization basic model includes: one-dimensional convolution layer, pooling layer, fully connected layer, classification layer and affine transformation layer; among them:

[0024] One-dimensional convolution layer: extracts feature information from sample data. cv , the output feature is represented as Y cv The one-dimensional convolution formula is expressed as follows:

[0025]

[0026] Among them, y cout，i Represents the output feature representation Y cv , the data at position i in the cout channel; C, H represent the input data X cv The number of channels (i.e. the total number of all sensor coordinate axes) and the length of the one-dimensional convolution kernel; step represents the step size of the one-dimensional convolution operation; x c，(i-1)×step+h Represents the input data X cv The data at position (i-1)×step+h in the cth channel; k cout，h is the data at position h in the cout-th one-dimensional convolution kernel;

[0027] Pooling layer: Pooling calculation is performed on the data with a pooling window of fixed length. The operation of calculating the maximum value of the elements in the pooling window is called max-pooling. For the pooling operation, the corresponding input is X pool , the output is Y pool , the calculation formula of the maximum pooling is as follows:

[0028]

[0029] Among them, y c，i Represents output data Y pool The data at position i in the cth channel of ; M represents the length of the pooling window; x c，M×(i-1)+k Represents the input data X pool The data at position M×(i-1)+k in the c-th channel;

[0030] Fully connected layer: used to transform the input vector X fc The output is a vector Y of the target dimension fc , and its calculation formula is as follows:

[0031] Y fc =F c (X fc )=W c X fc +b c

[0032] Among them, W c is the weight matrix, b c is the bias vector;

[0033] Classification layer: used to represent the extracted vector X classifier , mapped to the sample label space, including a fully connected layer and a normalized exponential function (Softmax); for the input vector X classifier , the output vector Y classifier , the length of which is the number of sample categories in the label space, and its formula is expressed as follows:

[0034]

[0035] Among them, e is a natural constant, W m is the weight vector corresponding to the mth category, and S is the number of sample categories in the feature space;

[0036] Affine transformation layer: used to perform data enhancement on source domain sample data through affine transformation during training, and use enhanced data to simulate domain shift in the real environment, so that the model can learn more data distribution during training, and then have preliminary generalization ability for target domains with unknown data distribution; the specific implementation is: different parameters are used for affine transformation of different channels of data, and the weight coefficients and bias values ​​used by affine transformation are sampled from the normal distribution determined by the hyperparameters. The formula of the affine transformation module is expressed as follows:

[0037] γ~N(1,softplus(θ γ )),θ γ ∈R C×1

[0038] β~N(0,softplus(θ β )),θ β ∈R C×1

[0039] y c =γ c × c +β c

[0040] Among them, N(,) is a normal distribution, θ γ and θ β is a hyperparameter, softplus is an activation function, and γ and β are sampled from θ γ and θ β The parameters of the determined normal distribution are used as the weight coefficient and bias value of the affine transformation, C is the number of channels of the input data (that is, the total number of all sensor coordinate axes), c represents the cth channel, γ c and β c represents the values ​​of parameters γ and β on the cth channel, x c Represents the input data on the cth channel, y c Represents the output on the cth channel.

[0041] Furthermore, the affine transformation layer continuously updates the hyperparameter θ during the training process. γ and θ β ; Use the meta-learning method to update the hyperparameters during the training process, so that the model can independently determine the direction of data enhancement, and then change the data enhancement in a direction that is beneficial to training; the specific method is that during the second propagation of model training, when the back-propagation gradient reaches the affine transformation layer, the gradient is used to update θ γ and θ β , the formula is as follows:

[0042]

[0043]

[0044] in, represents theta before update γ , represents theta before update β , represents the updated θγ, represents the updated θ β , t and t+1 represent two adjacent rounds in the training process, α and ε are learning rates, and Represents the gradient when back-propagating to the affine transformation layer during the second propagation process.

[0045] Furthermore, the multiple data enhancement modules act on the stage of optimizing the basic model of the target domain sample data, and are used to perform different degrees of data enhancement on the target domain sample data; different degrees of data enhancement are all designed using a method pool, that is, a method is randomly selected from the method pool for data enhancement.

[0046] Further, the multiple data enhancement modules include: a weak data enhancement module and a strong data enhancement module;

[0047] Weak data enhancement module, the weak data enhancement method pool includes three basic methods: data inversion, adding Gaussian noise and affine transformation; for the input data X wa , the result after data enhancement is Y wa , the formulas corresponding to the three data enhancement methods are as follows:

[0048] Y wa =WeakAugment reverse (X wa )=-X wa

[0049] Y wa =WeakAugment noisy (X wa )=X wa +σ

[0050]

[0051] σ~N(0,softplus(θ σ )),θ σ ∈R C×H

[0052]

[0053]

[0054] Among them, N(,) is a normal distribution, softplus is an activation function, C, H represent the input data X, wa The number of channels (i.e. the total number of all sensor coordinate axes) and the window length (i.e. the number of sampling points), θ σ , θ τ is a hyperparameter, and random sampling is performed according to the normal distribution determined by the hyperparameter to obtain the parameters σ and σ required for data enhancement. τ;

[0055] Strong data enhancement module, the strong data enhancement method pool includes two combination methods based on weak data enhancement and one method based on resampling; the first combination method based on weak data enhancement performs several random weak data enhancements on the sample data; the second method segments the sample data and uses different weak data enhancement methods in different data segments; for the input data X sa , the result after data enhancement is Y sa , the formulas of the two combination methods are expressed as follows:

[0056] Ysa =StrongAugment stream (X sa )=WeakAugment d (X sa )

[0057]

[0058] Among them, d represents the number of weak data enhancements in the first combination method, x i represents the i-th data segment of the input data in the second combination method, y i Represents x i Results after data augmentation;

[0059] The resampling-based method is as follows: select a sample data, copy the sample data to obtain a copy data, concatenate the sample data with its copy data, and use downsampling to obtain a data segment with a window length of W. sa , the result after data enhancement is Y s a. The formula corresponding to the resampling data enhancement method is expressed as follows:

[0060]

[0061] Among them, X′ represents the concatenated sample data.

[0062] Furthermore, the iterative calculation pseudo-label module uses the domain generalization basic model obtained by the domain generalization basic model training module to classify the unlabeled target domain sample data, selects the classification results with high confidence as the pseudo-labels of the corresponding sample data to add to the next round of iterative training, and calculates the loss function:

[0063]

[0064] p′ i = argmax(p i )

[0065] Among them, N s represents the number of samples, l(·) represents the 0-1 function (when max(p i )>δ, l(·) is 1; otherwise, l(·) is 0), δ represents the threshold for judging the confidence level, H represents the cross entropy loss function, and p i The softmax layer outputs the result. The argmax function is used to find the corresponding parameter when the function reaches its maximum value. The basic model is updated using backpropagation and the process is iterated until the iteration ends. The classification result is then output as the result of behavior recognition.

[0066] A domain adaptive behavior recognition method based on data enhancement and pseudo-labeling of the present invention comprises the following steps:

[0067] 1) Collect raw sensor data from mobile smart devices or wearable smart devices when users are exercising;

[0068] 2) Normalize and segment the source domain sensor data and the collected target domain sensor raw data in the public dataset to obtain source domain sample data and target domain sample data;

[0069] 3) Using the source domain sample data to train a domain generalization basic model;

[0070] 4) performing multiple data enhancement on the target domain sample data to obtain target domain sample data after data enhancement;

[0071] 5) Use the target domain sample data after data enhancement and the pseudo-labeling algorithm based on multiple rounds of iterations to continue to optimize the domain generalization basic model. The classification result at the end of the iteration is the behavior recognition result.

[0072] Furthermore, the step 1) specifically includes: collecting data from the built-in sensors of the mobile device or wearable device using a sampling frequency of 50 Hz, and saving the data of each sensor on the three axes of x, y, and z respectively.

[0073] Furthermore, the mobile device or wearable device in step 1) is worn on the user's left or right arms, head, waist or ankles.

[0074] Furthermore, the step 2) specifically includes:

[0075] 21) Normalize each dimension of the original data of the source domain sensor in the public data set and the original data of the target domain sensor collected, and normalize all data to the interval range of (0, 1);

[0076] 22) Use the sliding window algorithm to segment the normalized data into data segments of fixed window length, and each data segment is regarded as a sample; the window size is set to 256, and the sliding window step is 128; and then obtain the labeled source domain sensor data segments and the unlabeled target domain sensor data segments;

[0077] 23) Divide the labeled source domain sensor data segment into a plurality of source domain sample data.

[0078] Furthermore, the specific steps of step 3) are as follows:

[0079] 31) Randomly select Q source domain sample data from multiple source domain sample data as the support set And randomly extract samples from the source domain sample data support set to train the model. During the back propagation process of training, only the parameters of the one-dimensional convolutional network are optimized; this process is recorded as the first propagation;

[0080] 32) Randomly select one source domain sample data that is different from the above Q source domain sample data from multiple source domain sample data as the query set {T pu}, and randomly extract samples from the source domain sample data query set to train the model after the first propagation, and only optimize the parameters of the affine transformation layer in the back propagation; this process is recorded as the second propagation;

[0081] 33) Perform one first propagation and one second propagation in each round of training the model. The first propagation is used to update the model parameters and optimize the feature extractor, and the second propagation is used to update the hyperparameters of the affine transformation layer and enable the model to adaptively adjust the direction of data enhancement. Repeat this process until the end of training;

[0082] 34) According to the steps 31) to 33), a domain generalization basic model is trained to extract sample data into a 256-dimensional feature vector and classify it, and the domain generalization basic model is used to mark pseudo labels for the target domain sample data.

[0083] Furthermore, the step 4) specifically includes:

[0084] 41) Perform weak data enhancement on the data in the target domain to obtain corresponding weak data enhancement results;

[0085] 42) Perform strong data enhancement on the data in the target domain to obtain corresponding strong data enhancement results.

[0086] Furthermore, the step 5) specifically includes:

[0087] 51) passing the weak data enhancement result of the target domain sample data obtained in step 4) through the domain generalization basic model to obtain the classification result of the target domain sample data at the softmax layer;

[0088] 52) All target domain sample data whose confidence level of the target domain sample data classification result is greater than a certain threshold are selected as the high confidence sample data of this round, and the corresponding classification result is used as the pseudo label corresponding to the target domain sample data;

[0089] 53) The strong data enhancement results of the target domain high-confidence sample data and the pseudo-labels corresponding to the sample data are added to the next round of iterative training, and the domain generalization basic model is further optimized using cross entropy loss and back propagation;

[0090] 54) The classification result of the target domain sample data at the end of the iteration is the behavior recognition result.

[0091] Beneficial effects of the present invention:

[0092] The present invention can collect sensor data of users during exercise through common mobile devices and wearable devices, and provide domain-adaptive behavior recognition services based on the collected data. Users do not need to manually provide labeling information about the current motion state when using the service, nor do they need to use specific models of mobile devices or wearable devices, thereby reducing the cost of manually labeling samples and the constraints caused by using specific devices.

[0093] The present invention makes up for the poor performance of traditional behavior recognition models in new data domains with unknown distribution by using a method combining multiple data enhancements with pseudo-labels; improves the model's feature extraction capability for target domain sample data by training a domain generalization basic model, so that the model has a certain generalization capability; further optimizes the domain generalization basic model by using unlabeled target domain sample data, thereby improving the recognition performance of the model on target domain sample data; and simultaneously adds double data enhancement in the process of iteratively calculating pseudo-labels, and introduces a new data distribution through data enhancement to reduce the possibility that the model identifies the target domain sample data as the wrong category, thereby enhancing the accuracy and robustness of behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 Schematic diagram of the system of the present invention.

[0095] Figure 2 It is a schematic diagram of the behavior recognition module in the present invention.

[0096] Figure 3 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0097] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0098] Reference Figure 1 As shown, a domain adaptive behavior recognition system based on data enhancement and pseudo-labeling of the present invention includes: a data acquisition module, a data preprocessing module and a behavior recognition module; wherein,

[0099] A data acquisition module is used to collect sensor data of a mobile device or a wearable device when a user is exercising; the sensor data includes: accelerometer data, magnetometer data, and gyroscope data;

[0100] A data preprocessing module is used to process and obtain source domain sample data and target domain sample data, including: a data normalization module and a data segmentation module;

[0101] The data normalization module is used to normalize each dimension of the source domain sensor data and the collected target domain sensor data in the public data set to the interval range of (0, 1); specifically, to find the maximum value max(X nomal ) and the minimum value min(X nomal ), use the following formula to normalize the data of each dimension of the sensor to the (0, 1) interval:

[0102]

[0103] Where Y nomal is the normalized data output by the data normalization module, X nomal is the raw sensor data;

[0104] The data segmentation module is used to segment the normalized source domain sensor data of the public data set and the collected target domain sensor data into data segments of uniform fixed length to obtain source domain sample data and target domain sample data;

[0105] The data segmentation module uses a sliding window algorithm to segment all normalized sensor data into data segments of fixed window length, and the window overlap length is half of the window length;

[0106] The sliding window length in the sliding window algorithm is 256, the window overlap length is 128, and the step length is 128.

[0107] Reference Figure 2 As shown, the behavior recognition module includes: a domain generalization basic model training module, a multiple data enhancement module, and an iterative calculation pseudo label module;

[0108] The domain generalization basic model training module uses source domain sample data to train and obtain a domain generalization basic model;

[0109] The multiple data enhancement module is used to perform multiple data enhancements of different degrees on the target domain sample data;

[0110] The iterative calculation pseudo-label module is used to iteratively calculate pseudo-labels and optimize the domain generalization basic model, and the classification result at the end of the iteration is output as the behavior recognition result;

[0111] The domain generalization basic model training module extracts a sample data into a 256-dimensional feature vector by adding adaptive hyperparameters and data enhancement layers to a one-dimensional convolutional neural network. Specifically, a convolutional neural network with four one-dimensional convolutional layers is used as the basic architecture, and a data enhancement layer with adaptive hyperparameters is added after some convolutional layers.

[0112] The domain generalization basic model includes: one-dimensional convolution layer, pooling layer, fully connected layer, classification layer and affine transformation layer; among them:

[0113] One-dimensional convolution layer: extracts feature information from sample data. cv , the output feature is represented as Y cv The one-dimensional convolution formula is expressed as follows:

[0114]

[0115] Among them, y cout，i Represents the output feature representation Y cv The data at position i in the cout channel; C and H represent the input data X cv The number of channels (i.e. the total number of all sensor coordinate axes) and the length of the one-dimensional convolution kernel; step represents the step size of the one-dimensional convolution operation; x c，(i-1）×step+h Represents the input data X cv The data at position (i-1)×step+h in the cth channel; k cout，h is the data at position h in the cout-th one-dimensional convolution kernel;

[0116] Pooling layer: Pooling calculation is performed on the data with a pooling window of fixed length. The operation of calculating the maximum value of the elements in the pooling window is called max-pooling. For the pooling operation, the corresponding input is X poot , the output is Y pool , the calculation formula of the maximum pooling is as follows:

[0117]

[0118] Where M represents the length of the pooling window; x c，M×(i-1)+k Represents the input data X pool The data at position M×(i-1)+k in the c-th channel;

[0119] Fully connected layer: used to transform the input vector X fc The output is a vector Y of the target dimension fc , and its calculation formula is as follows:

[0120] Y fc =F c (X fc )=W c X fc +b c

[0121] Among them, W c is the weight matrix, b c is the bias vector;

[0122] Classification layer: used to represent the extracted vector X classifier Mapped to the sample label space, including a fully connected layer and a normalized exponential function (Softmax); for the input vector X classifier , the output vector Y classifier The length of is the number of sample categories in the label space, and its formula is expressed as follows:

[0123]

[0124] Among them, e is a natural constant, W m is the weight vector corresponding to the mth category, and S is the number of sample categories in the feature space;

[0125] Affine transformation layer: used to perform data enhancement on source domain sample data through affine transformation during training, and use enhanced data to simulate domain shift in the real environment, so that the model can learn more data distribution during training, and then have preliminary generalization ability for target domains with unknown data distribution; the specific implementation is: use different parameters to perform affine transformation on data from different channels (i.e. different coordinate axes), and the weight coefficients and bias values ​​used in affine transformation are sampled from the normal distribution determined by the hyperparameters. The formula of the affine transformation module is expressed as follows:

[0126] γ~N(1,softplus(θ γ )),θ γ ∈R C×1

[0127] β~N(0,softplus(θ β )),θ β ∈R C×1

[0128] y c =γ c × c +β c

[0129] Among them, N(,) is a normal distribution, θ γ and θ β is a hyperparameter, softplus is an activation function, and γ and β are sampled from θ γ and θ β The parameters of the determined normal distribution are used as the weight coefficient and bias value for affine transformation, C is the number of channels of the input data (i.e. the total number of all sensor coordinate axes), c represents the cth channel, γ c and β c represents the values ​​of parameters γ and β on the cth channel, x c Represents the input data on the cth channel, yc Represents the output on the cth channel.

[0130] Specifically, the affine transformation layer continuously updates the hyperparameter θ during the training process. γ and θ β ; Use the meta-learning method to update the hyperparameters during the training process, so that the model can independently determine the direction of data enhancement, and then change the data enhancement in a direction that is beneficial to training; the specific method is that during the second propagation of model training, when the back-propagation gradient reaches the affine transformation layer, the gradient is used to update θ γ and θ β , the formula is as follows:

[0131]

[0132]

[0133] in, represents theta before update γ , represents θβ before update, represents the updated θγ, represents the updated θ β , t and t+1 represent two adjacent rounds in the training process, α and ε are learning rates, and Represents the gradient when back-propagating to the affine transformation layer during the second propagation process.

[0134] The multiple data enhancement modules act on the stage of optimizing the basic model of the target domain sample data, and are used to perform different degrees of data enhancement on the target domain sample data; different degrees of data enhancement are all designed using a method pool, that is, a method is randomly selected from the method pool for data enhancement.

[0135] Wherein, the multiple data enhancement modules include: a weak data enhancement module and a strong data enhancement module;

[0136] Weak data enhancement module, the weak data enhancement method pool includes three basic methods: data inversion, adding Gaussian noise and affine transformation; for the input data X wa , the result after data enhancement is Y wa , the formulas corresponding to the three data enhancement methods are as follows:

[0137] Y wa =WeakAugment reverse (X wa )=-X wa

[0138] Y wa=WeakAugment noisy (X wa )=X wa +σ

[0139]

[0140] σ~N(0,softplus(θ σ )),θ σ ∈R C×H

[0141]

[0142] τ~N(0,softplus(θ τ )),θ τ ∈R c×1

[0143] Among them, N(,) is a normal distribution, softplus is an activation function, C, H represent the input data X, wa The number of channels (i.e. the total number of all sensor coordinate axes) and the window length (i.e. the number of sampling points), θ σ , θ τ is a hyperparameter, and random sampling is performed according to the normal distribution determined by the hyperparameter to obtain the parameters σ and σ required for data enhancement. τ;

[0144] Strong data enhancement module, the strong data enhancement method pool includes two combination methods based on weak data enhancement and one method based on resampling; the first combination method based on weak data enhancement performs several random weak data enhancements on the sample data; the second method segments the sample data and uses different weak data enhancement methods in different data segments; for the input data X sa , the result after data enhancement is Y sa , the formulas of the two combination methods are expressed as follows:

[0145] Y sa =StrongAugment stream (X sa )=WeakAugment d (X sa )

[0146]

[0147] Among them, d represents the number of weak data enhancements in the first combination method, x i represents the i-th data segment of the input data in the second combination method, y i Represents x iResults after data augmentation;

[0148] The resampling-based method is as follows: select a sample data, copy the sample data to obtain a copy data, concatenate the sample data with its copy data, and use downsampling to obtain a data segment with a window length of W. sa , the result after data enhancement is Y sa , the formula corresponding to the resampling data enhancement method is expressed as follows:

[0149]

[0150] Among them, X′ represents the concatenated sample data.

[0151] The iterative calculation pseudo-label module uses the domain generalization basic model obtained by the domain generalization basic model training module to classify the unlabeled target domain sample data, selects the classification results with high confidence as the pseudo-labels of the corresponding sample data to add to the next round of iterative training, and calculates the loss function:

[0152]

[0153] p′ i = argmax(p i )

[0154] Among them, N s represents the number of samples, l() represents the 0-1 function (when max(p i )>δ, l(·) is 1; otherwise, l(·) is 0), δ represents the threshold for judging the confidence level, H represents the cross entropy loss function, and p i The softmax layer outputs the result. The argmax function is used to find the corresponding parameter when the function reaches its maximum value. The basic model is updated using backpropagation and the process is iterated until the iteration ends. The classification result is then output as the result of behavior recognition.

[0155] Reference Figure 3 As shown, a domain adaptive behavior recognition method based on data enhancement and pseudo-labeling of the present invention comprises the following steps:

[0156] 1) Collect raw sensor data from mobile smart devices or wearable smart devices when users are exercising;

[0157] Specifically, the built-in sensors (accelerometer, magnetometer, gyroscope, etc.) of the mobile device or wearable device are used to collect data using a sampling frequency of 50 Hz, and the data of each sensor on the three axes of x, y, and z are saved respectively.

[0158] The mobile device or wearable device is worn on the user's left or right arm, head, waist or ankle.

[0159] 2) Normalize and segment the source domain sensor data and the collected target domain sensor raw data in the public data set to obtain source domain sample data and target domain sample data; specifically,

[0160] 21) Normalize each dimension (e.g., the x-axis of the accelerometer) of the original data of the source domain sensor in the public data set and the original data of the target domain sensor collected, and normalize all data to the interval range of (0, 1);

[0161] 21) Normalize each dimension of the source domain sensor data and the collected target domain sensor data in the public dataset, and normalize all data to the interval range of (0, 1);

[0162] 22) Use the sliding window algorithm to segment the normalized data into data segments of fixed window length, and each data segment is regarded as a sample; the window size is set to 256, and the sliding window step is 128; and then obtain the labeled source domain sensor data segments and the unlabeled target domain sensor data segments;

[0163] 23) Divide the labeled source domain sensor data segment into a plurality of source domain sample data.

[0164] 3) Use multiple source domain sample data to train and obtain a domain generalization basic model; the specific steps are as follows:

[0165] 31) Randomly select Q source domain sample data from multiple source domain sample data as the support set Then randomly extract samples from the source domain sample data support set to train the model. During the back propagation process of training, only the parameters of the one-dimensional convolutional network are optimized. This process is recorded as the first propagation.

[0166] 32) Randomly select one source domain sample data that is different from the above Q source domain sample data from multiple source domain sample data as the query set {T pu}, and randomly extract the model trained after the first propagation from the source domain sample data query set, and only optimize the parameters of the affine transformation layer in the back propagation; this process is recorded as the second propagation;

[0167] 33) Perform one first propagation and one second propagation in each round of training the model. The first propagation is used to update the model parameters and optimize the feature extractor, and the second propagation is used to update the hyperparameters of the affine transformation layer and enable the model to adaptively adjust the direction of data enhancement. Repeat this process until the end of training;

[0168] 34) According to the steps 31) to 33), a domain generalization basic model is trained to extract sample data into a 256-dimensional feature vector and classify it, and the domain generalization basic model is used to mark pseudo labels for the target domain sample data.

[0169] 4) performing multiple data enhancement on the target domain sample data to obtain the target domain sample data after data enhancement; specifically including:

[0170] 41) Perform weak data enhancement on the data in the target domain to obtain corresponding weak data enhancement results;

[0171] 42) Perform strong data enhancement on the data in the target domain to obtain corresponding strong data enhancement results.

[0172] 5) Use the target domain sample data after data enhancement and the pseudo-label algorithm based on multiple rounds of iterations to continue optimizing the domain generalization basic model. The classification result at the end of the iteration is the behavior recognition result; specifically, it includes:

[0173] 51) passing the weak data enhancement result of the target domain sample data obtained in step 4) through the domain generalization basic model to obtain the classification result of the target domain sample data at the softmax layer;

[0174] 52) Filter out all target domain sample data whose confidence level of classification results of target domain sample data is greater than a certain threshold (in this example, the threshold is 0.96) as the high confidence sample data of this round, and the corresponding classification results are used as the pseudo labels corresponding to the target domain sample data;

[0175] 53) The strong data enhancement results of the target domain high-confidence sample data and the pseudo-labels corresponding to the sample data are added to the next round of iterative training, and the domain generalization basic model is further optimized using cross entropy loss and back propagation;

[0176] 54) The classification result of the target domain sample data at the end of the iteration is the behavior recognition result.

[0177] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A domain adaptive behavior recognition system based on data augmentation and pseudo-labeling, characterized in that: include: Data acquisition module, data preprocessing module and behavior recognition module; A data collection module is used to collect sensor data from a mobile device or wearable device when a user is exercising; A data preprocessing module is used to process and obtain source domain sample data and target domain sample data, including: a data normalization module and a data segmentation module; The data normalization module is used to normalize each dimension of the source domain sensor data and the collected target domain sensor data in the public data set to the interval range of (0, 1); The data segmentation module is used to segment the normalized source domain sensor data and target domain sensor data into data segments of a uniform fixed length to obtain source domain sample data and target domain sample data; Behavior recognition module, including: domain generalization basic model training module, multiple data enhancement module, iterative calculation pseudo label module; The domain generalization basic model training module uses source domain sample data to train and obtain a domain generalization basic model; The multiple data enhancement module is used to perform multiple data enhancements of different degrees on the target domain sample data; The iterative calculation pseudo-label module is used to iteratively calculate pseudo-labels and optimize the domain generalization basic model, and the classification result at the end of the iteration is output as the behavior recognition result; The domain generalization basic model includes: one-dimensional convolution layer, pooling layer, fully connected layer, classification layer and affine transformation layer; One-dimensional convolution layer: extracts feature information from sample data. cv , the output feature is represented as Y cv The one-dimensional convolution formula is expressed as follows: Among them, y cout，i Represents the output feature representation Y cv The data at position i in the cout channel; C and H represent the input data X cv The number of channels and the length of the one-dimensional convolution kernel; step represents the step size of the one-dimensional convolution operation; x c，(i-1)×step+h Represents the input data X cv The data at position (i-1)×step+h in the cth channel; k cout，h is the data at position h in the cout-th one-dimensional convolution kernel; Pooling layer: Pooling calculation is performed on the data with a pooling window of fixed length. The operation of calculating the maximum value of the elements in the pooling window is called maximum pooling. For the pooling operation, the corresponding input is X pool , the output is Y pool , the calculation formula of the maximum pooling is as follows: Where M represents the length of the pooling window; x c，M×(i-1)+k Represents the input data X pool The data at position M×(i-1)+k in the c-th channel; Fully connected layer: used to transform the input vector X fc The output is a vector Y of the target dimension fc , and its calculation formula is as follows: Y fc =F c (X fc )=W c X fc +b c Among them, W c is the weight matrix, b c is the bias vector; Classification layer: used to represent the extracted vector X classifier Mapped to the sample label space, including a fully connected layer and a normalized exponential function; for the input vector X classifier , the output vector Y classifier The length of is the number of sample categories in the label space, and its formula is expressed as follows: Among them, e is a natural constant, W m is the weight vector corresponding to the mth category, and S is the number of sample categories in the feature space; Affine transformation layer: used to perform data enhancement on source domain sample data through affine transformation during training, and use enhanced data to simulate domain shift in the real environment, so that the model can learn more data distribution during training, and then have generalization ability for target domains with unknown data distribution; the specific implementation is: use different parameters to perform affine transformation on data from different channels, and the weight coefficients and bias values ​​used in affine transformation are sampled from the normal distribution determined by the hyperparameters. The formula of the affine transformation module is expressed as follows: Among them, N(,) is a normal distribution, θ γ and θ β is a hyperparameter, softplus is an activation function, and γ and β are sampled from θ γ and θ β The parameters of the determined normal distribution are used as the weight coefficient and bias value for affine transformation, C is the number of channels of the input data, that is, the total number of all sensor coordinate axes, c represents the cth channel, γ c and β c represents the values ​​of parameters γ and β on the cth channel, x c Represents the input data on the cth channel, y c Represents the output on the cth channel.

2. The domain adaptive behavior recognition system based on data enhancement and pseudo-labeling according to claim 1, characterized in that: The domain generalization basic model training module extracts a sample data into a 256-dimensional feature vector by adding adaptive hyperparameters and data enhancement layers to the one-dimensional convolutional neural network. Specifically, a convolutional neural network with 4 one-dimensional convolutional layers is used as the basic architecture, and a data enhancement layer with adaptive hyperparameters is added after some convolutional layers.

3. The domain adaptive behavior recognition system based on data enhancement and pseudo-labeling according to claim 1, characterized in that: The affine transformation layer continuously updates the hyperparameter θ during training γ and θ β ; Use meta-learning to update hyperparameters during training, so that the model can autonomously determine the direction of data enhancement, and then change the data enhancement in a direction that is beneficial to training; the specific method is that during the second propagation of model training, when the back-propagation gradient reaches the affine transformation layer, use the gradient to update θ γ and θ β , the formula is as follows: in, represents theta before update γ , represents theta before update β , represents the updated θ γ , represents the updated θ β , t and t+1 represent two adjacent rounds in the training process, α and ε are learning rates, and Represents the gradient when back-propagating to the affine transformation layer during the second propagation process.

4. The domain adaptive behavior recognition system based on data enhancement and pseudo-labeling according to claim 1, characterized in that: The multiple data enhancement modules act on the stage of optimizing the basic model, and are used to perform different degrees of data enhancement on the target domain sample data; different degrees of data enhancement are all designed using a method pool, that is, a method is randomly selected from the method pool for data enhancement.

5. The domain adaptive behavior recognition system based on data enhancement and pseudo-labeling according to claim 1, characterized in that: The iterative calculation pseudo-label module uses the domain generalization basic model obtained by the domain generalization basic model training module to classify the unlabeled target domain sample data, selects the classification results with high confidence as the pseudo-labels of the corresponding sample data to add to the next round of iterative training, and calculates the loss function: p′ i =argmax(p i ) Among them, N s represents the number of samples, l(·) represents the 0-1 function, H represents the cross entropy loss function, and p i Output the result of the softmax layer, use back propagation to update the basic model and iterate this process until the iteration ends. The classification result at this time is output as the result of behavior recognition.

6. A domain adaptive behavior recognition method based on data enhancement and pseudo-labeling, characterized in that: The steps include: 1) Collect raw sensor data from mobile smart devices or wearable smart devices when users are exercising; 2) Normalize and segment the source domain sensor data and the collected target domain sensor raw data in the public dataset to obtain source domain sample data and target domain sample data; 3) Using the source domain sample data to train a domain generalization basic model; 4) performing multiple data enhancement on the target domain sample data to obtain target domain sample data after data enhancement; 5) Use the target domain sample data after data enhancement and the pseudo-label algorithm based on multiple rounds of iterations to continue optimizing the domain generalization basic model. The classification result at the end of the iteration is the behavior recognition result; The specific steps of step 2) are as follows: 21) Normalize each dimension of the source domain sensor data and the collected target domain sensor data in the public dataset, and normalize all data to the range of (0,1); 22) Use the sliding window algorithm to segment the normalized data into data segments of fixed window length, and each data segment is regarded as a sample; the window size is set to 256, and the sliding window step is 128; and then obtain the labeled source domain sensor data segments and the unlabeled target domain sensor data segments; 23) dividing the labeled source domain sensor data segment into a plurality of source domain sample data; The specific steps of step 3) are as follows: 31) Randomly select Q source domains from multiple source domain sample data as support sets Randomly extract samples from the Q source domain sample data to train the model, and optimize the parameters of the one-dimensional convolutional network during the back propagation process; this process is recorded as the first propagation; 32) Randomly select a source domain from multiple source domain sample data that is different from the above Q source domains as the query set {T pu }, and randomly extract samples from the source domain sample data to train the model after the first propagation, and optimize the parameters of the affine transformation layer in the back propagation; this process is recorded as the second propagation; 33) Perform one first propagation and one second propagation in each round of training the model. The first propagation is used to update the model parameters and optimize the feature extractor, and the second propagation is used to update the hyperparameters of the affine transformation layer and enable the model to adaptively adjust the direction of data enhancement. Repeat this process until the end of training; 34) According to the steps 31) to 33), a domain generalization basic model is trained to extract sample data into a 256-dimensional feature vector and classify it, and the domain generalization basic model is used to mark pseudo labels for the target domain sample data; The step 5) specifically includes: 51) passing the weak data enhancement result of the target domain sample data obtained in step 4) through the domain generalization basic model to obtain the classification result of the target domain sample data at the softmax layer; 52) All target domain sample data whose confidence level of the target domain sample data classification result is greater than a certain threshold are selected as the high confidence sample data of this round, and the corresponding classification result is used as the pseudo label corresponding to the target domain sample data; 53) The strong data enhancement results of the target domain high-confidence sample data and the pseudo-labels corresponding to the sample data are added to the next round of iterative training, and the domain generalization basic model is further optimized using cross entropy loss and back propagation; 54) The classification result of the target domain sample data at the end of the iteration is the behavior recognition result.

Images